Community pull requests are the best and worst part of running an open source project. Someone you have never met spent their evening fixing your bug. Now it sits in the queue for three weeks because every maintainer is busy, and by the time anyone looks at it the branch has drifted, CI is red for unrelated reasons, and the contributor has moved on.
Agents can take most of that work. The question is how to let them without handing over the one thing a maintainer must never delegate, which is the decision to merge.
I have been studying a workflow that gets this balance right. It runs on a real project with real contributors, and I want to pull out the design choices that make it work, because they apply to any project. I will leave the project itself unnamed. The shape is what matters.
The whole interface is a label
The maintainer’s entire job at the start is to add one label: status/needs-review. That is it. No slash command, no form, no dashboard. A poll picks the label up, records a durable request so a crash cannot lose it, launches the pipeline, and posts a comment on the PR with a link to the live run. Then it swaps the label to status/reviewing.
From there the PR moves through a small state machine made entirely of labels:
needs-review → reviewing → awaiting-approval → merge-ready → merge-queued → mergedThis is the first principle. Use the primitives your contributors already understand. Labels, comments and commits are visible to everyone, survive without any custom tooling, and give the contributor a way to see what is happening to their work without logging in to anything new. If the automation disappeared tomorrow, the labels would still describe the state of the queue.
Preflight before anything expensive
The first real step is boring on purpose. Check that the description matches the diff. Rebase the PR onto today’s main in a fresh, isolated worktree. Check whether the fork allows maintainer edits, because that decides whether fixes can go back to the contributor’s branch or need a new one. Push nothing.
Most of the cost in a review pipeline is in the review itself, so anything that would make the review pointless should fail here, cheaply, before a model has read a line of the diff.
Review and CI start together
The old version of this workflow waited for CI to finish before reviewing. The new one starts both at the same moment. A CI snapshot records whether the checks pass, and that result is used later by the fix step and the gates. It never waits and it never repairs, because reviewers are reading the same worktree and a repair would change the code under them.
The lesson is to separate observation from action. Record the CI state as a fact. Decide what to do about it in a step that owns the decision.
Several reviewers, different briefs
The review is not one model reading the diff. It is two or three, each from a different model family, each in its own lane with a different brief: one reasons about intent and design, one hunts for evidence of bugs, and an optional third reads across files for knock-on effects. They run in parallel.
Different families catch different bugs. A single reviewer, however capable, has a consistent set of blind spots. Two with different training and different instructions overlap far less. This is the same reason you want two human reviewers on anything that matters, and it costs much less to get it from models.
Synthesis decides, and it is deterministic
The lanes produce findings. A synthesis step merges them, and when two lanes disagree about severity, the stricter one wins. Then a scorecard turns the merged findings into a number, and a fixed threshold decides whether the verdict is approve or request changes.
This is the part people skip, and it is the part that makes the system trustworthy. The approve decision should not come from a model’s mood. It should come from a rule that a maintainer can read, argue with, and tune. In the run I studied, pass one found one major issue, a cap precedence change with no docs or test, and three minors. The scorecard said request changes. Nobody had to interpret a paragraph of prose to know what happened next.
A bounded fix loop that stays local
When the verdict is request changes, a fix step addresses the findings. It adds one commit on top of the contributor’s commits, with a test that pins the corrected behaviour. Then the whole review runs again from the top, as a fresh pass, not a quick check of the diff of the diff.
Four details here are worth copying:
- Fixes stay local until the end. Nothing is pushed while the loop runs. The contributor sees nothing half finished on their branch.
- One fixup commit, amended per pass. Two passes do not produce two maintainer commits. The history stays clean.
- Later passes re-check fixes with mutation probes rather than trusting the fix commit’s message. A review that believes “fixed the null check” because the commit says so is not a review.
- The loop is bounded. Five passes, maximum. If it cannot converge, it stops and a human looks, rather than burning tokens forever.
When the verdict is done, the check attests the exact commit it reviewed. Only that head can move forward. If anyone, human or agent, pushes a later change, that attestation no longer matches and the PR gets one more pass. In the run I studied it converged on pass two of five.
Green CI, then a human
Before a maintainer is asked for anything, two things must be true: the reviewed head must match what is on GitHub, and required CI must genuinely pass. If CI is red, a bounded repair loop fixes it. If the repair touches code that was already reviewed, that change gets a delta review of just the delta, so the attestation still means something.
Then the pipeline stops and asks. It posts an approval packet on the PR: the decision, how many passes it took, what was fixed, what CI said, and how it intends to land the change. It adds status/awaiting-approval and emails the maintainer. Only an explicit reply of “approve”, “approved” or “proceed” counts. A thumbs up does not. Silence does not.
This is the principle the whole design hangs on. Automate everything up to the decision, and nothing past it. The maintainer reads a packet instead of a diff, which takes a few minutes instead of an hour, but the yes is still theirs.
There is an escape hatch, a separate needs-review-auto label that skips the human gate. It exists for end to end testing of the pipeline itself. If you build one of these, make the bypass a different label, make it loud, and make sure nobody can add it by accident.
Finalize, but never merge
After the approval, the finalize step re-verifies both the approval and the attested head, because the gap between asking and answering can be hours and things change. It refreshes onto the latest main, pushes to the contributor’s branch with their commits intact on top of the rebase, and waits for CI to go green again.
Then it posts a review comment addressed to the contributor: thanks, a summary of what was fixed on their behalf, and the follow-up work it noticed, filed as separate issues rather than scope creep on their PR. Finally it adds status/merge-ready.
It does not merge. A separate process takes merge-ready PRs one at a time, per repository, and that is the only thing that touches main. Keeping the merge in a different, serial, dumb process means the clever part of the system cannot merge by mistake, cannot merge twice, and cannot race another PR.
Cleanup removes the worktree and temporary refs. If any step fails hard, the run aborts but keeps the worktree, so whoever investigates has the evidence.
Spend the expensive model where it counts
One more design choice that is easy to miss. The steps are not all run by the same model. Steps that read or change code, the preflight, the fixes, the finalize, use the strongest model at maximum effort. Synthesis runs at medium effort. Cleanup runs at low. The orchestration steps, the label poll, the queue, the CI gates, the merge queue, use no model at all. They are scripts.
If a step can be a script, it should be a script. Models are for judgement, not for moving labels around.
What the contributor saw
From their side, the contributor opened a PR from a fork on a Tuesday. On Saturday a maintainer added a label. A bot commented that the PR was in the queue with a link. About thirty hours later a bot pushed one commit on top of their five, keeping their authorship, posted a thank you with a clear list of what changed and why, and the PR merged a few minutes after that.
Thirty six hours from label to merge, with two review passes, one maintainer commit, and a human who spent perhaps ten minutes on it. That is what a community PR workflow should feel like, and it is now within reach of any project willing to design the gates carefully.
The checklist
If you are building your own, here is the shape:
- Trigger on a label. Record the request durably. Post a run link.
- Preflight cheaply: description matches diff, rebase in isolation, push nothing.
- Start review and CI observation in parallel. Observe, do not act.
- Review with more than one model family, each with a different brief.
- Synthesize with stricter-severity-wins and a deterministic scorecard.
- Fix locally in one amended commit, re-review from scratch, bound the loop, attest the reviewed head.
- Gate on green CI and a matching head before you ask a human.
- Ask a human with a packet, not a diff. Accept only an explicit yes.
- Finalize by re-verifying, rebasing, pushing with contributor commits kept, thanking them, filing follow-ups.
- Never merge from the pipeline. Hand off to a separate serial queue.
- Scripts for orchestration, strong models for code, cheaper models for everything in between.
Your contributors deserve a response measured in hours, not weeks. Your maintainers deserve to spend their attention on the one decision that is actually theirs. This design gives both.