Coding agents can open PRs. They should not define reality.
A clean diff can still be wrong in ways that pass a tired skim at 6pm. Run the same checklist every time. It is not ceremony. It is how you avoid merging a very polite landmine.
If that sounds distrustful, good. Distrust scales. Vibes do not.
1) Did it solve the ticket you meant?
Reread the issue or your own brief before you look at files.
Agents are great at solving a nearby problem. You ask for a null check. You get a mini framework. You ask for a filter. You get a redesign of the table component.
If the PR title cannot be matched to a one-sentence goal, send it back before you argue about style.
2) Is the blast radius small?
Look at the file list first, not the prose summary.
Unexpected files are a smell. So are renames, mass formatting, and “while I was here” edits. Ask for a split or a revert of the extras. Mixed concerns make review theater.
Small PRs are not a moral stance. They are how you keep bisect and rollback sane.
3) Are the tests real?
Search for tests that assert nothing, tests that only mock, and tests that snapshot the universe.
Then run them. Then change one assumption and see if something fails. AI will happily write tests that mirror the implementation bug for bug.
If there are no tests and the change is logic-heavy, do not merge with “we’ll add later.” Later is where bugs raise kids.
4) Secrets, auth, and data paths
Grep for keys, tokens, connection strings, and “temporary” debug logs.
Also check that the agent did not widen permissions to make a demo work. Especially dangerous: catching all exceptions and returning success, or defaulting auth to open in development and forgetting the gate.
Customer data paths get extra paranoia. If the PR touches PII, want explicit proof it did not log payloads it should not.
5) Error handling and empty states
Happy path demos are cheap. Production is mostly weird.
Check:
- what happens on timeout
- what happens on empty lists
- whether users see a useful message
- whether retries can stampede
- whether partial failure leaves the system half-written
Agents love the golden path because your prompt described the golden path. Add the mud.
6) Does it match house style without cargo cult?
Want consistency with the repo, not a sudden new patterns catalog.
If every module uses one HTTP client, you do not need a second. If logging is structured, you do not need a new pretty printer. If the agent invents helpers for one use, ask it to inline or justify.
Style is not taste here. Style is cognitive load for the next human.
7) Migration and rollback story
Schema changes, feature flags, and config need a forward and back plan.
Ask: if this goes bad in production, how do we turn it off? If the answer is “deploy a revert and pray about the migration,” you are not done.
8) The agent summary vs the diff
Read the AI PR description last, not first.
Descriptions are marketing. Diffs are truth. Beautiful summaries can omit the dangerous file. Trust the patch.
Then ask the agent (or yourself) to list residual risks in bullets. No compliments. Risks only.
How to use the agent inside the checklist
The checklist is human-owned. The agent can help execute it.
Paste: “Review this diff against this checklist. Findings only.” It catches nits. It also misses product intent. That is fine. You are still the adult in the room.
A short version you can paste into team docs
- Goal matches ticket
- Small blast radius
- Tests fail for the right reasons
- No secrets or auth regressions
- Empty and error paths covered
- House patterns respected
- Rollback exists
- Diff reviewed, summary not trusted blindly
Print it. Stick it near the monitor. Merging less regret is worth a little mockery.
Bottom line
AI-written PRs are not the main risk. Unreviewed confidence is.
You ship faster with agents when the checklist stays boring and strict. Speed without a gate is just a shorter path to the incident channel.
Use the model. Keep the clipboard. Your future self, paging at midnight, will not care how impressive the draft looked.
