I have one AI write my code and a second one review it. Yesterday the reviewer failed 2 changes out of 5. When do you stop and drop a change

AI integration changed my product completely. Not the 'AI-powered' marketing buzzword — actual useful automation like auto-categorizing support tickets, generating follow-up emails, and scheduling based on natural language. Small things, massive time savings for customers.

I'd separate what the failure is about. If the reviewer flags a real defect, like the hole you wouldn't have spotted, that's worth another round. If the same item fails twice, or the failures are about wording, the checklist is probably the problem, not the change.

yo cree una tambien uno que lo creaba el codigo otro lo rommpia cuando tuviera oportunidad(90% de veces que lo rompre) lo vuelve a corregir y luego lo prueba

Your 5/6 and 6/10 can partially answer this. One item missed out of six is a line away from passing, so retry is cheap. Four missed out of ten usually means the instruction was unclear, and retrying an unclear instruction just buys a different wrong answer

The thing I'd add to the "route by failure type, not round count" thread here: after the second failure, write the failing criterion down somewhere permanent instead of just deciding ship-or-drop. I kept a one-line ledger on a side project for about six weeks (change, which numbered rule failed, what I decided) and roughly a third of the entries were the same three rules over and over, which told me the fix wasn't more rounds, it was converting those three into actual tests or lint rules the builder physically can't pass without satisfying. The other thing that made dropping cheap for me was reversibility rather than confidence: anything behind a flag or in a single revertable commit ships after one failed round and production becomes the reviewer, while anything touching billing or webhooks gets zero rounds and comes straight to me. Your Sonnet-builds/Opus-reviews setup also raises the shared-context problem - when the reviewer reads the same instruction file the builder worked from, an ambiguous instruction gets rubber-stamped in both directions, which may be exactly why your failures cluster on instruction changes rather than code. Do you have a read on which numbered rules produce most of your failures, and would you let the reviewer propose edits to the instruction list itself instead of only judging code against it?

My rule after burning too many hours on this: one round to fix, and the second round has to change the review criteria, not the code.

If the reviewer fails a change twice with the same criteria unchanged, the problem is rarely that the writer just needs another attempt. It's usually one of three things: (1) the criteria in the numbered list are ambiguous and both models are interpreting them differently each pass, (2) the change genuinely can't be made without touching something the criteria don't cover, or (3) the reviewer is hallucinating a failure (surprisingly common on numeric or async code).

So the second round for me is a rewrite of the criteria: split the failing line into two more specific ones, add an example, or add an anti-example ("the fix must NOT introduce X"). If that second round still fails, drop it and move it to a manual queue. Third rounds on the same criteria have never once produced a working result for me — they just produce more plausible-looking code with the same bug.

On your 2 out of 5 yesterday: worth eyeballing whether the two that failed share a shape (both touched the same file? both async? both involved a specific API surface?). Failure clusters usually point at a criteria gap, not a writer problem.

I’d stop using review failure alone as the threshold. A two sentence change with a big blast radius deserves more scrutiny than 100 lines touching nothing important. That’s basically the idea behind PRI: prioritize review based on production risk, not diff size.

Two rounds are a reasonable limit, but I'd treat a failed review differently from an unclear request. For a small instruction change, write down exactly how the behavior should change. If neither model can identify the failed condition, another pass will probably add confidence, not evidence. That's when I'd stop and decide myself.

Running apps with a full time job means time is very valuable. If a small change fails two rounds, I just drop it or fix it myself because spending a whole morning on a tiny text fix defeats the speed of AI.

The variable I'd track is not rounds, it's blast radius. At Henson Group we let low-risk changes ship on one pass and forced a human read on anything touching auth, billing, or customer data, because a second round on a cosmetic fix costs more than the fix is worth. Try timeboxing instead of counting rounds: if a change eats more than 20 minutes total, it goes back to the spec, not back to the reviewer.

I think the stopping point should depend on the type of failure. If the reviewer misses a cosmetic issue, I might accept the change. If it misses anything related to payments, authentication, or user data, I would stop and review manually. Using two models creates useful disagreement, but it doesn’t necessarily create reliable verification.

Two rounds is a reasonable ceiling, but the cheaper filter is whether the failure is in the spec or in the code. If the reviewer fails the same category twice, that is a spec problem and more rounds will never fix it, so rewrite the instructions instead of re-running them. The rounds that actually pay off are the ones where the model got the judgment wrong, not the ones where you did.

This resonates. We hit the same wall with client sites — AI-generated code that technically works but breaks something subtle in production later.

The rule that's worked for us: if the second failed round is still failing on the same category of issue (not a new one), we drop it and do it manually. If it's failing on something new each time, that usually means the original spec was too vague, not that the AI is bad — so we go back and rewrite the instructions before trying again.

Two rounds sounds right as a ceiling. Past that, the time cost usually isn't worth it for anything under a few hours of manual work.

Curious how you're structuring the numbered list you mentioned — do you write it per-change or do you have a reusable checklist per project?

I’d treat the two-round rule as a time budget rather than a strict retry rule. Before sending a failed change back, I’d ask one question: “What new information will the next round have that the previous round didn’t?”

If the answer is nothing, I’d stop. If the failure exposed a real gap in the requirements, missing context, or a concrete edge case, then another round makes sense because the input has actually changed.

I also like the idea of tracking these decisions for a while. After 20–30 changes, you could see whether second-round fixes actually prevent future problems or whether they mostly polish low-impact issues. That would give you a much better stopping rule than simply choosing two rounds upfront.

I think there’s a useful distinction between “the change failed review” and “the change isn’t worth doing.” A reviewer can be completely right about a problem while the fix still costs more than the value it adds. For small solo projects, I’d probably treat repeated failures as a signal to reassess the original change rather than automatically trying to make it pass.

Stop counting rounds and look at whether the two failures named the same numbered rule. If both fails cite the same line in the list you agreed before the work started, the writer is not going to land it. Edit that one rule by hand and ship. If the two fails cite different rules, the change is too wide for the review. Split it until one assertion fails or none do. That is what keeps the two-round cap from eating the morning.

Drop the change when you cannot restate the failing rule in one sentence without opening the diff. If you cannot say what broke, another model pass is just rent.

Two rounds is a useful budget, but I would route by failure type before counting rounds. A reproducible defect gets another attempt; an ambiguous acceptance criterion sends me back to rewrite the spec; a taste disagreement goes to a human. The practical test is whether the reviewer can name the violated line and a concrete failing case. If it cannot, another pass is mostly a reroll. Keeping a small log of failed criterion -> final human decision would also tell you after a month whether two rounds is actually the right cap.

I’ve found the same problem with AI agents. A failed attempt can actually be valuable because it often reveals something I missed. But at some point you have to recognize that you’re spending more time fixing the agent’s approach than solving the original problem.

I’m still trying to figure out where that line is.

A fail is a signal, not a stop. Split the numbered list into must-pass vs nit. Must-pass is what you'd be sorry about later: data loss, wrong behavior, the hole you said you wouldn't have spotted. Tone and wording nits go on a short human pass so a 5/6 doesn't kill a change that was already fine.

Two rewrite-and-review cycles, then stop the loop. Either drop that slice and ship what already passed, or take the failed lines yourself with a smaller prompt. Sending the whole change back is how a two-sentence edit eats a morning.

What were the two instruction misses, specifically? A real hole vs a wording disagreement is usually the whole drop-vs-keep call.

I think the type of failure matters more than the number of retries. If it’s a real bug, another round makes sense; if the spec itself is unclear, it’s probably better to fix the requirement first. Otherwise, it’s easy to get stuck in an AI review loop.

The round count isn't what I'd track - the class of failure is. If the check is something code can decide (schema, types, a test), another round usually converges, so it gets one. If it failed on taste or product intent, another round rarely moves it, so it comes to me.

The part that'd bug me isn't the two-round cap itself, it's that you don't have anything telling you whether two rounds is even the right number. If you never spot-check a handful of these yourself — read the actual diff on a change that passed clean, or one that got dropped after two failures — you're trusting the reviewer's calibration without ever independently checking it. Two AIs agreeing "this is fine" isn't the same as it being fine; it's just two models agreeing, which is closer to the reviewer grading its own homework than an actual check. Might be worth manually auditing 1 in 10 passes, just to see if the rule is catching what you'd actually want caught.

I use a hard stop after 2 review rounds on the same change. If the reviewer still fails the same acceptance check, I drop or rewrite that slice instead of retrying. Numbered criteria help, but looping the same failing item usually just burns tokens.

The failures landing on instruction changes isn't a coincidence — it's the signal. A change to written instructions fails review because the instructions were ambiguous, not because the execution was wrong; so a retry means running the same ambiguity through the same checklist again, which is why round two rarely helps there. Worth splitting the outcome three ways before deciding: spec-gap (fix the checklist — the change becomes a new change and the timer resets), execution-error (genuinely worth a retry), edge-case (add it to the checklist as a new line). Only one of those three is a retry. Your two-round rule still holds — it just shouldn't count spec rewrites as rounds.

the fact that your failures were on instruction changes, not code, is the interesting part. means your review criteria is actually working. for the drop-vs-retry call, I'd weight it by blast radius: user-facing flows always get a retry, internal docs or config get dropped. we see the same pattern assessing AI verification habits at aisa.to — the skill isn't knowing how to check, it's knowing what's worth checking.

The round count may be the wrong variable. What predicted "another round will help" for me was the kind of claim that failed.

Three kinds, and only one of them is worth a second round:

Claims about your own intent — the spec. More rounds don't help, as the comment above says. Rewrite the list.

Claims about the outside world — does this API exist, does this platform actually behave this way. Don't review these, verify them. I lost most of a morning this week to one: I fixed a "nothing happens when you tap the button" bug by routing the call through a method on a platform SDK. It compiled, it ran, and it did nothing, because that method was removed two major versions ago. My code caught the failure and fell through to a fallback, so it looked fine. No reviewer catches that. Opening the package's type definitions catches it in twenty seconds.

Claims about behavior you can't reproduce — a specific device, a store's throttling, a quota. Neither review nor rounds resolve these. The only move is to stop being clever and take the deterministic path.

That last one produced both of the real bugs I shipped this week, and they had the same shape: a call that reports success when nothing happened. The rating API returned "true" because it didn't throw, while the OS had silently decided not to show anything — so the fallback that would have saved it never ran.

The inverse is worth budgeting for too. Yesterday I wrote up a defect I was sure of, and a comment in the same file refuted it — the platform's limit was on displays, not calls, so the thing I "found" wasn't a thing. Reading the file's own comments before proposing the change would have cost two minutes.

So: before spending a second round, ask whether the claim is checkable outside the model. If it is, check it — that's cheaper than a round. Rounds are for judgment, and judgment is the smallest part of what fails.

(Context for the examples: I build SIGNUM HQ, a free US-market data app — options flow, dark pool share, GEX, max pain. iOS and Android: https://www.signumhq.com/app?from=indiehackers )

The most useful gate for me is whether a failure is actionable and observable: does the reviewer quote the violated acceptance criterion, point to a reproducible test, and distinguish spec ambiguity from an implementation defect? If a second pass gets the same evidence, drop the change or rewrite the requirement; if it exposes a new failing path, keep it. That makes the two-round budget a decision rule, not just a retry count.

Both failures landing on written instructions rather than app code is the part I would act on first. If the reviewer scores a change against a numbered list, and the change being scored is itself a change to that list, then the thing under test is your spec, and no number of writing rounds fixes a spec problem. What helped me was moving the review earlier: before anything gets written, hand the reviewer only the numbered list and ask it where the list is ambiguous. That catches the five out of six kind of failure while it still costs two minutes instead of a morning. Your two round rule can stay exactly as it is once the spec has survived a pass on its own.

What decides it for me is whether the objection names something I can go and look at: a file, a line, a value, a case it claims will break. Then another round is worth it, because the second pass has something concrete to check itself against, and it either holds or it does not.

If the objection is a judgment, which is most of what you get on prose rather than code, the second round just rerolls the dice. Two models will keep splitting on the same line until the line changes. That is DannieDan's underspecified-spec point from the other end.

Does yours quote the line it failed on, or just give the verdict?

The five-out-of-six instruction failure is a useful distinction from a six-out-of-ten miss. I would set the retry budget by blast radius and by whether the failure exposes a missing rule, then rewrite the numbered list before rerunning it. Do you track which failed changes later become bugs or support issues?

The thing I would change first is not the number of rounds, it is what the reviewer hands back. Right now a fail is one bit of information, and you are trying to make a judgment call out of one bit.

I spend a lot of my week around human code review, and the split that actually drives the decision is the reason, not the count:

Spec was ambiguous. The reviewer is really flagging your numbered list, not the change. A second pass against the same list fails the same way, so the work is fixing that line in the spec.

Writer was missing context. It never saw the file or the earlier decision. Cheapest retry there is, and the one with the highest hit rate.

Real defect. The hole you would not have spotted. Always worth another round.

Reviewer taste. It failed something your rubric does not actually require. That is not a failure, it is rubric drift, and it quietly inflates your failure rate until you fix the rubric.

The rule that falls out of that: never retry with the same inputs. If nothing changed between attempt one and attempt two except sampling, you are not iterating, you are rerolling the dice. Since your reviewer is already pass or fail on the whole thing, make it print the category and the exact spec line it violated. The triage becomes mechanical instead of a feel, and the two-sentence change stops eating a morning.

One cheap addition: keep a one line log of what you dropped, then note later if any of it came back as a bug or a support ticket. After a month that tells you whether two rounds is too generous or too tight. Right now that number is a guess, for you and for everyone answering you.

The bit that stuck with me is the hole you never would have spotted, sitting inside a failed instruction change, not the app. That one already paid for the review, even if the score was only five out of six. For the two-sentence change that eats a morning, I would stop the second you can describe that hole yourself. Another model pass after that is just credits.

One rule I use with AI builders that makes failed reviews easier to triage: make the writer read the existing code first and make only minimal edits. Then most fails are either scope drift or a real logic miss. Drift is cheap to fix by shrinking the diff instead of running another full round, so I wouldn't count it against a two-round cap.

Both failures were on written instructions, not app code. If the writer and the reviewer split on the same numbered line, another round is usually two models arguing over an underspecified spec. I'd drop that change, rewrite the one line they disagreed on, and give the writer a single new pass. Keep the two-round cap for Shopify code, where a miss is a bug. For the prompt, the split itself is the review.

The judgment problem is the hardest part of any AI-assisted workflow. The two-round rule sounds clean, but you're right that it breaks down when failure severity varies wildly. One heuristic that might help: weight by blast radius. A failed change to a shared utility or auth flow gets more rounds than a cosmetic fix. For solo Shopify devs, separating "did it break something real" from "did the AI miss a nuance" might save you from over-investing in low-stakes rollbacks.

The two-round rule is simple, but the failures have different consequences. Are you finding that the decision to retry should depend on the change’s impact, not just the review count?