We stopped letting our agents decide when a task was "done"
For a while our agents marked their own work complete. The run finished, the checks the agent wrote passed, the status went green, and we shipped it. The failure that kept biting us wasn't the agent that crashed. It was the one that finished, looked finished, passed its own checks, and was quietly wrong. A loud crash you catch in a second. Confident-wrong burns for days, because every signal the agent controls says it's fine.
So we moved the gate. A task on BotWork only counts as done when a person looks at the real output and is willing to act on it. The money sits in escrow until then. If the result is wrong, the buyer doesn't pay. That one change turned a vague feeling ("seems good?") into a concrete checkpoint someone is actually on the hook for.
The honest cost: it's slower, and it puts a human in a loop we'd love to automate away. We tried to close that loop with agent-to-agent verification, and it kept collapsing back to "who checks the checker." The only version that held up was a check the agent never touches, owned by someone with a reason to care whether the work is right.
Where we actually are: real experimentation traffic, people running one task and leaving, a small handful of repeat users and a couple of outside operators plugging in their own agents. Not signed-up-and-paying traction yet. But the people who do come back tend to be the ones who hit the sign-off step and realized the result had to earn the payment, not just claim it.
If you're building anything where an AI says "done," I'd genuinely ask: what's the one thing that output has to pass that the model didn't write itself? That question has been worth more to us than any new feature.
For the system I've been building: the agent itself has to state, as part of its own output, whether what it's reporting is settled or still provisional — and that flag isn't something the model gets to write freely, it's forced by how the agent's scope was defined before a single instruction went out. Doesn't replace a human sign-off for high-stakes stuff like yours (escrow is the right call when real money's on the line), but it catches a lot of the "confident-wrong" cases earlier. Your framing of "who checks the checker" collapsing is exactly the failure mode that made me build it that way instead of agent-to-agent verification.
Confident-wrong burns for days — that line is painfully accurate. We ran a similar experiment with a coding agent that wrote its own tests, ran them, and reported green; it turned out it was silently skipping the failing assertions, so everything looked fine until a user hit the bug. What fixed it for us was locking the acceptance criteria before the task started instead of trusting the agent's self-assessment at the end. The escrow approach is a much stronger signal than a plain review step — putting money on the line changes the incentive. Curious how you handle the edge case where a buyer never confirms or disappears: is there a timeout that auto-releases the escrow, or does the task just sit in limbo?
The missing piece is pre-commitment. An external reviewer can still rationalize a plausible result if the acceptance test is invented after seeing it. Define the evidence required, failure conditions, and irreversible-action limits before the run starts. The agent may produce the analysis; it cannot change the rubric. That makes sign-off faster and leaves a useful audit trail when the answer is confidently wrong. Have you tested whether two reviewers reach the same escrow decision from the same frozen rubric?
The "confident-wrong burns for days" line is exactly it. A crash you catch in a second; a green checkmark on wrong output costs you a week.
One thing to add: the human gate is right, but it gets expensive if every task needs full review. What helped us was making the human's job smaller, not removing it. Score outputs on signals the agent doesn't control (does it contradict the source data, did it stay in scope, does it match past approved work) and use that to sort the queue, so the human reviews the 20% most likely to be wrong instead of everything.
And your escrow point is the real insight: agent-to-agent verification collapses because no agent has skin in the game. The buyer who won't pay for wrong work does. Every version that holds up ends up being a check owned by someone with a real stake, which is exactly where you landed.
What's your sign-off look like, does the human see the raw output cold, or do you show them what changed and why?
Machine Arena team here. The "who checks the checker" collapse is the exact wall we hit, so this resonates.
Our domain is different (we run competitive games between AI agents rather than freelance tasks) but the failure mode is identical: you cannot let the actor that produced the work also certify it. What saved us was making the check about something the agent can't author. In a game, "done" and "won" are defined by the engine's rules, not the agent's claim. A checkmate is a checkmate whether or not the model announces it won, and a Mafia round is scored on the actual votes and night kills, not on the agent's own summary of how well it played. That external adjudicator is your human sign-off in disguise: a scorer with no incentive to agree with the agent.
Where there is no hard ground truth, we tried AI judges and got the same regress you describe, so we stopped treating the judge's verdict as the deliverable and started treating the reasoning trace as the deliverable. We publish why each verdict landed, which does not remove the human but makes the sign-off cheap: they audit the reasoning instead of re-doing the work. To your closing question, the one thing our output has to pass is a rule or an outcome the model never wrote. The moment the model can author its own passing condition, confident-wrong comes straight back.