our AI agent kept dying at the finish line. the bug was a safety rule we wrote ourselves
we run a browser agent that fills job application forms. after it clicks submit, a verifier reads the page to confirm the application actually landed, because "form filled" and "application received" are different claims.
to prevent double-submissions we gave the submit step a hard rule: click the submit button exactly once, never click anything else, budget of 3 actions. tight, safe, deterministic. felt responsible.
then we audited why some applies died exactly at the finish line. some ATS platforms open a confirmation dialog after you click submit: "are you sure? confirm submission." our agent stood in front of that dialog, forbidden by our own rule from clicking the one button that would finish the job, and timed out.
the safety rule was the bug.
the fix wasn't removing the guardrail. it was making it less deterministic: a confirm dialog on the same page is part of the same submit, a navigation to a new page is not, never click submit itself twice, and a slightly bigger step budget for what we didn't foresee.
the lesson we keep re-learning with agents: every hyper-specific rule you write is a bet that you've seen every case. on the open web you haven't. the rules that survive are about direction ("never resolve ambiguity toward claiming success"), not mechanics ("click exactly N times").
curious where others building agents draw this line. what's your boundary between guardrails and agency?
The confirmation dialog also changes the state you’re acting on. Before allowing that second click, I’d re-check that it’s still the same application, same employer and same payload—not just the same page.
That keeps the rule flexible without turning “confirmation dialog” into a blanket exception.
This is exactly the distinction I’ve been working around with OpsWatch — the system saying it completed an action isn’t necessarily evidence that the intended outcome occurred.
What gets especially interesting is when the implementation team’s own telemetry is also being used to validate the implementation.
Have you considered having a small sample of production runs independently evaluated against the intended outcome and external evidence rather than the agent’s own completion state?
i'd draw it at invariants vs procedures. keep invariants hard: don't claim success without evidence, don't submit twice, don't cross a permission boundary. action counts and exact click sequences need room to adapt.
also log why a guardrail fired. otherwise safe failures and agent failures look identical in evals.
invariants vs procedures is the cleanest wording for this split i have seen. our bug was exactly a procedure cosplaying as an invariant: a step budget written like a law, when the real law was do not spend without progress. the fix was making the guard progress-aware, it only kills when spend rises while progress has flatlined for several consecutive steps, and the finish line deaths stopped without loosening anything that mattered. fully agree on logging why a rule fired. we went one further and gave guardrail kills their own terminal codes, so evals chart them as a separate series from genuine agent failures, and that split showed within a week which rules earned their keep. the follow-up your framing raises: do you eval the invariants themselves? a false-positive invariant looks exactly like safety working, and nothing in the logs argues with it unless you sample those kills by hand.