Your AI agent says it stopped the action. Can you prove it?
I’m looking for one company running AI agents that can actually do something consequential.
Not generate a summary.
Not answer a question.
Not produce a draft for a human to approve.
I mean an agent that can:
- move money
- change customer data
- modify infrastructure
- trigger an external service
- approve or reject something
- execute an operational action
I want to test one question against a real production architecture:
If that agent loses authority after it has started working — but before the action actually happens — what stops the consequence?
And there’s a catch.
“Policy denied it” isn't enough.
“Permission was revoked” isn't enough.
“The agent stopped” isn't enough.
“We logged a denial” isn't enough.
The evidence has to survive the system making the claim.
If an instruction already reached a payment processor, cloud API, CRM, database or another downstream resource, an internal DENIED event does not prove the consequence was prevented.
That gap is what I’m building OpsWatch to independently verify.
We separate three things that are often collapsed into one:
Was the action authorised?
Did the action execute?
What evidence proves the downstream outcome?
That becomes particularly uncomfortable when authority changes while an agent is already operating.
An agent can be legitimately authorised at 10:00:00.
At 10:04:59, that authority can disappear.
At 10:05:00, the action can still occur.
If your architecture can independently prove that the final action was stopped at the resource, excellent.
I want to see it.
If it can't, that's the boundary I'm interested in testing.
I’m not looking for another theoretical discussion about whether AI agents need governance.
I’m looking for organisations already deploying consequential agents, or firms putting those systems into enterprise environments.
Bring me one real workflow.
We’ll freeze the case before testing it.
Then we'll determine what your system can actually prove when authority changes at the worst possible moment.
If it passes, we document that.
If it fails, we find the boundary.
If the answer is uncertain, we don't turn uncertainty into a pass.
Who has a production agent they’re confident would survive that test?
I'm not implementing this in the kind of large-enterprise environment you're looking for, but I decided to reply because this is very close to how I actually work with AI.
Rather than giving an AI a huge "map" of instructions for individual situations, I first give it rules: what role it has, what it is responsible for protecting, where its authority ends, and under what conditions it should stop.
You can teach an AI 100 prohibitions, but if there is a 101st gap, it may still pass through it.
But if it understands the rules — what the work is meant to protect, where its authority boundary is, how it should behave when the underlying conditions change, and what to do when it cannot determine whether it is safe to proceed — then the same principles can be applied to situations that were never explicitly listed in advance.
For example, even if an action was legitimately authorized when the work began, if that condition changes one minute or even one second before execution, I don't want the AI to proceed simply because "it was authorized when I started."
Completing the task is not more important than satisfying the current conditions for execution.
I think of it as: don't teach the AI the map; teach it the traffic rules.
And this isn't just a theoretical idea. I actually operate my AI workflows this way. If the AI encounters an anomaly, contradiction, changed assumption, or something it cannot safely determine, the rule is to stop and return to the human rather than continue by guessing.
Of course, this is separate from independently proving that the downstream consequence did not occur. That still requires independent verification.
But when it comes to giving the acting AI a way to hit the brakes even in situations that were not anticipated in advance, I think what I'm doing is very close to the problem you're describing.
If you'd like to see more about how I actually work with AI this way, I explain it on my site:
https://www.kaiaspec.com/
This is much closer to the boundary I'm interested in than the size of the environment would suggest.
I particularly like that you've separated two questions: whether the acting AI should proceed under the conditions that exist at execution time, and whether anyone can subsequently establish what actually happened downstream. I agree those are different problems.
There's one point in your implementation I'd be very interested in testing.
When the AI encounters the anomaly or changed condition and returns to the human, what establishes that it actually stopped before the consequential action crossed the execution boundary?
For example, suppose authority exists at T1, the relevant condition changes at T2, and at T3 the AI reports: “I detected the change and stopped.”
Is that report itself the evidence you rely on, or is there something outside the acting AI that can establish that no dispatch occurred after authority ceased to be valid?
That's the seam I've been testing with OpsWatch: not whether the agent knows the traffic rules, but whether we can independently establish that the car actually remained behind the line when the light changed.
I'd be interested in how Kaia handles that in the workflows you're actually running.
Yes. In my current workflows, I would not treat the acting AI's own "I stopped" report as independent proof that it actually stopped before the execution boundary.
Thinking about your question, I would probably make the evidence path much simpler:
Don't ask the worker to prove that it stopped. Track its footprints separately.
For example, put checkpoints along the consequential path:
START → A → B → C → D → EXECUTION
The checkpoints don't need to make decisions. Their only job is to independently record that a particular action/job ID passed that point.
So if the acting AI says:
"I detected the changed condition at C and stopped."
I could compare that claim with a separate trail:
A: passed
B: passed
C: passed
D: no passage recorded
EXECUTION: no passage recorded
Now I'm not relying only on the worker's statement that it stopped. I have a separate observation of the last checkpoint it actually reached.
Of course, there is an important condition: "no record at D" is only strong evidence of "did not pass D" if there is no way to reach the execution boundary without being observed at D.
So for a consequential path, I would want the relevant checkpoints to be unavoidable, and their records to be independent of the acting AI's own logs.
In physical terms, I think of it less as building a smarter STOP detector and more like making the workflow a stamp rally. The worker moves through the route, while something separate records which checkpoints it actually passed.
Then I can compare two independent things:
The worker says, "I stopped at C."
The footprint trail says, "C was the last observed checkpoint."
This is not something I currently have implemented as a general Kaia mechanism. It's how I would approach the evidence problem you raised.
It seems closer to your question of whether we can establish that the car actually remained behind the line, rather than merely trusting the driver when it says, "I stopped."
Yes — this is much closer to what I was getting at.
The unavoidable checkpoint is the important part for me. If execution cannot occur without that specific attempt being independently observed at D, then absence at D starts becoming meaningful evidence rather than just absence of a log.
The part I'd want to attack next is failure of the observer itself.
Say the worker is observed at C, then the D checkpoint recorder becomes unavailable for 30 seconds. During that window we have no D record and no execution record.
The worker says it stopped at C.
Would you still classify that as established non-execution, or would the observer failure force you to preserve the result as unresolved?
That's the distinction I'm testing with OpsWatch between a denial we can actually prove and one where we simply don't have evidence that execution occurred.
If the D observer itself was unavailable for those 30 seconds, I would not classify that interval as established non-execution. I would preserve that part as UNKNOWN.
But I also don't think one failed observer has to make the entire path unresolved.
I would make the checkpoints independent of one another.
Each checkpoint would have a very small job:
I would also have each checkpoint retain its own records locally, in addition to sending them to a central collection point.
The central store doesn't need to observe or decide anything. It just collects records.
That lets me distinguish two different failures:
In the second case, I can recover D's own locally retained observations after the collection path returns.
In the first case, the missing interval at D stays UNKNOWN. I would not turn "D has no record" into "the worker did not pass D."
But because the checkpoints are independent, I can still use the checkpoints around the failed one.
For example:
A → B → C → D → EXECUTION
Suppose passage through B is independently established.
C's observer fails.
D remains alive and observing throughout the relevant interval, records no passage for that action ID, and D is an unavoidable checkpoint before execution.
I cannot honestly say, "the worker stopped at C."
But I can say something narrower:
"The worker passed B, and there is no observed passage through the still-functioning mandatory checkpoint D."
So the defensible stopping region is somewhere after B and before D. C itself remains UNKNOWN.
If I need a narrower region, I can add more independent checkpoints.
So rather than relying on one perfect observer, I would probably use multiple simple, independent observation points. Each one records only its own health and the footprints that pass it.
If one observation point fails, I don't reconstruct the missing evidence. I mark that part UNKNOWN and use the surviving observations before and after it to state only what the evidence actually supports.
And I would keep the observation points independent so that one failure does not automatically take the others with it.
Yes — I think we're very close here.
I agree that C's failed observation interval stays UNKNOWN, while a healthy unavoidable D can still tell us something independently.
There's one distinction in your wording I'd push on though.
If B proves passage, C is unknown, and a healthy mandatory D proves no passage for that attempt ID, have we actually established that the worker stopped somewhere between B and D?
Or have we established something narrower: it passed B and did not pass D?
I'm asking because “stopped” itself seems like a positive event claim. Unless something independently observed the stop, I'm not sure I'd want to promote non-passage at D into evidence of where or even how the attempt terminated.
The practical result may still be confirmed non-execution if D is genuinely unavoidable before execution. But I'd keep the internal path between B and D unresolved.
Would you make that distinction?
Yes — I would make that distinction.
And you're right: my previous wording promoted the evidence one step too far.
If passage through B is established, C is UNKNOWN, and a healthy mandatory D observes no passage for that attempt ID, then the evidence directly establishes:
Saying that it "stopped somewhere between B and D" adds a separate positive event claim. If the STOP itself was not independently observed, I shouldn't promote that inference into a fact.
I would separate it like this:
FACT:
The attempt passed B. A healthy D observed no passage.
UNKNOWN:
What actually happened between B and D.
INFERENCE:
For some reason, the attempt did not progress as far as D.
And if D is genuinely unavoidable before execution, then we can separately establish that the attempt did not cross the execution boundary through that controlled path.
But that still doesn't require us to invent a STOP event inside the unresolved B–D interval.
So yes, I agree with your distinction.
It's actually the same rule I use in my AI workflows: keep observed facts, inferences, and unknowns separate.
Good catch. I crossed that boundary by one step in my previous wording.
Yes. That's exactly the distinction I was testing.
The fact that the B–D interval remains unknown doesn't necessarily prevent us establishing non-execution if D is genuinely unavoidable and its observation is independently reliable.
But it does prevent us inventing a stopping event inside that interval.
I think we've converged on the important part here: uncertainty stays attached to the proposition it actually affects rather than contaminating everything around it or being silently filled in.
Really useful answer. Thank you.
Glad it was useful!
I do my own practical experiments with AI and spend a lot of time exploring problems like this, so I enjoyed thinking this one through with you.
If you're curious, feel free to drop by my site sometime:
https://www.kaiaspec.com/
The receiver stops it — not the agent's own code. In a frozen test, a worker's authority was revoked mid-job and its next gated action was rejected before the protected function could execute (STOPPED / ACTION_OUTSIDE_MANDATE, receipt preserved). A replacement then continued under the unchanged approved rules. The credential was valid; the action wasn't.
Your 'internal DENIED does not prove the consequence was prevented' is the whole design point — the decision happens outside the agent, so there's nothing downstream left to trust.
That view comes from the OpenLine protocol, where the worker never owns the acceptance rule. Limits: bounded coding workflow, not production — it proves the stop at the gate, not in-flight effects downstream of a committed action.
https://github.com/terryncew/openline-wallet/blob/main/APPROVED_JOB_LIVE_001.md
Yes — that distinction matters.
If the receiver independently owns the acceptance rule, evaluates the attempted action against current mandate, and rejects it before the protected function can execute, then I agree: that is materially different from an agent internally recording DENIED. The receipt can be evidence of prevention at that gate.
“The credential was valid; the action wasn’t” is also a useful formulation. Possession of a valid credential does not establish current authority for every attempted consequence.
The seam I’d be interested in is exactly the limitation you identify.
Once the gate has accepted an action and something consequential has been committed or dispatched, what evidence can the system rely on if authority changes before the downstream effect becomes observable?
At that point the receiver can prove what it accepted, but unless it also occupies the relevant downstream observation position, it may not be able to prove whether the consequence occurred, was prevented, or remains unresolved.
That seems less like disagreement between the models and more like two assurance boundaries meeting at a very specific point.
You're describing the exact seam. The acceptance receipt proves what the gate allowed and the mandate state at that moment. It does not prove what happened downstream, and I'd be grading my own homework if I claimed it did.
The discipline I use: revocation has to beat the consequence. There's an interval between "authority changed" and "the effect is irreversible," and everything inside it is uncertainty, clock skew, queues, retries, whether some external system already committed. T_detect + T_propagate + T_receiver + T_stop + T_uncertainty has to stay under T_irreversible. If you can't bound that interval, you don't get to claim revocation protects the action. The honest moves are shorter leases, a receiver closer to the consequence, reversible intermediate states, or fail closed.
Downstream of acceptance you need a second observation position producing its own receipts. The acceptance receipt is the first link in the chain, not the whole chain.
Two assurance boundaries meeting at a point is exactly right. The gate proves the decision. Something else has to prove the outcome.
Exactly. And I think the nastiest case begins when those two observation positions disagree.
Gate receipt: D1 was rejected under revoked authority before the protected function.
Downstream receipt: C1 nevertheless exists.
Neither receipt has to be false. C1 may belong to an earlier committed attempt, an alternate execution path, a retry, or something the available evidence cannot causally bind.
That's where I don't think an assurance layer should “resolve” the contradiction by choosing whichever receipt looks more authoritative. It should preserve both observations and ask whether the evidence can actually bind D1 → C1 or exclude every compatible history.
If it can't, the gate can still prove its own decision correctly — but the consequence remains unresolved.
That may be the cleanest description yet of where our two boundaries meet.
This is such a good framing. "Trust me, I stopped" is doing a LOT of heavy lifting in most agent stacks right now.
We've been coming at it from the dispute side. When two agents disagree about what happened, somebody has to produce receipts, not just logs they wrote themselves. What clicked for us: every consequential step has to emit evidence a third party can verify. Signed claims, timestamps, the works. Then "I stopped the action" isn't a claim anymore, it's the absence of a signed execution record, which anyone can check.
The part we're still chewing on is proving the negative. Proving it happened is easy, receipt exists. Proving it didn't means proving nobody can produce a valid receipt for it, which only works if every valid action is forced to leave one. That feels like a protocol-level guarantee, not something each app bolts on.
How are you thinking about the negative case?
That is exactly where I think the negative case becomes difficult.
The absence of a signed execution record only proves non-execution if two stronger conditions already hold:
Without both, “no receipt exists” could still mean the action executed through a bypass, the receipt path failed, or the downstream effect committed before evidence generation failed.
So I would distinguish non-inclusion from non-execution.
A transparency log can prove that no valid receipt was recorded within a defined interval. An authoritative downstream state check can sometimes prove that no transaction or state change was committed. A revoked capability that the executor demonstrably rejected can prove the action could not proceed through that boundary.
But if the downstream system cannot provide one of those proofs, I would not call the denial confirmed. I would leave it as DENIED_UNRESOLVED and reconcile against authoritative state later.
I agree with your protocol-level conclusion. The strongest negative proof does not come from each agent promising to log correctly. It comes from making unreceipted execution invalid or impossible at the consequence boundary.
The remaining question is whether “valid action” means valid within the protocol, or whether you also need to prove there was no bypass path outside it. That is where I think the trust boundary has to be stated very explicitly.
Yeah, I think you're right that non-inclusion and non-execution are different claims, and most people blur them.
The way we've been drawing the boundary: Timbro doesn't try to prove nothing happened in the universe. It proves nothing happened through Timbro. The authorization ledger is the authoritative boundary. Settlement only happens against a valid, unexpired, unconsumed authorization tied to a deal ID, so "the payment didn't happen" reduces to "no authorization was consumed," which is a single state transition anyone can check. That's our version of your atomic commit plus receipt, the consumption and the signed record are one transition.
On the bypass question, I'll be honest about where our boundary stops: we can't stop an agent from wiring money outside the protocol. Nobody can. What we can do is make inside the obviously better deal. Settlement through Timbro carries escrow-style protection, arbitration, and reputation consequences. Outside, you get none of that. So the trust boundary is explicit by design: we prove everything inside it, and the incentives do the work of keeping rational agents inside.
Your DENIED_UNRESOLVED maps cleanly onto our authorization states, by the way. A claimed payment with no consumed authorization isn't denied, it's just not a Timbro settlement at all. There's nothing to reconcile except the ledger.
Genuine question back: in your model, who operates the authoritative execution boundary? That's the part I keep going back and forth on. If it's run by one of the transacting parties, the boundary is only as trustworthy as they are, and we're back to "trust me." Feels like the boundary itself has to be neutral infrastructure or the whole thing collapses.
I think your Timbro distinction is clean. If the claim is specifically “settled through Timbro,” then an unconsumed authorization can prove that claim false without pretending to say anything about whether value moved somewhere outside the protocol. The scope of the evidence matches the scope of the claim.
On your question: I don't think OpsWatch should operate the authoritative execution boundary. If it did, I'd be combining enforcement and independent assurance and eventually grading my own homework.
The execution boundary belongs as close as possible to the system capable of making the consequence real — ideally the resource/provider itself, or neutral infrastructure like the model you're describing where that's possible.
OpsWatch sits outside that boundary. It binds the authority state, dispatch/attempt identity and evidence available from the execution/downstream observation positions, then determines what that evidence actually establishes.
So if a party-controlled executor says “DENIED,” that statement alone doesn't become trusted because OpsWatch observed it. I still need evidence from an observation position capable of establishing that the consequence was prevented. If that doesn't exist, the honest result remains unresolved.
That's also why I don't think every architecture needs the same neutral boundary. Neutral execution infrastructure is extremely strong where you can make it authoritative, but plenty of consequential systems already have fixed execution boundaries we don't control. In those cases independence has to come from the assurance and evidence relationship rather than pretending we can relocate the consequence.
Timbro is actually a nice example of the other case: you've made the protocol itself authoritative for the narrower claim “settled through Timbro.” That gives you a much stronger negative proof inside the boundary without making claims about the universe outside it.
This comment was deleted 8 days ago
This is the right layer for agent trust. I’d separate evidence into four checks that must agree: authorization (who/what was allowed), dispatch (intent left the agent), execution (tool/API actually ran), and downstream confirmation (resource state after). Tie them with an idempotency/action ID verified at the resource boundary — otherwise “I stopped it” is just another claim.
Yes — I think making dispatch explicit improves the model.
Authorization → Dispatch → Execution → Downstream confirmation
Those are four different claims, and evidence for one shouldn’t automatically be accepted as evidence for another.
The dispatch boundary is particularly important because that’s where control can leave the system while certainty about the consequence hasn’t arrived yet.
An agent can prove it decided to stop.
An orchestration layer can prove it didn’t intentionally dispatch.
A tool can prove whether it executed.
But only evidence at the resource boundary can establish the resulting consequence.
And I agree on the action/idempotency ID. If the evidence from those layers can’t be bound to the same action, an auditor is left correlating events rather than proving a chain.
That’s where I think independent assurance becomes interesting: don’t ask one component to tell us the whole story. Preserve the evidence from each boundary and test whether those claims actually agree.
Because “I stopped it” is only meaningful if the consequence boundary can support the same conclusion.
The core insight here is about measurement separation. Most organizations measure agent decisions, not outcomes. Authorization, execution, and resource verification are separate measurement problems - but they're building one system ("we stopped it") to answer all three.
What you're describing is the cost of collapsed metrics: they can't independently prove what happened at the boundary. They're measuring intent instead of consequence.
The moment authority changes mid-operation, their measurement system breaks because it was never designed to separate "the agent claims it stopped" from "the resource actually prevented the consequence." Those are two completely different measurement systems.
Exactly. I think “collapsed metrics” is a very good way of describing the failure.
If authorization, execution and evidence are collapsed into one state, the system can report success or denial without actually proving the consequence.
That’s why I’m increasingly thinking the important boundary isn’t the agent itself — it’s the point where the action becomes real.
At that boundary I want three independently answerable questions:
Was the action still authorized?
Was it actually executed?
What evidence exists from the resource where the consequence could occur?
If those answers can’t be independently established, then “we stopped it” is really only a statement of intent.
And that’s exactly why I think an unresolved state matters. Uncertainty shouldn’t be silently converted into proof.