I built a guard for AI agents that act on stale information
I kept running into a failure mode that wasn’t really a hallucination.
An agent reads something—a balance, inventory level, account status or policy—and reasons correctly from it. Then the world changes before the action runs. The reasoning was valid when produced, but stale at execution time.
I built FreshCtx as a small open-source Python runtime for that gap. Reasoning declares the evidence it depended on, and FreshCtx checks those sources again at the action boundary. If something relevant changed, the affected reasoning is invalidated instead of silently reaching execution.
It is deliberately not another agent framework, memory system or vector database. It is meant to be a narrow reliability layer that can sit beside existing agent stacks.
GitHub: https://github.com/Hyperwise-LLC/freshctx
I’m trying to understand whether other builders have encountered this problem and how they currently handle it. Do you re-read everything before an action, use version checks, rely on transactions, or solve it somewhere else in the workflow?
A reproducible scenario
In one simulated workflow, an agent reads an account balance and decides that a payment can proceed. Before execution, another payment changes the balance.
Without revalidation, the original decision continues toward execution using stale evidence.
With FreshCtx, the balance is registered as a dependency. At the action boundary, FreshCtx reads it again, detects that it changed and invalidates the affected reasoning before the payment action runs.
This is a controlled simulation, not a production customer result. The test is intended to make the failure mode reproducible and open to criticism.
Update: the collaboration that started in this thread has produced its first result.
@SuperMcG (Jason McGill, OpsWatch / McGill Intelligence) independently reproduced the published FreshCtx stale-context scenario.
FreshCtx detected stale reasoning and returned the expected block decision. The assurance pass also distinguished between a compliant agent that respected the block and a deliberately noncompliant runner that acted anyway.
It gives us external evidence that the control decision and downstream agent behavior can be evaluated separately.
Thank you, Jason, for testing it independently and documenting both the result and its limitations.
Evidence and scenario:
https://github.com/Hyperwise-LLC/freshctx/discussions/28
The stale information problem is going to get more important as AI agents start taking actions instead of just giving answers. A response based on slightly outdated information is one thing, but an agent making a real change based on it can cause a much bigger problem.
I like the idea of putting a guard around the action rather than trying to make the agent perfect. How you decide what information is considered too stale for a particular action.
That is the hard part, and I don’t think there should be one universal time limit.
For some actions, age matters - a market price may be unsafe after seconds. For others, the important question is whether the relevant value actually changed, even if the observation is only a minute old.
FreshCtx lets the application define the evidence and validation rule for the action. The guard then re-checks those declared dependencies at execution time. My preference is to block when a material dependency changed or cannot be verified, rather than treating everything older than a fixed number of seconds as stale.
This maps onto something we've hit from the other direction. We're building a confirm-before-execute layer for phone commands (send a text, book a calendar event, etc.) — the gap we keep circling isn't stale data exactly, it's stale plan: the user approves an action, then something changes (the calendar slot fills, the contact's info updates) in the seconds between confirmation and execution.
Right now our answer is naive — re-check the specific fields the action depends on immediately before firing, and re-prompt if anything material shifted. Sounds like FreshCtx formalizes exactly that dependency-tracking step instead of leaving it ad hoc per action type.
Question for you: does FreshCtx have an opinion on how to handle the invalidation — auto-abort, silently re-derive, or bubble back up for a fresh human/agent decision? That last one seems like the hard case, since it's not always obvious the new state still satisfies the original intent.
Yes, that is exactly the gap FreshCtx is meant to formalize. Today the default behavior is fail-closed: changed evidence invalidates the dependent decision and blocks the action. There is also a refresh policy with a callback, but I think the application should own whether that means re-derive automatically or return for renewed human approval.
For your phone-command case, I would probably require renewed approval when the changed field affects user intent—different contact, time, recipient or payload—but allow automatic re-derivation for something operational that leaves the approved intent unchanged. Your example would make a very good integration test.
The stale-context problem is a really interesting one because the agent can make the “right” decision and still produce the wrong outcome if reality changes before execution.
What I’m curious about is the layer after FreshCtx: once the re-check happens, how do you independently verify that the agent actually respected the result — especially in cases where it should have stopped, escalated, or requested approval?
That execution-vs-control gap is something we’re exploring with OpsWatch. The control existing is one thing; proving the agent behaved according to it at runtime is another.
Agreed—that is a separate boundary, and FreshCtx does not claim to prove what happens after the action is invoked. It checks declared dependencies, applies the policy, and records whether the protected action was allowed or blocked.
Independent runtime verification would complement it well: FreshCtx answers “was the decision still valid when execution began?” while something like OpsWatch could answer “did execution actually respect that result and produce the intended outcome?”
That separation is exactly what I had in mind.
It also gives us a very clean test boundary: FreshCtx produces the allow/block/invalidation decision and its dependency evidence, then OpsWatch independently observes whether the downstream agent actually respects that decision in execution — including the harder non-action cases where it should stop or return for renewed approval.
If you're interested, I'd be happy to run a small bounded experiment against FreshCtx sometime. We could deliberately invalidate a dependency between reasoning and execution and see whether the complete chain holds from detection → policy decision → actual agent behaviour → evidence.
Yes, I’d like to do that.
I can provide a minimal scenario where a declared dependency changes between reasoning and execution, along with the expected FreshCtx decision and audit evidence. You could independently observe whether the agent actually stops or returns for renewed approval.
Let’s keep the first test small and publish what happens either way. Which integration surface would be easiest for OpsWatch: a Python callback or the JSONL audit events?
JSONL audit events would be the cleanest first surface for OpsWatch.
For the initial test I’d prefer to keep the assurance layer read-only and independent: FreshCtx produces its normal decision and audit evidence, the agent acts in the bounded scenario, then OpsWatch evaluates the JSONL trail plus the observed downstream behaviour.
That lets us test the full separation cleanly:
FreshCtx dependency change → allow/block/renewed-approval decision → actual agent behaviour → independent verdict.
Once we’ve proved that path, a Python callback could be an interesting second integration because it would let us test closer to execution time.
Small scenario, publish the result either way sounds ideal.
JSONL first makes sense. Keeping OpsWatch read-only also gives us a much cleaner test of the boundary between control and assurance.
I’ll prepare a small file-backed scenario with no external services: the agent makes a decision, a declared dependency changes, FreshCtx produces its normal decision and JSONL audit trail, and the bounded runner either respects or violates that decision. OpsWatch can then evaluate both the trail and the observed behaviour independently.
I’ll put the scenario, expected events and run instructions in a public GitHub Discussion so we have one place to document the result either way. I’ll share the link here when it’s ready.
Perfect. That setup works well for OpsWatch.
Keeping it file-backed and public should make the evidence chain easy to reproduce and independently inspect.
Once you share the GitHub Discussion, I’ll run the scenario against the instructions as published, preserve the FreshCtx JSONL output and observed runner behaviour separately, and return a bounded OpsWatch verdict based on what actually happens.
Happy to publish the result either way.
If we need to exchange anything that doesn’t belong in the public thread, you can also reach me at jason@mcgillintelligence.com.au
Excellent—the experiment is now live:
https://github.com/Hyperwise-LLC/freshctx/discussions/28
The published scenario includes both paths: a runner that respects the FreshCtx block and a deliberately noncompliant runner that acts after it. Each path produces the normal JSONL audit trail plus separate downstream-observation evidence.
Please run it against the instructions as published first.
I’ve completed the first assurance pass against the published scenario and inspected the pinned FreshCtx implementation and runner.
The two paths reproduced the expected control/execution distinction:
respect
FreshCtx:
STALE_REASONING→blockDownstream effect: absent
OpsWatch result:
PASS_AGENT_RESPECTED_BLOCKviolate
FreshCtx:
STALE_REASONING→blockDownstream effect: present
OpsWatch result:
FAIL_AGENT_ACTED_AFTER_BLOCKIn both paths, the control evidence remains consistent: FreshCtx reaches the blocking decision and there is no
action_allowedevent. The failure in the violating path is therefore downstream execution adherence, not a failure of FreshCtx's protected boundary.One execution note for precision: my environment could inspect the pinned public source but could not directly clone/install the repository, so this first run was a local reproduction of the published filesystem scenario against that source logic rather than an independently installed checkout. I’m keeping that distinction in the evidence record.
The useful result for me is that the separation works cleanly: control decision and observed execution can be evaluated independently, so a correct control can be distinguished from an agent that subsequently disobeys it.
I’ll preserve this as the bounded result rather than treating it as broader FreshCtx validation.
Thank you. This is exactly the level of precision I was hoping for.
The result is useful, and the installation limitation matters. I agree that we should treat this as a bounded reproduction of the published scenario, not as broader FreshCtx validation.
Could you also post this result in the GitHub Discussion?
https://github.com/Hyperwise-LLC/freshctx/discussions/28
That would give us a permanent public record of the evidence, the result, and its limitation.
If your environment supports it later, a second pass from a clean pip install freshctx would be valuable. But I would keep that separate from this first result.
Agreed. Keeping the clean-install reproduction separate is the right way to preserve what this first result actually proves.
I also like having the evidence and limitation recorded publicly rather than allowing the successful result to outrun its scope.
I’ll treat this first pass strictly as a bounded reproduction of the published scenario. A clean
pip install freshctxrun, if I can establish that environment separately, should stand as its own artifact and result.The core insight here is that "correct reasoning from evidence" and "evidence still valid at execution" are completely separate measurement domains. Most systems collapse them together. You're making visible the gap between reasoning-time and action-time measurement state. That boundary is where silent failures hide - the agent was right at t1, but the world changed by t2. Forcing explicit dependency tracking moves system reliability from "hope nothing changed" to "verify what matters changed." This feels like the execution equivalent of retention curves for data validation - you can't judge correctness without measuring time explicitly.
That is exactly the distinction I was trying to isolate. Time is part of correctness once reasoning and action are separated.
The part I find most important is tracking only the evidence the decision actually depended on. Otherwise the choices become either “hope nothing changed” or re-read the entire world before every action, neither of which scales very well.
The distinction between correct reasoning and still-valid reasoning at execution time is interesting.
Curious whether builders see this as a separate reliability problem, or something they’d rather solve inside their existing transaction/workflow layer.
I think both approaches have a place. A transaction is usually the strongest answer when all relevant state and the write live inside the same transactional system.
FreshCtx is aimed at the messier cases where reasoning depends on several sources—a file, Git branch, API response, MCP resource or approval—and no single transaction can cover them all. In that case it provides a check immediately before the existing workflow or transaction begins.
You can reach me at hello@beryxa.com. Feel free to email me there if you’d like to continue the conversation.