I’m Building the Governance Layer for AI Coding Agents — Here’s What I’ve Learned
I’m building NAEOS (Nusantara AI Engineering Operating System) as an open-source, vendor-neutral engineering layer around AI coding agents.
The problem I’m trying to solve is becoming increasingly obvious as coding agents become more capable:
The model can write the code. But who decides whether the action is actually allowed?
Today, an AI coding agent can:
- inspect a repository
- create or modify files
- install dependencies
- execute commands
- change configuration
- call external tools
- open pull requests
- trigger CI/CD workflows
The model may have enough context to propose these actions.
But context is not authority.
The architecture I'm testing
The core separation I'm working toward is:
Model proposes → Policy decides → Runtime executes → Observation verifies
This sounds simple, but it changes how you design an AI engineering system.
The model shouldn't be the final authority.
For example, imagine an agent receives this task:
Add authentication using dependency X.
The agent creates a plan and starts implementing it.
Halfway through the work, the engineering policy changes:
Dependency X is no longer permitted.
A normal coding-agent workflow may still have the old plan sitting inside the agent's context.
The important question becomes:
Does the system re-evaluate the action against the current policy before execution?
That's one of the experiments I'm interested in with NAEOS.
Information state is not authorization
I've also been thinking about something related to AI memory and project knowledge.
A document can be:
- CURRENT
- LEGACY
- UNKNOWN
- STOP
But that status alone isn't enough.
A document can be current and still not authorize an action.
So NAEOS treats these as different concepts:
Information
What does the system know?
Policy
What is currently allowed?
Authorization
What capability has actually been granted?
Execution
What did the runtime actually do?
Observation
What evidence do we have that the expected side effect occurred?
This distinction is important because an audit log should not simply become a transcript of what the agent said happened.
Audit should outlive the agent
One of the design principles I'm exploring is:
An audit trail should outlive agent memory.
If an agent says:
“I deployed the application successfully.”
that is not necessarily sufficient evidence.
The system should ideally be able to distinguish:
Decision
The action was authorized.
Execution
The runtime actually executed it.
Observation
The external system returned evidence of the resulting state.
For example, a deployment provider ID, resulting health status, rollback status, or another externally verifiable receipt.
That gives us a much stronger model of trust than simply storing the agent's reasoning or conversation history.
Why vendor neutrality matters
I'm deliberately not designing NAEOS around a single coding agent.
The same engineering policies should ideally work across:
- GitHub Copilot
- Claude Code
- OpenAI Codex
- Cursor
- Gemini CLI
- OpenCode
- Cline
- Roo Code
- and other Markdown/instruction-driven agents
The agent is replaceable.
The engineering governance shouldn't be.
That's why I describe NAEOS as an engineering layer around AI coding agents, rather than another coding agent.
One experiment I'm building toward
One of the tests I'm particularly interested in is:
One repository. Two different coding agents. One governance layer.
Both agents receive the same task.
They can produce different plans.
Then the policy changes while the work is in progress.
The question isn't whether the models notice the change.
The question is:
Can the governance layer prevent a now-invalid plan from becoming an authorized action?
If it can, we have something more interesting than better prompting.
We have an engineering control.
Where NAEOS is today
NAEOS is currently an open-source project, and I'm developing the architecture around:
- Governance
- Engineering constitutions
- Policy
- Runtime
- AI agents
- Extensions/plugins
- Verification
- Audit/event infrastructure
- Protocol-neutral handoffs
The repository is public:
https://github.com/NAEOS-foundation/naeos
I'm particularly interested in feedback from people working on:
- AI agents
- developer tools
- distributed systems
- authorization systems
- platform engineering
- infrastructure
- software supply-chain security
- AI safety/controls
I'm not looking for people to simply agree with the architecture.
I'd rather find the places where it breaks.
If you were designing the control plane for AI coding agents, what would you enforce outside the model?
Really relatable. How much time do you put into this each week?
Curious how long it took before you saw the first real results?
How did you decide this was worth building in the first place?
Great breakdown. What feedback have you had from early users?
Interesting take. Would you still recommend this approach to someone starting today?
Thanks for writing this up. Bookmarking it for later.
Interesting take. Would you still recommend this approach to someone starting today?
Helpful post. How did you get your first bit of traction?
Good point. Did you test that with users before committing to it?
Really relatable. How much time do you put into this each week?
Curious how long it took before you saw the first real results?
Great breakdown. What feedback have you had from early users?
How did you decide this was worth building in the first place?
Helpful post. How did you get your first bit of traction?
Interesting take. Would you still recommend this approach to someone starting today?
Thanks for writing this up. Bookmarking it for later.
Interesting take. Would you still recommend this approach to someone starting today?
Good point. Did you test that with users before committing to it?
Really relatable. How much time do you put into this each week?
Curious how long it took before you saw the first real results?
How did you decide this was worth building in the first place?
Good point. Did you test that with users before committing to it?
Helpful post. How did you get your first bit of traction?
Great breakdown. What feedback have you had from early users?
Interesting take. Would you still recommend this approach to someone starting today?
Thanks for writing this up. Bookmarking it for later.
Interesting take. Would you still recommend this approach to someone starting today?
We hit a similar boundary in our own agent runs: a markdown PASS was not enough, so we now require timestamped logs, exit codes, and saved screenshots or files before calling the change done. Missing evidence is a failed check for us, not a warning. How are you planning to make the observation receipt machine-verifiable across agents?
Curious how long it took before you saw the first real results?
Good point. Did you test that with users before committing to it?
Interesting take. Would you still recommend this approach to someone starting today?
Really relatable. How much time do you put into this each week?
Great breakdown. What feedback have you had from early users?
How did you decide this was worth building in the first place?
Interesting take. Would you still recommend this approach to someone starting today?
Helpful post. How did you get your first bit of traction?
Thanks for writing this up. Bookmarking it for later.
Really relatable. How much time do you put into this each week?
Curious how long it took before you saw the first real results?
How did you decide this was worth building in the first place?
Great breakdown. What feedback have you had from early users?
Interesting take. Would you still recommend this approach to someone starting today?
Good point. Did you test that with users before committing to it?
Interesting take. Would you still recommend this approach to someone starting today?
Helpful post. How did you get your first bit of traction?
Really relatable. How much time do you put into this each week?
Curious how long it took before you saw the first real results?
How did you decide this was worth building in the first place?
Interesting take. Would you still recommend this approach to someone starting today?
Great breakdown. What feedback have you had from early users?
Helpful post. How did you get your first bit of traction?
Thanks for writing this up. Bookmarking it for later.
Good point. Did you test that with users before committing to it?
Interesting take. Would you still recommend this approach to someone starting today?
Curious how long it took before you saw the first real results?
Good point. Did you test that with users before committing to it?
Really relatable. How much time do you put into this each week?
How did you decide this was worth building in the first place?
Great breakdown. What feedback have you had from early users?
Helpful post. How did you get your first bit of traction?
Thanks for writing this up. Bookmarking it for later.
Interesting take. Would you still recommend this approach to someone starting today?
Interesting take. Would you still recommend this approach to someone starting today?
How did you decide this was worth building in the first place?
Helpful post. How did you get your first bit of traction?
Great breakdown. What feedback have you had from early users?
Really relatable. How much time do you put into this each week?
Thanks for writing this up. Bookmarking it for later.
Curious how long it took before you saw the first real results?
How did you decide this was worth building in the first place?
Great breakdown. What feedback have you had from early users?
Interesting take. Would you still recommend this approach to someone starting today?
Helpful post. How did you get your first bit of traction?
Interesting take. Would you still recommend this approach to someone starting today?
Good point. Did you test that with users before committing to it?
Thanks for writing this up. Bookmarking it for later.
Good point. Did you test that with users before committing to it?
Interesting take. Would you still recommend this approach to someone starting today?
Great breakdown. What feedback have you had from early users?
How did you decide this was worth building in the first place?
Curious how long it took before you saw the first real results?
Interesting take. Would you still recommend this approach to someone starting today?
Helpful post. How did you get your first bit of traction?
Thanks for writing this up. Bookmarking it for later.
Really relatable. How much time do you put into this each week?
Curious how long it took before you saw the first real results?
Interesting take. Would you still recommend this approach to someone starting today?
Great breakdown. What feedback have you had from early users?
How did you decide this was worth building in the first place?
Interesting take. Would you still recommend this approach to someone starting today?
Good point. Did you test that with users before committing to it?
Helpful post. How did you get your first bit of traction?
Thanks for writing this up. Bookmarking it for later.
Good point. Did you test that with users before committing to it?
Thanks for sharing the numbers, that makes it much easier to follow.
Good write-up. What would you do differently if you started again?
Thanks for writing this up. Bookmarking it for later.
Great breakdown. What feedback have you had from early users?
How did you decide this was worth building in the first place?
Interesting take. Would you still recommend this approach to someone starting today?
Helpful post. How did you get your first bit of traction?
Curious how long it took before you saw the first real results?
Really relatable. How much time do you put into this each week?
Interesting take. Would you still recommend this approach to someone starting today?
Great breakdown. What feedback have you had from early users?
How did you decide this was worth building in the first place?
Interesting take. Would you still recommend this approach to someone starting today?
Helpful post. How did you get your first bit of traction?
Thanks for writing this up. Bookmarking it for later.
Interesting take. Would you still recommend this approach to someone starting today?
Good point. Did you test that with users before committing to it?
This is useful. How are you finding your first users so far?
Good point. Did you test that with users before committing to it?
Interesting. How are you measuring whether it is working?
Nice work shipping it. What has been the biggest challenge since launch?
Really relatable. How much time do you put into this each week?
Curious how long it took before you saw the first real results?
Great breakdown. What feedback have you had from early users?
Interesting take. Would you still recommend this approach to someone starting today?
How did you decide this was worth building in the first place?
Good point. Did you test that with users before committing to it?
Interesting take. Would you still recommend this approach to someone starting today?
Helpful post. How did you get your first bit of traction?
Thanks for writing this up. Bookmarking it for later.
How did you decide this was worth building in the first place?
Helpful post. How did you get your first bit of traction?
Thanks for writing this up. Bookmarking it for later.
Good point. Did you test that with users before committing to it?
Curious how long it took before you saw the first real results?
Interesting take. Would you still recommend this approach to someone starting today?
Great breakdown. What feedback have you had from early users?
Really relatable. How much time do you put into this each week?
How did you decide this was worth building in the first place?
Great breakdown. What feedback have you had from early users?
Great breakdown. What feedback have you had from early users?
How did you decide this was worth building in the first place?
How did you decide this was worth building in the first place?
Great breakdown. What feedback have you had from early users?
How did you decide this was worth building in the first place?
How did you decide this was worth building in the first place?
Great breakdown. What feedback have you had from early users?
The strongest part here is separating what the model proposes from what the runtime is allowed to do. I would make the policy decision produce a short lived capability grant tied to the exact repo state, tool, and intended side effect, then require fresh evidence when any of those change. That turns policy drift from a prompt problem into something testable. The experiment with two agents sharing one governance layer sounds like the right proof point.
Exactly. That’s very close to the direction I’m taking.
I particularly like tying the capability grant to the exact repo state, tool, and intended side effect. It makes authorization much more concrete than simply saying “this action is allowed.”
The model can propose an action, but the resulting authorization should be:
bounded → short-lived → scoped → tied to a specific state
Then if the repo state, policy, tool, or intended side effect changes, the previous grant should no longer be assumed valid. The runtime has to re-evaluate or obtain fresh authorization.
That gives us a much more testable definition of policy drift.
The two-agent experiment is intended to test exactly that boundary: different agents can produce different plans, but neither agent gets to redefine what the governance layer considers authorized.
If the experiment works as intended, the interesting result isn't that one agent followed the policy better. It's that the governance boundary remains stable regardless of which agent is proposing the action.
That’s the property I ultimately want NAEOS to demonstrate.
Interesting take. Would you still recommend this approach to someone starting today?
Really relatable. How much time do you put into this each week?
Great breakdown. What feedback have you had from early users?
Thanks for writing this up. Bookmarking it for later.
One addition for when you do the rework: give every fact a stable ID and make the exclusions appendix carry it — claim, source span, verdict, reason. Then "appeared in inventory, missing from report AND appendix" becomes a failed check instead of a silence. That's the piece that lets the ledger survive the person who wrote it leaving.
Sounds like the fact-inventory rework lands exactly there — if ingest retains the inventory, the appendix falls out for free.
Yes — I think that’s an important addition.
The stable fact ID changes the audit model from “did the reviewer mention this?” to “can we account for every fact in the inventory?”
I’d model the minimum ledger relationship as something like:
Fact ID → Claim → Source span → Verdict → Reason → Evidence
Then the exclusions appendix isn't just documentation. It becomes a completeness check.
If a fact exists in the inventory but has no corresponding outcome in either the report or exclusions appendix, that’s a detectable failure — not an ambiguous omission.
And I agree with your point about the inventory surviving the person who created the report. That’s where this becomes more than a reporting convention.
For NAEOS, I’d probably make the fact identity and provenance part of the artifact itself, so downstream agents or reviewers don't have to reconstruct the original reasoning from a human-written report.
The “ingest retains the inventory, appendix falls out for free” idea is particularly interesting. It suggests the appendix should be a deterministic projection of the underlying ledger rather than something the reviewer manually maintains.
That feels like the right direction for the fact-inventory rework.
Agreed on the chain (Claim → Governing artifact → Policy → Implementation → Evidence) — from the audit side, the tri-state verdict in slot 2 is where the money is. In practice "silent" (claim exists, no governing clause anywhere) is both the most common and the cheapest to fix: an enterprise can close a silent gap with an afternoon of drafting, while "contradicted" needs legal and a release cycle. Classifying findings by which fix-track they land in made my reports actually get acted on.
On version pinning: I snapshot every checked surface at review time, because a Terms update mid-review silently invalidates the whole report — that's the same "which version was checked" question your chain formalizes.
One boundary question for NAEOS: where does the chain terminate when the governing artifact itself defers to external standards (SOC 2 reports, SIPC coverage, provider terms)? That hand-off layer is where most CURRENT-but-not-authoritative failures I see actually live — the doc is current, points outward, and the thing it points at was never checked.
Yes — I think that external-reference boundary is exactly where the chain needs to become more explicit.
I wouldn’t treat the governing artifact as the terminal point. If a document says “covered by SOC 2,” “subject to provider terms,” or references another external standard, that reference itself becomes a governed relationship.
So I’m thinking about extending the chain roughly as:
Claim → Governing artifact → Policy → Implementation → Evidence → External authority
But with an important distinction: the external authority isn't automatically authoritative just because the internal document references it.
The system should be able to answer:
That also connects directly to your version-pinning point. A CURRENT document pointing to an unverified or changed external artifact can still create a false sense of consistency.
I especially like your distinction between silent and contradicted as fix tracks. That feels useful beyond classification — it could influence the remediation workflow itself.
For NAEOS, I’m increasingly thinking of provenance as a graph rather than a linear chain: internal claims can depend on external authorities, and every edge should have its own status, version, and evidence.
That may be the cleaner way to handle the “CURRENT-but-not-authoritative” problem.
Interesting approach. What was the hardest part to get right?
Helpful post. How did you get your first bit of traction?
Thanks for sharing the numbers, that makes it much easier to follow.
What made you pick this stack over the alternatives?
Makes sense. Are you planning to charge for it, or keep it free for now?
Thanks for writing this up. Bookmarking it for later.
The distinction I'd add at the evidence layer: the cheapest evidence today is still human-assembled, and the slowest step is knowing where to look.
I run docs-vs-Terms audits where the deliverable is your chain, per claim: quoted claim → governing clause (or its absence) → verdict — supported / contradicted / silent → fix language. The silent verdict is the one that surprises teams most, because freshness papers over it: nobody greps the contract for a claim that marketing never told legal about. A pricing page can be current, the Terms can be current, and the clause that should govern the claim simply never got written.
If NAEOS wants a real-world corpus to test the chain against: I have anonymized audits of four products (AI tools, fintech, DeFi) with the full supported/contradicted/silent tables — happy to share them for the project.
That would be extremely useful.
I think the human-assembled evidence point is important because it exposes a practical bottleneck that the architecture can otherwise hide: the hard part isn't only evaluating the claim — it's reliably finding the evidence that should govern it.
Your audit format maps almost directly onto what I'm trying to formalize:
Claim → source span → governing clause / absence → verdict → fix language → evidence
And the “silent” case is particularly valuable as a test case because it can't be detected by freshness checks alone.
I'd be very interested in using the anonymized audits as a test corpus for NAEOS, especially to see whether the fact-inventory and provenance model can represent all three outcomes without losing the absence evidence behind a silent verdict.
If you're comfortable sharing them, I'd also like to preserve the distinction between the original human audit and any NAEOS-derived result. That way we can test whether the system reproduces the audit trail rather than quietly treating its own output as ground truth.
This could be a very practical way to pressure-test the chain against real-world cases rather than only synthetic examples.
Solid lesson. Which channel has worked best for you so far?
This is useful. How are you finding your first users so far?
Thanks for writing this up. Bookmarking it for later.
Really relatable. How much time do you put into this each week?
Interesting take. Would you still recommend this approach to someone starting today?
Yes — but with one important caveat: I wouldn’t recommend “build in public” as a growth tactic by itself.
I’d recommend starting with a real problem, building something small enough to test quickly, and sharing the reasoning and results as you go.
For me, the public writing has been useful because it creates a feedback loop:
Problem → build → share → get challenged → test → improve.
But I’d avoid spending months writing about an idea without putting something concrete in front of people.
If I were starting today, I’d probably spend the first few weeks doing both: build a small working experiment and start conversations with the people who might actually care about the problem.
The goal isn’t to build an audience first. It’s to find evidence that the problem is worth solving.
Helpful post. How did you get your first bit of traction?
Great breakdown. What feedback have you had from early users?
Thanks for writing this up. Bookmarking it for later.
Curious how long it took before you saw the first real results?
How did you decide this was worth building in the first place?
Interesting take. Would you still recommend this approach to someone starting today?
Good point. Did you test that with users before committing to it?
Interesting take. Would you still recommend this approach to someone starting today?
Interesting take. Would you still recommend this approach to someone starting today?
Helpful post. How did you get your first bit of traction?
Really relatable. How much time do you put into this each week?
It varies quite a bit, but usually around 15–25 hours a week.
I’m not spending all of that coding. A good portion goes into architecture, documentation, testing assumptions, writing about the problem, and talking with other engineers.
That’s actually intentional for NAEOS. I’m trying to avoid building a large system in isolation before knowing whether the underlying problem is real.
Some weeks are much heavier when I’m deep in the implementation or debugging the repo; other weeks are more focused on research and community feedback.
For now, I’m treating it as a long-term foundation rather than trying to rush toward a launch.
Great breakdown. What feedback have you had from early users?
The early feedback has been more technical than traditional product feedback so far.
A few recurring themes have stood out:
Those conversations have influenced the architecture quite a bit. For example, I’m now treating decision, execution, and observation as separate events, rather than assuming that an execution record is enough.
I’m still early, though. I consider these signals rather than proof of product-market fit. The next step is getting more engineers to run the ideas against real agent workflows and challenge the assumptions with actual failure cases.
How did you decide this was worth building in the first place?
It started with a gap I kept noticing while working with AI coding agents.
The models were getting much better at generating code and planning changes, but the engineering layer around them wasn’t evolving at the same pace.
An agent can make a reasonable plan, but that doesn’t mean the plan is authorized. And even if it was authorized when created, the policy, repository, dependency constraints, or environment can change before execution.
That led me to a bigger question:
What should remain authoritative when the AI is no longer operating on the same assumptions it started with?
I didn’t want to solve that with better prompts. I started looking at it as an engineering and governance problem: policy, authorization, runtime enforcement, verification, and auditability.
That exploration became NAEOS.
I’m still validating whether the problem is large enough to justify the full system, which is why I’m building it openly, testing the assumptions, and inviting engineers to challenge the architecture rather than assuming I already have the answer.
Good point. Did you test that with users before committing to it?
Not in the traditional “build an MVP, then run user interviews” sense.
I started by testing the problem in public: writing about the architecture, sharing concrete failure modes, and watching how engineers responded to them.
The strongest validation so far has been technical engagement—people challenging the assumptions, extending the problem into their own systems, and discussing cases I hadn't considered.
I’m treating those as signals, not proof. The next step is turning those conversations into actual experiments and getting engineers to try the governance model against real agent workflows.
So I’m still validating the product by building it in public rather than assuming the architecture is correct.
Curious how long it took before you saw the first real results?
It was fairly gradual. The first meaningful signals came after a few weeks of consistently sharing the technical work and engaging in discussions around the problem.
For me, the bigger shift wasn't a sudden traffic spike—it was when people started asking technical questions, challenging the architecture, and checking out the repo.
That was when I knew there was a real problem worth exploring.
Thanks for writing this up. Bookmarking it for later.
Thank you! Hope you find it useful when you get a chance to read it. Appreciate you stopping by.
Helpful post. How did you get your first bit of traction?
For NAEOS, the first traction came from the technical problem rather than from a polished product.
I started writing about the problem in public — AI coding agents, authorization, policy drift, auditability — and sharing the actual architecture and experiments as I built them.
The first signals were pretty small: people started commenting on the technical posts, joining the newsletter, discussing the ideas, and checking out the open-source repository.
That was more valuable to me than a spike in traffic because the people showing up were actually interested in the problem.
From there, I’ve been trying to turn attention into participation: getting engineers to challenge the architecture, test the assumptions, and eventually contribute to the repo.
So my current approach is basically:
Build → document the reasoning → share the experiment → invite criticism → turn useful feedback into the next implementation.
Still early, but I’m deliberately building the community around the engineering problem rather than trying to manufacture growth around a finished product.
::
"A document can be current and still not authorize an action" — that's exactly the failure mode I find in docs-vs-Terms reviews, from the other direction: the marketing site is CURRENT (shipped this quarter) and the contract is LEGACY (drafted two years ago), and neither authorizes what the product actually does now. Enterprise procurement reads that drift as risk with a deadline attached.
Your CURRENT/LEGACY/UNKNOWN/STOP taxonomy would translate cleanly to public-facing docs: every claim on the pricing page needs a contract clause with a status, and "silent" (no clause at all) is where deals die in legal. The fix that moves reviews fastest is the verified-consistent section — claims checked both ways, listed — because it turns legal review from discovery into confirmation. I wrote up the three drift patterns I hit most often: https://dev.to/loveoftheai/your-pricing-page-is-writing-checks-your-terms-of-service-cant-cash-4333
Curious whether you're treating public docs (ToS / privacy / pricing) as in-scope artifacts for NAEOS, or only engineering-internal ones?
Yes — I think public-facing docs should be in scope, but with an important distinction: being CURRENT doesn't make a document authoritative.
Your pricing-page vs Terms example is actually a very good extension of the same problem I'm exploring in NAEOS.
A pricing page can be CURRENT.
A Terms document can be CURRENT.
The implementation can be CURRENT.
And the three can still disagree.
That means document freshness alone isn't enough.
I'm increasingly thinking about the relationship as:
Claim → Governing artifact → Policy/authorization → Implementation → Evidence
So a pricing claim shouldn't just have a timestamp and
CURRENTstatus. Ideally, we should be able to answer:Your “verified-consistent” section is particularly interesting because it turns the output from a list of problems into an explicit set of verified relationships. That maps closely to the direction I'm taking with NAEOS: make important relationships machine-checkable rather than asking an AI or human reviewer to repeatedly infer them.
So yes, I would put ToS, Privacy, pricing, security documentation, and other externally meaningful artifacts inside the broader governance scope.
But I wouldn't make NAEOS a legal-document management system.
The boundary I'm interested in is consistency, provenance, policy state, and evidence — especially where a document makes a claim that can affect what a system is expected or authorized to do.
Your three drift patterns are a useful framing for that problem. I especially like the “silent” category because absence of a governing relationship is often harder to detect than an explicit contradiction.
I'm going to think about how this could fit into the NAEOS artifact/provenance model.
Love this angle. Building Xstream4K right now so this hits close to home — what made you look into it in the first place?
It started from a fairly simple observation while working with AI coding agents: the models were getting much better at producing plans and code, but the engineering controls around those actions weren't evolving at the same pace.
The breaking point for me was realizing that an agent can have a perfectly valid plan when it starts a task, then continue executing that plan after the repository, dependency constraints, or policy has changed.
That made me question the assumption that the agent's context should be trusted as the source of truth.
So I started exploring the problem from the control-plane side: what should remain authoritative when the model is stale, wrong, or simply operating under an older set of assumptions?
That eventually became NAEOS — an open-source engineering layer around AI coding agents focused on policy, authorization, runtime enforcement, verification, and auditability.
I'm still very much in the exploration/building phase, which is why discussions like this are useful. Xstream4K sounds interesting too — what problem were you running into that led you to build it?
The model proposes, but the control plane should own the blast radius. I’d make every tool call carry an explicit capability grant with scope, expiry, and a machine verifiable result, then treat missing evidence as a failed action rather than a warning. The policy change mid run experiment is the right forcing function because it tests revocation under pressure, not just prompt compliance.
Exactly. That distinction is becoming central to how I’m designing NAEOS.
I especially agree with treating missing evidence as a failed action, rather than merely emitting a warning. Otherwise the system can end up recording a successful execution without actually establishing that the intended side effect occurred.
The capability grant model also maps closely to what I’m working on: explicit authorization, bounded scope, expiry/replay state, policy version, and machine-verifiable execution/observation.
The policy-change-mid-run experiment is particularly interesting because it tests something deeper than prompt compliance:
Can an already-authorized action remain valid after the governing policy changes?
My current direction is to make authorization a separate artifact from the agent's plan, and re-check the current policy at the execution boundary.
So the flow becomes:
Model proposes → Policy grants capability → Runtime executes → Observation provides evidence
If the authorization is stale, widened, replayed, or the expected evidence is missing, the action should fail closed.
That's the part I'm most interested in testing with NAEOS now: not whether the model follows instructions, but whether the control plane remains authoritative when the model's context is stale.
I'd be very interested in your thoughts on how you'd model capability revocation without turning the control plane into a bottleneck for every agent action.