We stopped treating agents as features once they could take actions
We build AiKey, an AI credential and governance layer for teams running models, tools, and agents.
One idea has changed how we think about the product: an agent stops being “just a feature” the moment it can do something outside its own context window.
It reads a document. It queries a system. It opens or updates a ticket. It calls an API. It takes the result and decides what to do next.
At that point, the difficult questions are no longer mostly about prompt quality.
They are operational questions:
- Which identity is this agent using?
- Which actions can it take, and in which environment?
- What happens if a tool times out or the loop keeps going?
- Can we reconstruct the decision path later?
- Can we see what this task cost before the bill lands?
We did not arrive at that view because we think agents need more ceremony. We arrived there because the opposite trade-off becomes expensive very quickly.
The demo boundary is real
A prototype can get away with a shared API key, a permissive tool wrapper, and a few logs. That setup is often exactly right for learning whether a workflow is useful.
The problem starts when the prototype reaches a real system.
A shared key no longer tells you which agent made a request. A broad permission no longer tells you whether the agent should be allowed to close a ticket instead of creating one. A pile of uncorrelated logs does not help much when a workflow makes several model calls, touches two services, retries a failing tool, and then returns a result that nobody can explain.
The model did not suddenly get worse. The environment got more consequential.
That distinction matters. Teams often react by looking for a better model, a smarter planner, or another orchestration layer. Those may help, but they do not solve the question of who is allowed to do what at runtime.
We now separate judgment from control
The agent should be free to make task-level judgments: which context matters, which tool to call next, whether an answer is complete.
The runtime should own the guardrails around those decisions.
For us, that means treating credentials, tool permissions, budgets, policy checks, audit events, and fallback behavior as parts of the execution path, not as documentation next to it.
The agent can propose an action. The runtime still decides whether that action is valid for this identity, environment, tool scope, and budget.
This is also how we think about the “agent harness” idea. A harness is not another word for an agent framework. It is the layer that connects reasoning to execution and makes the loop legible:
- assemble context;
- call the model;
- validate and execute a tool request;
- record the result and policy outcome;
- continue, stop, fail safely, or hand off.
The unglamorous parts matter. Timeouts. Retry limits. Idempotency. Model fallback. Step limits. Cancellation. Budget exhaustion.
Those are the parts that keep a helpful agent from becoming a surprisingly persistent background process.
The control layer should produce useful operational data
A good runtime layer does more than block risky actions. It produces the data needed to operate the system.
When a task goes wrong, we want to trace one chain: the triggering user or service, the agent identity, the credential or token used, the model route, the tool calls, the policy decisions, the failures, and the cost events.
Without that chain, every question becomes an investigation across separate dashboards. With it, the questions get simpler:
Was the request denied by policy? Did a tool fail? Did a retry loop inflate usage? Did the task run under the wrong environment? Which workflow owns the spend?
That is the practical reason we care about runtime governance. It makes agent systems easier to operate before they become harder to explain.
What this changed for us
We no longer describe the problem as “securing model access.” That phrase is too narrow.
The problem is governed execution: giving agents the ability to act while keeping the identity, permissions, cost, and audit trail connected to the same runtime.
At AiKey, that is the layer we are building around AI credentials, runtime policy, auditability, and cost attribution. We are not trying to replace existing IAM or gateway systems. The goal is to make AI calls fit into a control model that operators can actually work with.
If you are building agents that are beginning to touch real tools and systems, I would be curious how you draw this boundary. What was the first operational problem your prototype exposed?
More context on our approach: https://aikeylabs.com/zh/i/ih33
Business inquiries: aikeyfounder@gmail.com
The connected audit trail is the part that stands out to me.
Once an agent is taking real actions, having the identity, tool calls and policy decisions recorded is necessary — but I’m curious how you verify that the resulting behaviour was actually correct.
For example, do you independently sample production runs to test whether the agent should have been allowed to act, whether escalation happened at the right point, and whether the evidence trail supports the eventual outcome?
That gap between having governance controls and proving they behaved correctly in practice is something I’ve been exploring with OpsWatch.
The sampling question is exactly the right one. Two things we're doing: policy shadow mode — new policies run in observe-only mode against live traffic before they ever block; and replay — completed runs get re-evaluated to check whether the agent stayed in scope and escalation fired when it should.
Honest caveat: sampled review proves the runs you looked at, not the system. The real loop is turning findings back into policy changes. How does OpsWatch close that loop — manual review, or automated comparison of expected vs. actual behavior?
Hybrid, but with independent human judgement remaining the final assurance layer.
We define the expected behaviour and control conditions up front, then compare those expectations against actual/replayed runs — things like whether the action was permitted, whether grounding supported it, whether escalation should have fired, and whether the resulting evidence is internally consistent.
Automation can do a lot of the comparison and exception-flagging at scale, but I wouldn’t want OpsWatch to become a system that effectively certifies itself. Material findings still need independent review.
The useful loop is:
expected behaviour → observed run → deviation/evidence → policy/control change → re-verification
Your shadow-mode + replay setup is actually very close to the evidence layer I’d want underneath that process. The interesting bit would be seeing whether OpsWatch could sit independently on top of it rather than duplicating what you already do.
“Governed execution” is a useful way to frame it.
The interesting shift happens when an agent stops producing output and starts changing state in other systems.
At that point, identity, permissions, cost, rollback, and auditability become part of the product behavior, not infrastructure details.
Agreed. Once an agent mutates state, retries can duplicate side effects and timeouts can leave operations half-applied — so rollback and idempotency become product behavior, not ops hygiene.
The open question for us: how far should the runtime enforce rollback when side effects live in someone else's system? Do you lean toward agents declaring their own compensation steps, or expect the runtime to handle it?
I’d lean hybrid. The runtime should own the mechanics: idempotency, checkpoints, retry limits, and durable execution logs.
But compensation for external side effects probably needs to be declared by the workflow, because a generic runtime can’t know whether “undo” means refund, delete, reverse a status, or escalate to a human.
The runtime can then enforce that a compensation path exists before risky actions and orchestrate it when something fails.
A lot of agent projects focus on smarter reasoning while ignoring what happens when a tool call fails halfway through or a retry repeats the same action. I think AiKey’s next direction should combine short-lived credentials, idempotency controls, and policy simulation before deployment. As a full-stack engineer with agent and MCP experience, I’m very curious about this infrastructure and would love an opportunity to contribute.
You've named the exact failure mode most demos skip — a tool call that dies halfway or retries and repeats a side effect. Short-lived credentials, idempotency controls, and policy simulation are on our roadmap for that reason, and hands-on MCP experience is exactly what we value. Happy to talk more.