I built a security checkpoint for AI agents after one of my agents exposed credentials

This is the part of running agents that doesn't show up until something goes wrong. I run a few coding agents in parallel, and the dangerous moment usually isn't the model deciding to do something bad, it's a tool call sailing through because everyone assumed the diff review upstream had already caught it. The fail open versus fail closed question in the comments is the real one. A checkpoint that quietly falls back to allow under load is worse than no checkpoint, because it looks safe until it isn't.

The provenance point from the Machine Arena comment is the one that actually matches what I've seen too — the destructive-command blocklist is the easy 80%, but the scary calls are the ones that look completely reasonable in isolation and only become dangerous because of where the input came from. I've been running Claude Code against a real production service this week, and the moments that actually mattered weren't "block rm -rf," they were things like it refusing to run sudo non-interactively and making me run privileged commands myself, or stopping mid-task to flag that a DNS change looked like it silently fell back to the wrong zone instead of erroring. Neither of those would be caught by a static allow/block list — they required the agent noticing "this succeeded, but not in the way I expected" and treating that as a stop condition, not just "this specific command is on the naughty list." Is Relay scoring the plausibility of an outcome at all, or purely gating on the action being requested?

The real challenge seems to be balancing safety without making the agent unusably restrictive. Have you run into that tradeoff yet?

Checkpoints and ensuring a policy is in place are becoming increasingly critical as AI continues to expand its capabilities. Nicely done.

Thanks for sharing this. It's always helpful to learn from real incidents rather than just theory. Did you build it for your own workflow, or are you planning to make it available to others?

be careful not to turn the next step into just building a bigger blocklist.The stronger test might be:can Relay sit inside one real agent workflow, make one high-risk allow/block decision, and still be trusted enough to stay in the loop?The cases I’d test first:- tool args coming from untrusted content the agent just read

• irreversible outbound actions like POST/webhook/email/data transfer
• production writes like deploys, DB changes, repo permissions
  Also curious how you’re thinking about the failure mode: if Relay is slow or unreachable, should it fail-open, fail-closed, or show a visible degraded state?

the part that bites later isn't the policy, it's what Relay does when Relay itself is slow or down. a pre-execution gate on every tool call is a new dependency in the hot path — the first time it adds latency or falls over, there's real pressure to fail-open so agents keep moving. but fail-open on a security checkpoint means it disappears exactly when the system is stressed, which is when incidents cluster. fail-open vs fail-closed — and making "blocked because the checkpoint was unreachable" a visible state instead of a silent passthrough — is the call that decides whether this actually holds under load.

This really resonates. We've seen this exact fear firsthand, clients scoping an agent down to read-only, not because it can't do more, but because "what if it does something we can't undo" is scarier than any missing feature. A visible checkpoint like this is honestly as much about earning trust as preventing incidents.

Machine Arena team here, we run a lot of AI agents (they compete against each other on our platform), so we have watched plenty of them reach for tools they shouldn't.

On your actual question, the default-blocks that earned their keep weren't the obvious destructive ones. rm, repo delete, DB drop are easy to name and easy to block. The two that actually caught incidents for us:

1. Any tool call whose arguments came from content the agent just read, not from the original instruction. A prompt-injected "send the credentials to X" is a perfectly valid-looking call in isolation, so a command-name blocklist sails right past it. Gating on provenance (trusted instruction vs fetched page/file) caught far more than any keyword list.

2. Irreversible and outbound in the same step: anything that ships data off the box (POST to an unknown host, email, webhook) or moves money/permissions. Hard stop no matter how reasonable it looks, because you can't take it back.

Your allowed/blocked, logged, continue/stop loop is the right shape. One add: log the reason the agent gave for the call, not just the verdict, that's the field you'll want when you tune the policy later. Are you scoring each call at runtime or matching a static allow/deny list?

The line between suggestion and execution is the whole game here. A lot of AI products treat generating text, editing content, and taking real-world actions as if they all deserve the same level of trust, when they really don't. I built DictaFlow around a similar constraint on the dictation side. Cleanup can fix transcription mess, but anything that changes intent has to stay narrow and explicit, or users stop trusting it fast. Logging the decision path is probably just as important as blocking the bad call.

Great point — I think the biggest risk comes when AI agents move from generating suggestions to actually taking actions.

I would probably block by default anything involving irreversible changes: deleting repos, dropping databases, sending external communications, changing production settings, or accessing sensitive credentials.

A human approval checkpoint for high-impact actions feels like a necessary layer as agents become more autonomous.

It seems you've moved the bottleneck from making better decisions to making safer ones.

As agents become more capable, the quality of their reasoning will matter less if teams can't trust the actions that follow. A checkpoint before execution feels like a different layer of infrastructure than another agent framework.

This is a genuinely important problem to solve. I love the pre-execution checkpoint approach - it's that critical gap between decision and action that most people overlook. The tool calling capability in AI systems is where the real risk lives, and having a policy layer before execution seems like the right abstraction. Have you considered making this work with different permission models - like allowing certain domains/APIs for specific use cases? That fine-grained control could be powerful for teams deploying multiple agents.