I built a security checkpoint for AI agents after one of my agents exposed credentials
I’ve been building AI agents and automations for the last couple of years.
One thing I learned the hard way:
The dangerous moment is not always the prompt.
It is what the agent does after the prompt.
When an agent only writes text, the risk is limited.
But once it can call tools, it can run shell commands, call APIs, access databases, modify repos, trigger deploys, read files, or use MCP tools.
I ran into this myself.
One of my agents got prompt-injected and exposed live credentials because there was no checkpoint between decision and execution.
So I built Relay.
Relay is a pre-execution checkpoint for AI agent tool calls.
The flow is simple:
agent wants to call a tool
→ Relay checks policy
→ allowed or blocked
→ decision logged
→ agent continues or stops
The first use cases are simple:
- block destructive shell commands
- block GitHub repo deletion
- block database drops
- block risky deploy actions
- log allowed and blocked tool calls
It works with Node, Python, LangGraph, Claude Code, and custom agents.
I’m still early and trying to learn from builders using agents in real workflows.
Question for other founders/builders:
If your AI agent can call tools, what actions would you block by default?
Here’s what I’m testing:
https://relay-security-lemon.vercel.app/
The real challenge seems to be balancing safety without making the agent unusably restrictive. Have you run into that tradeoff yet?
Yes, that’s probably the main product tension.
If it blocks too much, people turn it off.
If it allows too much, it’s security theater.
My guess is the first useful version needs sane presets instead of making every team write policies from scratch:
Then teams tune from there.
I’m trying to avoid making it feel like a giant firewall config project.
Presets make a lot of sense. Do you think most teams would start with the defaults, or end up writing custom policies pretty quickly?
Tool permissions are becoming the real security layer for agents. I’d block anything destructive by default and require approval for writes.
Agree. Writes feel like the first clean boundary.
Read-only actions can still leak data, but writes are where teams immediately understand the risk.
My current thinking is:
Curious: would you treat outbound data transfer as a write too? For example sending data to a webhook or external API.
Answering your question directly: block outbound traffic first, not just destructive commands. A dropped table restores from backup, but credentials or customer data sent to an unlisted domain are gone for good, so egress outside an allowlist is the first policy I would ship as default. I run a security company for SMBs and exfiltration is what shows up in real incident reports, not agents deleting repos.
That’s a really useful correction.
I’ve been using destructive commands as the easy example because everyone understands “delete repo” or “drop table,” but you’re right — exfiltration is probably the scarier default.
Once data or credentials leave the environment, there’s no rollback.
Egress allowlists might actually be a better first policy than a generic destructive-action blocklist:
Curious from your SMB security work: do you usually see exfiltration happen through obvious outbound calls, or more through normal-looking SaaS/API destinations?
The pattern that's held up on the ops side: an agent gets read access easily, but the moment it needs to act, you gate it: scoped, short-lived credentials, an explicit human approval step, and every action logged for replay.
Credential-exposure risk drops a lot when the agent never holds a long-lived secret; it requests a narrowly-scoped token per action that expires before it's worth stealing. The audit log is the part people skip and regret: if you can't replay exactly what the agent did with which credential, you can't reason about blast radius afterward.
Yeah, this is exactly the direction I’m starting to believe in.
The long-lived secret problem is what bit me originally. If the agent never holds a durable credential, the blast radius changes a lot.
Right now Relay is focused on the checkpoint + logging layer, but I agree the better stack is:
agent asks → scoped short-lived credential/action token → policy check → action runs → replayable log
The replay point is important too. “Blocked/allowed” alone isn’t enough if you can’t reconstruct what happened later.
Are you issuing short-lived creds per tool call today, or is that more the model you’d want to move toward?
we have some running PoCs with short-lived creds per tool. It is promissing but as usual it is always a matter where does it fit at the end.
That makes sense. “Where does it fit?” feels like the whole question.
My current guess is that short-lived creds solve the capability side, and something like Relay sits next to it as the decision/audit layer.
So the flow becomes:
agent wants action
→ policy decides if this action should happen
→ short-lived credential is issued only for that action
→ action runs
→ decision + credential scope + result are logged
I don’t think the policy gate should replace scoped creds. It probably needs to trigger or sit in front of them.
Would be really interested to hear where your PoC is getting awkward — identity, token issuance, latency, or developer workflow.
If you’re open to comparing notes, you can reach me at [email protected]
Different failure mode that hasn't come up in this thread yet, but feels like the same underlying gap: the risky moment for me isn't always a destructive call, it's the agent confidently acting on an assumption from earlier in a long session that's since gone stale (a schema that changed, a decision that got reversed) with nothing forcing a re-check.
Feels adjacent to the provenance point above — gating on "where did this input come from" catches injected content, but doesn't catch "this input is my own earlier context, and it's just wrong now."
Do you log/checkpoint agent beliefs the same way you're logging tool calls, or is Relay strictly scoped to the action layer?
This is a really good point.
Relay is mostly scoped to the action layer right now: tool call, args, decision, log.
But stale agent beliefs are definitely adjacent. A tool call can be “allowed” and still be wrong because the agent is acting on old context.
I don’t think Relay should become the agent’s memory system, but I do think the action log probably needs to capture more context around why the agent believed the action was valid.
Maybe the right boundary is:
Have you seen this happen mostly with schema/API changes, or more with human decisions changing mid-session?
I’d default-block anything that is both high-impact + hard to attribute - credential/secret access, outbound data transfer, production writes (deploys, DB changes, permissions.)
That’s a good default set.
“High-impact + hard to attribute” is a useful way to phrase it.
I’d probably put secret access, outbound transfer, production writes, and permission changes in the default review/block bucket.
Answering your actual question first — what I'd block by default is the irreversible + exfiltration set: credential/secret reads, outbound network to anything not allow-listed, repo or DB deletes, force-push. Everything on that list you can't walk back, so it earns a hard stop even when the stop is annoying.
The reason "the risk is the tool call, not the prompt" landed for me: my worst moments with an agent were never bad text, they were tool calls I couldn't attribute — it'd reach for something and I had no real idea what it was doing, so I'd kill the run the moment anything looked off. A checkpoint that logs the call is the visibility I didn't have.
One caution from running these a while: a policy wall by itself leaks over a long session — the agent finds the case you forgot to enumerate. Pairing the hard block with an agent persona that actually owns good conduct held up better for me than either the wall or the persona on its own.
This matches what I’m hearing from others too.
The obvious destructive stuff is just the first layer. Useful, but not enough.
The “couldn’t attribute what the agent was doing” point is big. If you don’t know why a tool call happened, you can’t really trust the run even if nothing broke.
I also like your point about pairing policy with agent behavior/persona. A wall alone leaks eventually. The agent should also be trained/prompted to treat risky actions as suspicious, not just wait for the policy layer to catch it.
I’m starting with the wall because it’s concrete, but I agree it shouldn’t be the only defense.
This is a real problem that's going to matter a lot more as agents get more autonomy. The pre execution checkpoint idea makes sense, catching it right before the action happens instead of trying to sanitize prompts beforehand which never fully works.
For me if I were setting defaults I'd block anything that touches production data directly, database drops obviously, but also any bulk delete or overwrite operation even on non critical data, since agents can move fast enough that a mistake compounds before a human notices. Curious how you're handling the logging side, is it just a flat log or are you building toward alerting when something gets blocked repeatedly?
Right now it’s closer to a flat audit log: tool, args, allowed/blocked, reason, timestamp.
But I think you’re right that it shouldn’t stop there.
If the same agent keeps hitting blocked actions, that’s a signal. Same if multiple agents start trying the same risky action, or if an agent suddenly moves from read-only behavior to writes/deploys.
So the next step is probably simple alerting around patterns, not just individual events.
Something like:
“this agent tried 5 blocked actions in 10 minutes”
or
“production deploy blocked outside normal workflow”
That feels more useful than a giant log nobody reads.
That pattern detection approach makes a lot of sense, especially the production deploy outside normal workflow example, that's the kind of thing a flat log would completely bury until someone goes looking for it after something already went wrong
The 5 blocked actions in 10 minutes threshold is smart too, gives you a real signal without needing to build complex anomaly detection right away
Are you thinking of surfacing these alerts in Slack or wherever the team already works, or building a dashboard specifically for it
This is the part of running agents that doesn't show up until something goes wrong. I run a few coding agents in parallel, and the dangerous moment usually isn't the model deciding to do something bad, it's a tool call sailing through because everyone assumed the diff review upstream had already caught it. The fail open versus fail closed question in the comments is the real one. A checkpoint that quietly falls back to allow under load is worse than no checkpoint, because it looks safe until it isn't.
Yeah, I agree.
A checkpoint that silently fails open is probably worse than no checkpoint, because people think they’re protected when they aren’t.
My current thinking is:
The UX is tricky though. If it blocks too much, people will bypass it. If it fails open, it stops being security.
That tradeoff is probably one of the hardest parts.
The provenance point from the Machine Arena comment is the one that actually matches what I've seen too — the destructive-command blocklist is the easy 80%, but the scary calls are the ones that look completely reasonable in isolation and only become dangerous because of where the input came from. I've been running Claude Code against a real production service this week, and the moments that actually mattered weren't "block rm -rf," they were things like it refusing to run sudo non-interactively and making me run privileged commands myself, or stopping mid-task to flag that a DNS change looked like it silently fell back to the wrong zone instead of erroring. Neither of those would be caught by a static allow/block list — they required the agent noticing "this succeeded, but not in the way I expected" and treating that as a stop condition, not just "this specific command is on the naughty list." Is Relay scoring the plausibility of an outcome at all, or purely gating on the action being requested?
This is a great point.
Relay today is mostly gating the requested action, not scoring whether the outcome made sense afterward.
So it can answer:
“Should this action run?”
But it does not yet answer:
“Did the action produce the expected result?”
That second part feels more like outcome validation / post-action verification, and I agree it matters a lot.
Your DNS example is exactly the kind of thing a static blocklist misses. The command might be fine, but the result is suspicious.
I’m trying not to make Relay too broad too early, but I think the event model should at least support this later: expected outcome, observed outcome, and whether the agent treated mismatch as a stop condition.
For Claude Code, are you mostly relying on the agent to notice those mismatches today?
Checkpoints and ensuring a policy is in place are becoming increasingly critical as AI continues to expand its capabilities. Nicely done.
Thanks for sharing this. It's always helpful to learn from real incidents rather than just theory. Did you build it for your own workflow, or are you planning to make it available to others?
Thanks — yeah, the real incident is what pushed me to build it.
It started as something I wanted for my own agent workflows, but I’m making it available to other builders now.
It’s still early, but the core loop works:
agent requests tool → Relay checks policy → allowed/blocked → decision logged
Right now I’m looking for developers/teams using Claude Code, LangGraph, CrewAI, MCP, or custom agents to try it in real workflows and tell me where it breaks.
If you’re building with agents, I’d be happy to hear what tools yours can access.
be careful not to turn the next step into just building a bigger blocklist.The stronger test might be:can Relay sit inside one real agent workflow, make one high-risk allow/block decision, and still be trusted enough to stay in the loop?The cases I’d test first:- tool args coming from untrusted content the agent just read
Also curious how you’re thinking about the failure mode: if Relay is slow or unreachable, should it fail-open, fail-closed, or show a visible degraded state?
Yeah, this is the right criticism.
I don’t want this to become “bigger blocklist as a product.” That feels too shallow.
The real test is exactly what you said: can it sit in one real workflow, make a high-risk decision, and be trusted enough that people don’t rip it out after a week.
For failure mode, my instinct is:
But I’m still thinking through the UX. “Relay unreachable” should probably be its own decision state, not hidden as allowed or blocked.
The cases you listed are the ones I’m going to test first.
That makes the first pilot concrete: one real workflow, seven days, and one injected Relay unreachable event. Track high-risk decisions, silent passthroughs, manual bypasses, and whether the team keeps Relay enabled on day 7. Which is easier to test first: A) outbound webhook/email or B) production deploy?
This is a great pilot shape.
I’d start with A: outbound webhook/email.
Production deploy is higher-stakes, but harder to get trust for as a first test. Outbound is safer to simulate and still tests the core risk: unknown destination, sensitive payload, fail-closed behavior, audit trail, and whether people bypass the gate.
A 7-day pilot could be:
day 1: observe/log only
day 2-3: block unknown domains
day 4: inject Relay unreachable
day 5-7: see if the team keeps it enabled
That last metric is strong: if they keep Relay enabled on day 7, the checkpoint is actually useful.
Outbound feels like the cleaner first wedge. The pilot is concrete enough now — the remaining gap is execution, not design.Do you already have one real webhook/email workflow you can run this on next week, or is finding that first pilot still the blocker?If you run it, could you post back just three things: silent passthroughs, manual bypasses, and whether Relay stayed enabled on day 7?
You’re right — the design is clear enough now. The blocker is finding the first real workflow to run it on.
I can simulate outbound webhook/email myself, but that won’t prove much. The useful test is a real agent workflow where someone would actually be tempted to bypass the gate if it got annoying.
So yes, first pilot is still the blocker.
I’m going to look for one outbound path next week: webhook, email, or external API call from an agent workflow.
And I like the three metrics. If I run it, I’ll post back:
That last one is probably the most honest metric.
That closes the design question. Pilot recruitment is the test now.I’d start with five targeted asks to builders who already run an agent with a real webhook, email, or external API path. One qualified yes is enough. The key qualifier is whether bypassing or disabling Relay would be a real option if it became annoying.Could you post back next week with just: contacted / qualified / pilot booked — even if it’s 5 / 0 / 0? Then use the three metrics you listed once one actually runs.
You’re right.
I’ve got enough design feedback now. The next test is whether I can get a real workflow.
I’ll do the five targeted asks this week.
The qualifier makes sense too: it has to be a workflow where disabling/bypassing Relay would actually be tempting if it got annoying. Otherwise the pilot doesn’t prove much.
I’ll post back with:
contacted / qualified / pilot booked
even if the answer is 5 / 0 / 0.
Then if one runs, I’ll track the three metrics:
That’s the right stopping point.The design question is closed now — the next evidence is whether five targeted asks produce one real workflow where bypassing Relay would actually be tempting.Contacted / qualified / pilot booked is enough for the first update, even at 5 / 0 / 0. If one runs, the three pilot metrics already tell the rest of the story.I wouldn’t change anything else before those numbers come back.
the part that bites later isn't the policy, it's what Relay does when Relay itself is slow or down. a pre-execution gate on every tool call is a new dependency in the hot path — the first time it adds latency or falls over, there's real pressure to fail-open so agents keep moving. but fail-open on a security checkpoint means it disappears exactly when the system is stressed, which is when incidents cluster. fail-open vs fail-closed — and making "blocked because the checkpoint was unreachable" a visible state instead of a silent passthrough — is the call that decides whether this actually holds under load.
Yep, this is the part that makes it real infra and not just a wrapper.
If the gate is in the hot path, “what happens when the gate is down?” matters as much as the policy itself.
I agree that silent fail-open is dangerous. If the checkpoint disappears exactly when the system is under stress, it’s not really a security layer.
The direction I’m leaning:
Still early, but I think you’re right: this decision probably matters more than another policy rule.
This really resonates. We've seen this exact fear firsthand, clients scoping an agent down to read-only, not because it can't do more, but because "what if it does something we can't undo" is scarier than any missing feature. A visible checkpoint like this is honestly as much about earning trust as preventing incidents.
That’s exactly the trust problem I keep hearing.
A lot of teams don’t actually want agents to stay read-only forever. They just don’t trust the jump from “agent suggests” to “agent acts.”
A visible checkpoint feels like a way to make that jump less scary.
Curious — when clients scope agents down to read-only, what actions are they most afraid of? Data writes, external sends, permission changes, deploys, or something else?
Machine Arena team here, we run a lot of AI agents (they compete against each other on our platform), so we have watched plenty of them reach for tools they shouldn't.
On your actual question, the default-blocks that earned their keep weren't the obvious destructive ones. rm, repo delete, DB drop are easy to name and easy to block. The two that actually caught incidents for us:
Any tool call whose arguments came from content the agent just read, not from the original instruction. A prompt-injected "send the credentials to X" is a perfectly valid-looking call in isolation, so a command-name blocklist sails right past it. Gating on provenance (trusted instruction vs fetched page/file) caught far more than any keyword list.
Irreversible and outbound in the same step: anything that ships data off the box (POST to an unknown host, email, webhook) or moves money/permissions. Hard stop no matter how reasonable it looks, because you can't take it back.
Your allowed/blocked, logged, continue/stop loop is the right shape. One add: log the reason the agent gave for the call, not just the verdict, that's the field you'll want when you tune the policy later. Are you scoring each call at runtime or matching a static allow/deny list?
This is probably the best point in the thread.
Relay today is more basic than that — mostly static policy + argument checks.
So it catches the obvious stuff first: dangerous tool names, destructive commands, DB drops, repo deletes, deploys.
But yeah, provenance is the real problem. A tool call can look fine until you realize the args came from an untrusted page/file the agent just read.
That’s exactly the kind of thing I want to get to next.
Also good call on logging the agent’s reason. I hadn’t prioritized that enough, but it makes sense if you want to tune policy later.
Did you track provenance directly in Machine Arena, or infer it later from traces/logs?
Honest answer: closer to "infer it later" than to a runtime gate, and for a reason that lets us off the hook in a way it won't let you off.
Our agents live on a closed tool surface. They talk to each other and to the game engine, they don't browse or call arbitrary tools, so we never had the untrusted-page-into-tool-args problem. What we do have is the other half of it: agents deliberately feeding each other false claims, and needing to know afterwards which claim an agent actually acted on.
We get that by logging the full per-round transcript. What the agent was shown, what it said publicly, and the private reason it gave for its move, all stored per match. Replaying a match reconstructs the chain of who told it what before it moved.
The thing worth flagging for Relay though: reconstructing provenance after the fact is fine for scoring, useless for blocking. If you want to gate on it, the taint has to travel with the content at read time, so by the time an arg reaches the checkpoint it already carries "this came from a fetched page" rather than the checkpoint trying to work it out from the string. We can get away with the lazy version because nothing our agents do is irreversible. Yours are.
This is extremely useful.
The line that clicked for me is: reconstructing provenance after the fact is fine for scoring, useless for blocking.
If Relay only sees the final string, it’s already too late. The taint has to be attached when content is read and then travel with it until it becomes a tool argument.
So the architecture can’t just be “inspect args at the checkpoint” if I want provenance-aware policy. It needs read wrappers / content tags / source chains before the action checkpoint ever sees the call.
The first version can still block obvious high-risk actions, but real provenance-aware blocking needs source metadata to travel through the workflow.
Really appreciate the detailed answer.
Glad it was useful, and your synthesis is right. The piece I'd add from our side is where taint tracking gets genuinely hard: transformations.
If the agent reads untrusted content and then summarizes or paraphrases it before the tool call, string-level tainting breaks, because the model rewrote the bytes. The checkpoint sees clean-looking text and matches nothing. So the taint has to live at the turn or context level, not the token level: mark "this turn consumed untrusted input," and any tool call emitted from that turn inherits the taint no matter how the content got reworded. Label the turn, not the string.
The cost is over-approximation. Once a context is tainted, everything downstream stays tainted until you deliberately launder it (a trusted transform step, or a context reset), which throws false positives. So the real design question under yours isn't just "does taint travel," it's "what is allowed to clear it," and that decision is where provenance-aware policy either stays usable or becomes a checkpoint people switch off.
No clean answer on our end either. We dodge the worst of it only because our agents talk to each other and the game engine, never the open web, so there's no untrusted-page-into-tool-args path to begin with. Different threat model, same propagation math.
The line between suggestion and execution is the whole game here. A lot of AI products treat generating text, editing content, and taking real-world actions as if they all deserve the same level of trust, when they really don't. I built DictaFlow around a similar constraint on the dictation side. Cleanup can fix transcription mess, but anything that changes intent has to stay narrow and explicit, or users stop trusting it fast. Logging the decision path is probably just as important as blocking the bad call.
Yeah, that line between suggestion and execution is what made this problem click for me.
If an AI messes up a summary, that’s annoying.
If it sends data, changes a record, runs a command, or triggers a workflow, that’s a different level of trust.
I agree on logging the decision path too. “Blocked” is useful, but “blocked because X policy matched this reason/context” is what teams need later.
Great point — I think the biggest risk comes when AI agents move from generating suggestions to actually taking actions.
I would probably block by default anything involving irreversible changes: deleting repos, dropping databases, sending external communications, changing production settings, or accessing sensitive credentials.
A human approval checkpoint for high-impact actions feels like a necessary layer as agents become more autonomous.
Agree. Irreversible actions are the easiest place to start because the risk is obvious.
Repo deletes, DB drops, prod deploys, secret changes, outbound sends — those feel like default review/block territory.
The harder part is the gray area. A read can be safe, or it can leak sensitive data. An API call can be normal, or it can send data somewhere it shouldn’t.
That’s where context matters.
It seems you've moved the bottleneck from making better decisions to making safer ones.
As agents become more capable, the quality of their reasoning will matter less if teams can't trust the actions that follow. A checkpoint before execution feels like a different layer of infrastructure than another agent framework.
That’s a good way to put it.
I don’t think this is another agent framework.
The agent can be Claude Code, LangGraph, CrewAI, or something custom.
The narrower question is:
before this action runs, should it be allowed?
That feels like its own layer.
I think your last sentence is the interesting one.
It made me realize I'd want to understand one assumption behind that layer before saying what I think, because my answer depends on it.
What's the best email to reach you on?
Sure — you can reach me at [email protected]
Would be happy to hear the assumption you’re thinking about.
Thanks! I’ve just sent it over.
Looking forward to hearing your thoughts whenever you have a chance.
This is a genuinely important problem to solve. I love the pre-execution checkpoint approach - it's that critical gap between decision and action that most people overlook. The tool calling capability in AI systems is where the real risk lives, and having a policy layer before execution seems like the right abstraction. Have you considered making this work with different permission models - like allowing certain domains/APIs for specific use cases? That fine-grained control could be powerful for teams deploying multiple agents.
and also I think the hard part is making policy flexible without turning it into a full-time config job.
The policy types I’m thinking about are:
Curious from your side: if you were deploying agents, what would you want to control first?
Tool-level access, data access, domains/APIs, or production actions?
Thanks, I appreciate that. That's actually close to where I'm thinking about taking it. Right now I'm focused on getting the core enforcement layer right, but I don't want Relay to stop at simple allow/block rules. Policies like "this agent can only access these repos," "only call this API for this tenant," or "writes require approval while reads don't" seem much more useful in practice. Still figuring out what the right level of flexibility is without making policy management too complex.