Had an agent almost drain my API budget over a weekend — here's what actually stopped it happening again

Left an agent running over a weekend to do some scheduled cleanup work. Came back Monday to a bill that made no sense. Turned out a retry loop in my own orchestration code was re-firing the same expensive call every time a downstream service hiccuped — nothing malicious, no prompt injection, just a bug plus an agent that had no concept of "this is too much" baked into it anywhere.

What got me was realizing the system prompt telling it to "be cost-conscious" was doing basically nothing. It's not that the agent ignored it — it's that nothing was actually checking. The limit lived in a sentence, not in anything that could stop a request.

So I rebuilt the whole thing around one idea: the agent should never be the thing deciding whether it's allowed to spend. Something outside it, that it can't see or reason about, has to be the one saying yes or no, every single time, based on an actual running total — not a per-call cap (which is trivially beaten by making more, smaller calls), a real cumulative limit tracked server-side.

Plus a kill switch I can hit instantly without redeploying anything, and a log of every request that I can actually trust wasn't edited after the fact when I'm trying to figure out what happened.

None of this is exotic. It's just applying "don't trust the client" to a client that happens to be an LLM calling your own code — which somehow doesn't feel obvious until you've been burned by it once.

Anyone else building agent stuff hit this? Curious if people are just eating the risk, building this per-project, or if there's a common pattern people have landed on.