We spent a year talking to 30+ teams about AI costs. Caching saved them 90% of tokens—and their bills still went up.
We build aikey, a credential, budget, and audit layer for AI calls. For the last year we've talked to dozens of teams—startups, mid-size companies, a few enterprises—about the same problem: LLM bills that keep climbing no matter what they optimize.
Here's the pattern we kept seeing, and the hard lessons we learned from it.
The story everyone believes
Memory systems are the current darling of LLM cost optimization. Compress chat history into summaries, cache stable prompts, retrieve long-term facts on demand. Vendors promise 50–90% token savings. The demos are convincing, and honestly, the mechanics work.
We've watched teams ship this carefully: summaries tuned, cache order optimized, retrieval tested. Input tokens dropped like a stone.
Then the monthly invoice arrived, and it was higher than before.
What we learned when we opened the bills
The savings didn't disappear. They were never in the part of the bill that memory systems can touch.
Three patterns showed up in almost every conversation:
1. Scattered keys, zero attribution. Everyone has their own API key. Nobody watches spend until finance walks over with the invoice. Then nobody can answer the basic questions: which project, which environment, which failed retries are mixed into prod costs. One team found a staging environment that had been quietly leaking a meaningful slice of monthly spend for months without anyone noticing.
2. Agent loops that don't stop. Agent tasks burn an order of magnitude more compute than single-turn chat. In compound tasks, tool calls alone account for 85–90% of total token consumption. When a loop goes wrong, the damage compounds fast. There are documented incidents of agent experiments consuming catastrophic spend overnight. No hard exit conditions, no budget ceiling—it only takes one bad loop.
3. No budgets, no alerts, no circuit breakers. Fewer than one in ten teams could tell us what next month's AI spend would be. Budget thresholds, anomaly alerts, and overrun cutoffs are standard practice in cloud cost management. Almost nobody applies them to model calls. Money doesn't burn in one day—it leaks daily until the quarterly invoice makes it visible.
The macro shift made it worse
We also watched the market move against the "tokens are cheap" assumption in real time. After a year of aggressive price cuts, providers are rebalancing toward revenue quality—one major platform moved from free preview to paid and adjusted pricing up dramatically within a quarter.
Big companies are tightening too. Microsoft pulled back Claude Code licenses after individual engineers hit $500–2,000 per month. Uber burned a full year of AI budget in four months and capped individuals at $1,500/month. Meta's internal token leaderboard got so gamified it had to be taken down. Goldman Sachs expects token consumption to compound geometrically as agents scale.
Unit prices rising, usage exploding—the two curves stack. Optimizing one call at a time stops holding the line.
The reframe that changed our product roadmap
The turning point for us was a simple reframe: "save tokens" is the output of governance, not the substitute for it.
Memory systems decide the floor of a single call. Governance decides the ceiling of the whole month. They're different jobs, and you need both.
What teams that could actually explain their costs had in common: they made every call attributable first. Who, at what time, with which key, against which model, at what price. Only after that did budgets, alerts, cutoffs, and fallbacks start to work.
That reframe shaped aikey around three layers:
- Credentials—revocable derived keys bound to project, environment, and owner. Leak one, kill it in seconds; rotate without touching code.
- Budgets—daily and monthly quotas per key and per project. Warn near the threshold, throttle or downgrade past it.
- Audit—every call traced from key issuance onward, so "was that spend worth it" becomes a discussable question.
What we'd do differently if we started over
Honestly: we'd ship attribution before optimization. In the early days we spent too much effort helping teams squeeze tokens out of prompts, when the real wins came from visibility. If a team can't say where the money goes, every optimization is a guess.
We'd also push budgets earlier. The moment someone says "our AI bill feels high," the first deliverable isn't a cheaper prompt—it's a hard ceiling and an alert. Everything else can wait.
The practical checklist we now share
If you're running agents or heavy LLM usage, start here:
- Replace shared keys with per-project, revocable keys—small change, immediate visibility.
- Set daily/monthly quotas on the two or three highest-volume keys before anything else.
- Add an audit trail that ties every call to a project and an environment.
- Only then optimize context, caching, and model routing.
The honest bottom line
Memory systems are worth using. They're just not the whole answer, and they were never meant to be. The teams that stopped the bleed did the unglamorous work first: attribution, budgets, audit.
We're still building aikey around that principle. If you're fighting the same fight—or just curious about how we structured the credential and budget layers—the product is at https://aikeylabs.com/zh/i/ih34 and we read every email: Business Inquiries: aikeyfounder@gmail.com.
Engineering Teardown: How to Stop Rogue Agent Loops and Explosive Tool-Call Inflation Under Strict Budgets
Read the conclusion that 85–90% of token consumption is swallowed by runaway tool calls and un-monitored agent loops. Relying on superficial frontend monitors or basic, native max_iterations parameters is like putting a band-aid on a blown pipe. If your agent experiences an alignment anomaly or a validation defect, it will inevitably enter a multi-stage churn loop that bleeds corporate capital overnight.
The structural cause of this leakage is what I call the "LLM God Trap"—allowing a probabilistic language model to directly dictate, mutate, or read state without an external, hard-coded deterministic referee boundary.
To run simulations or multi-agent workflows safely under rigid compute constraints, you have to decouple probabilistic intelligence from authoritative state mutation.
Here is the exact architectural blueprint we deployed on a Cloudflare Worker edge network to completely quarantine agent loops and lock down unit economics to a sub-penny threshold [INDEX, INDEX]:
## 1. The 5-Phase Transaction Airlock
Never let an agent write directly to your database or execute a state mutation based on natural language prose. The model is structurally reduced to a pure Proposal Generator. It submits a candidate change, which must pass an un-yielding backend validation gate before it is committed:
[INPUT PROSE] → [INTENT CLASSIFICATION] → [STRUCTURED MODEL PROPOSAL] → [DETERMINISTIC VALIDATION] → [ATOMIC LEDGER COMMIT]
Narration and downstream presentation execution occur only after the authoritative state row is safely written to disk.
## 2. Stage-Centric Bounded Retries
When a model outputs an invalid data schema or violates a business logic constraint, standard apps let it retry the entire prompt lifecycle from scratch, causing instant token duplication and context bloat.
Instead, enforce Stage-Centric Retries. If a proposal fails validation, the referee gate rejects the transaction atomically—nothing is partially committed. The retry loop is legally bounded strictly to that exact operational stage and scope, forcing a localized, validation-informed resampling. The system is structurally blocked from running an uncontrolled global regeneration, preserving your upstream data truth and capping your cost runway.
## 3. Progressive Instantiation (The Lazy-Loading Data Gate)
Guard against loading or generating a massive, monolithic state grid at bootup. Establish an explicit separation between Existence, Resolution Depth, and Active Discovery. Keep macro data coordinates frozen as lightweight rows in your database. Only authorize high-resolution internal child detail to instantiate dynamically when an agent’s active vector explicitly crosses a coordinate boundary. If the agent is just looking around, the parent scope itself becomes the subject—no target entity is required, eliminating empty string panics and UNBOUNDED_PROPOSAL exceptions.
By turning your infrastructure into a self-auditing environment where data and authority cross system boundaries only after passing deterministic contract validation, your models can maintain absolute creative latitude inside an un-crashable sandbox.
This is interesting because once you have attribution, budgets, cutoffs and audit trails in place, the next question becomes whether those controls actually behaved as intended during execution.
For example, if an agent exceeds a boundary, gets blocked, or should escalate, how do you verify the real behaviour independently rather than relying only on the same system’s logs?
That separation between control configuration and independently verified execution is something we’re exploring with OpsWatch, particularly for production AI agents.
Fair point — logs written by the same system enforcing the rule are weak evidence.
This is why we treat audit as evidence, not self-report. Every call is traced from key issuance onward in an append-only trail, so when a budget cutoff fires, you can replay the sequence and confirm the throttle actually held at the boundary. If the only way to know a control worked is the control's own log, you're one bug away from a false sense of safety.
We also push teams to break their own systems on purpose: fire a synthetic call past the quota, use a revoked key, watch what the runtime actually does. If a team can't demo a control failing correctly before production depends on it, the control isn't done.
OpsWatch's direction makes sense to me — separating control configuration from independently verified execution is the right instinct. We're early in that territory too, and I think it's where the whole category is heading.
Exactly. That distinction is the heart of what we're building with OpsWatch.
The interesting part for us is taking that one step further: not just asking whether the control generated an audit trail, but whether the available downstream evidence is actually sufficient to establish that the intended business outcome occurred — especially at the boundary conditions where controls are most likely to fail.
Your point about deliberately breaking the system is exactly the kind of evidence we want to surface. Quota exceeded, revoked credential, stale execution, denied action, partial completion — then independently determine what actually happened rather than accepting the system's interpretation of it.
I think there's a potentially useful overlap between what AiKey Labs is building and where we're taking OpsWatch. Would be keen to compare notes on the technical boundary you're seeing between control enforcement, runtime evidence and independent verification.
When diving into AI costs and the use of memory systems, I can share that streamlining your model's interactions can have a significant impact on overall expenses. In my experience, we found that incorporating structured memory for frequently used prompts and user interactions led to a reduction in repeated processing, which saved us about 20% on our compute costs over several months.
We started by identifying patterns in how users interacted with our content tools. For example, consistently caching responses to common queries allowed for a much faster response time while utilizing less processing power. This kind of insight can significantly enhance user experience as well, since the interaction becomes more fluid and tailored.
Additionally, we experimented with summarizing longer user histories into condensed insights that our AI could reference. This not only preserved key context but also accelerated response generation. We tracked average response times pre- and post-implementation, noting a reduction of about 30%, which directly correlated to improved user engagement.
Layering these methods on top of effective content strategies helped us optimize not just the costs, but also the perceived value of our offering. If I were to suggest a next step, consider segmenting your user interactions based on frequency and complexity. Apply memory selectively to maximize cost efficiency while keeping user experience at the forefront.
It’s a continuous learning process, so testing different implementations and monitoring metrics closely will help find the best balance for your specific context.
Nice to see real numbers — 20% on compute and 30% on response time is the kind of result most teams claim but can't measure.
That tracks with what we've seen: memory and caching genuinely lower the floor on each call. The problem we kept hitting in conversations wasn't that these optimizations don't work — it's that teams couldn't see where the savings went. One workflow gets cheaper while a staging key leaks quietly for months, and the bill ends up flat or higher. Optimization is real; it just needs a baseline you can trust.
The last point you made is the one I'd underline: segmenting interactions by frequency and complexity and applying memory selectively is exactly the right instinct. That's governance thinking disguised as prompt engineering.
It’s good, but this version sounds more natural and keeps the conversation moving:
That’s a great way to put it optimization lowers the cost of a call, but governance shows whether the business is actually saving money. Of the teams you’ve worked with, what usually reveals the biggest hidden cost first: unused keys, staging environments, or runaway agent loops?