tokenmix

Most teams running AI API costs hit the same wall around month 4 — the bill quietly 2-3x's as soon as the product gets real traffic. We hit it last quarter and went deep on what actually compresses it.

The short version: caching in AI gateways is not one feature — it's two. And most aggregation platforms (OpenRouter, Portkey, even newer ones) only ship half by default. That half saves real money. The other half, stacked on top, is what cuts the total bill roughly in half.

Full technical breakdown: [AI Gateway Caching 2026 → tokenmix.ai](https://tokenmix.ai/blog/ai-gateway-caching-l1-l2-guide-2026)

The two layers in indie-hacker terms:

- L1 — result cache: skip the model entirely on duplicate or semantically similar requests. 100% savings per hit. Helicone does this via a 1-line base_url swap. Self-hosted Redis works too.

- L2 — prompt cache: the vendor (Claude, OpenAI, DeepSeek, Gemini) caches the KV state of your stable prompt prefix. Model still runs, but input cost drops 50-90%. Claude reads cache at 10% of input price (90% off). DeepSeek same. OpenAI auto-caches anything ≥1024 tokens.

Most teams get only L2 because vendors auto-enable it. L1 is what gateways typically skip.

Real math for 10M requests/month (Claude Sonnet 4.6, 4K avg input tokens):

- No caching: $195K/month baseline

- L2 only (80% prefix hit rate): $119K/month (−39%)

- L1 + L2 stacked (25% L1 hit + L2 on rest): $90K/month (−54%)

Savings are compound, not additive — the requests L1 absorbs never touch L2, so L1 savings don't dilute.

4 things we learned running this in production at TokenMix.ai:

1. Prefix stability is everything for L2. Middleware that rewrites system prompts (adds timestamps, user IDs, etc.) kills the cache key. Pinning our system prompt format alone 4x'd our real cache hit rate.

2. Semantic cache false positives hurt. We set cosine threshold at 0.88 early and got some wrong answers. Moved to 0.95+, accepted lower hit rate for correctness.

3. Vendor lock-in is worse without gateway caching. Direct Claude integration gets L2. But migrating to another model means rebuilding cache infra. Gateway-level caching makes the switch cheap.

4. Measurement is not optional. No cache-hit-rate dashboard = flying blind. Shipping this as a first-class metric in TokenMix.ai's usage panel.

If you're early: just make prompt prefixes stable so L2 works when you scale. Don't build L1 yet.

If you're already at scale: the full write-up with pricing tables, code examples, and architecture patterns is on [our blog](https://tokenmix.ai/blog/ai-gateway-caching-l1-l2-guide-2026) (~2500 words), tighter dev version on [Dev.to](https://dev.to/tokenmixai/ai-gateway-caching-explained-why-l1-l2-cache-layers-cut-90-of-your-llm-bill-45ab).

Happy to answer questions on L1 semantic cache tuning in comments — that's where most teams screw up first.

I got tired of juggling API keys.

Every time I wanted to test a new model — switch from GPT-4o to Claude, or try a Qwen variant — I had to manage separate accounts, separate billing, separate rate limits. For a solo developer, that overhead adds up fast.

So I built TokenMix.ai.

The idea is simple: one OpenAI-compatible endpoint, 170+ models behind it. You point your existing code at api.tokenmix.ai, swap the model name, and that's it. No SDK changes, no refactoring.

client = OpenAI( base_url="https://api.tokenmix.ai", api_key="your_key" )

Why this matters for builders

When you're prototyping, you don't want to commit to one model upfront. Different tasks call for different models — Claude tends to be better for writing, GPT-4o for structured outputs, smaller open-source models for high-volume cheap tasks. Being able to benchmark them with the same code in 5 minutes changes how you build.

Pay-as-you-go, no monthly fees. You pay for what you use. Good for early-stage projects where traffic is unpredictable.

Where we are now

Still early. 170+ models live, 99.9% uptime target, actively adding more. Looking for feedback from developers actually building with it — what models are missing, what's broken, what would make this actually useful for your workflow.

If you're doing anything with AI APIs, give it a try and let me know what you think.

→ tokenmix.ai