A behind-the-scenes look at how we manage LLM keys without seeing user data
As we started shipping more LLM-powered features, one problem kept coming up: API keys and cost visibility.
Between multiple providers, different environments, and growing usage, it became hard to answer basic questions:
- Where are the keys?
- Who’s using what?
- How much is this actually costing us?
We didn’t want a solution that required logging prompts or responses, or pulling sensitive data into a central backend.
So we built our own setup.
At a high level:
- API keys are encrypted client-side and never stored in plaintext
- We use a single virtual key instead of juggling provider-specific keys
- Usage tracking is metadata-only (token counts, model names, timing)
- No prompts or responses are collected
- Inference stays on the client, so it works with cloud APIs and local models like llamafile
We ran this quietly in a small alpha to see if it held up in real usage.
It is now in open beta and free.
We’re fixing issues as they come up.
I’m sharing this mostly to sanity-check the approach with other builders:
- How are you handling LLM keys today?
- At what point did cost tracking become painful for you?
- What’s missing for this to be actually useful day-to-day?
Trending on Indie Hackers
This resonates — I'm building an AI-powered tech news aggregator and managing API costs across multiple providers is one of those "hidden complexity" problems that compounds quickly.
To your questions:
How I'm handling keys today:
Environment variables + provider-specific dashboards. It works, but I'm checking 3-4 different dashboards to understand monthly spend. The "virtual key" abstraction you mention sounds like it would simplify this significantly.
When cost tracking became painful:
Around $50-100/month. At that point, I needed to know which features were driving costs, not just total spend. Token counts by endpoint would've been helpful.
What would make this useful day-to-day:
The "no prompts/responses logged" part is important — that's usually the dealbreaker for trying third-party key management solutions.
Are you planning to support cost estimation before requests? That would be huge for setting up rate limits or showing users "this action costs ~X tokens."
Thanks for sharing this, yamamoto. That’s a unique angle; I'll raise it internally and think it through more.
Appreciate you considering it. Looking forward to seeing how the tool evolves.