A behind-the-scenes look at how we manage LLM keys without seeing user data

As we started shipping more LLM-powered features, one problem kept coming up: API keys and cost visibility.

Between multiple providers, different environments, and growing usage, it became hard to answer basic questions:

1. Where are the keys?
2. Who’s using what?
3. How much is this actually costing us?

We didn’t want a solution that required logging prompts or responses, or pulling sensitive data into a central backend.

So we built our own setup.

At a high level:

• API keys are encrypted client-side and never stored in plaintext
• We use a single virtual key instead of juggling provider-specific keys
• Usage tracking is metadata-only (token counts, model names, timing)
• No prompts or responses are collected
• Inference stays on the client, so it works with cloud APIs and local models like llamafile

We ran this quietly in a small alpha to see if it held up in real usage.

It is now in open beta and free.

We’re fixing issues as they come up.

I’m sharing this mostly to sanity-check the approach with other builders:

• How are you handling LLM keys today?
• At what point did cost tracking become painful for you?
• What’s missing for this to be actually useful day-to-day?