Margin Ai

Hi IH,

I was building a multi-step research agent a few months ago and hit a massive wall: Unit Economics.

I realized my agent was spending ~80% of its tokens on "dumb" background loops—formatting JSON or extracting dates—but I was paying the full GPT-4o price for every single call. My bill was exploding before I even had a single customer.

Then came the privacy headache. I couldn't demo to anyone serious because my agent was leaking customer PII to the cloud LLMs by default.

I spent the last few weeks building Margin AI to solve this. It’s an open-source, local-first control plane that you run in Docker. It does three things in one line of code:

1. Intent-based Routing: Automatically routes simple tasks to cheap models (Llama-3/Groq) and saves the "big guns" for complex reasoning.

2. Sub-ms Semantic Caching: Intercepts repetitive agent loops locally.

3. PII Firewall: Scrubs sensitive data locally before it ever leaves your VPC.

I just opened the repo today and I'm hoping it helps other indie founders ship enterprise-grade agents without the soul-crushing costs.

Repo: https://github.com/ramprag/margin_ai

Question for you guys: How are you managing your LLM bills as you scale? Are you doing manual routing in your code, or just eating the costs for now?