I'm a college senior shipping an AI study app and the math on AI costs is what's keeping me up at night.
Quick context: I'm a bioinformatics senior at UCSD with five weeks left to ship before graduate(see last update). I built Notie (https://notieapp.com), an AI-native study app where the AI lives inside your PDF instead of a separate tab. Free for now, subscription tiers later.
Here's what nobody told me about building an AI product as a solo founder. The infra you'd think would be expensive is cheap. Supabase, Vercel, Clerk, Cloudflare R2 combined stay under $800/month even at five thousand users. The Anthropic API bill is what dominates. At fifty users I'm projecting around $150/month. At five hundred, $1,550. At five thousand, around $15,600. AI is roughly 90 to 95 percent of total cost at every scale.
This means the entire viability of the free tier depends on cost discipline. Two things have done the most work: a retrieval pipeline for documents, and prompt caching on top of it.
The naive version of Notie attached the full PDF to every AI request. That worked fine for a 20-page paper. It fell apart on a 700-page textbook, where a single highlight-chat turn was eating 200K input tokens, or just hard-failing past Anthropic's page cap. Sending the whole document on every turn is the silent killer of unit economics for an AI study app.
So I built a small RAG-style pipeline. On upload, I extract the PDF to plain page-text once and store each page as its own row. At query time, instead of sending the whole document, I send the page the user highlighted plus its immediate neighbors, plus up to 4 other pages scored against the question. For the global 'ask across all my docs' surface, each document contributes 3 to 6 of its most relevant page excerpts depending on how many docs are attached. The summary endpoint head+tail packs the text and caps it at 150K characters so even a textbook fits.
The numbers shifted hard. Highlight chat on a 100-page paper went from ~30K input tokens per turn to ~3-5K. Same chat on a 700-page textbook went from ~200K (or hard fail) to ~3-5K. Cross-doc /ask with two PDFs went from 150-300K to 10-20K. Summary of a 700-page textbook went from impossible to ~37K and succeeding.
One nuance worth mentioning: my retrieval is currently lexical, not embeddings. Keyword overlap, phrase matching, page-number bonus. I deferred embeddings on purpose until I can measure lexical actually failing real students on real exam questions. Cheaper to ship, easier to debug, and good enough until proven otherwise.
The part that's been genuinely uncomfortable: I have to design the free tier knowing that every additional question someone asks costs me real money, and a meaningful chunk of users will never convert. The 'just throw AI at it' era of product building is not the same product-building reality if you're the one paying the API bill.
What I'm currently planning: free tier capped at three PDFs or one hundred AI questions per month, pro tier at $10/month. The free tier limit isn't to be stingy. It's the largest amount I can give away before each free user costs more than a pro user pays.
If you've shipped an AI-heavy product and figured out how to keep the free tier sustainable without crippling it, I'd love to hear what worked. Especially curious about when you knew it was time to upgrade lexical retrieval to embeddings, and how aggressive your free tier limits had to be at launch. Still calibrating.
This is a really good breakdown. The free tier question gets much easier once you think in terms of “token budget per user” instead of “questions per user.”
A pattern I’ve seen work: set an internal monthly token ceiling for free users, then translate it into a friendly product limit. For Notie that might be something like 3 PDFs + 100 questions, but the real guardrail is the max context size per answer and whether expensive requests silently degrade to cheaper retrieval before they hit the model.
Also think you’re right to delay embeddings until lexical fails on real student queries. Debuggable retrieval is underrated when you’re still learning the usage pattern.
Tiny related plug since this is exactly the problem space: I’m working on TokenBar, a macOS menu bar token counter for LLM/API work: https://tokenbar.site/ . Might be useful while you’re calibrating prompts and context sizes.
This is a super common problem for AI startups and the math really does keep you up at night. We went through the same thing about 6 months ago - AI costs were eating our margins and we had no visibility into which features were burning the most tokens. What helped was pulling all our AI billing into one place. We started using aicosts.ai to consolidate billing from Claude, GPT, and a few other providers. The setup was about 10 minutes. The per-model breakdown was the unlock - we found one feature burning 3x what anyone expected because it defaulted to the most expensive model. A few routing changes cut our overall AI spend by about 40%. For a college project, I would start with the cheapest models that work (Gemma, Qwen, etc.) and only use the expensive ones for tasks where quality actually matters. The key is having visibility into which tasks are using which models so you can make those decisions with data instead of guessing.
The uncomfortable truth you've surfaced: unit economics don't work at free-tier scale unless you engineer for them from day one. Your RAG pipeline cutting token consumption by 85-95% isn't just technical optimization, it's the business model. Most AI founders burn cash on "growth" instead.
The lexical-first strategy is underrated smart. You're deferring embeddings until you prove they're necessary for actual student outcomes, not just using the fanciest stack.
On free tier: consider a softer gate like unlimited questions on 1 PDF, capped on multi-doc workflows. Lets power users self-select into paid without alienating casual users who'd never hit 100 anyway. Your real moat isn't the AI, it's cost discipline at scale.
Fascinating cost breakdown! I run poll-sim.com (AI polling agents) and face similar token economics. Lexical retrieval is smart - we're using embeddings for poll answers but simple keyword matching for user segmentation. Free tier caps are necessary; we use 100 polls/month. Good luck!
Thank you very much for your feedback! Im still keeping track of costs per user to gauge if I needed to switch into embeddings. I also agree that free tier caps should definitely be implemented, but it is always hard to gauge the quota you want to allow the user to have, would appreciate if you can share some insights on how to get a good estimation!
Incredible technical breakdown on token optimization! Your RAG pipeline approach (90-98% reduction) is brilliant - especially the pragmatic choice to start with lexical retrieval instead of embeddings. The free tier monetization strategy is smart too: "largest amount I can give away before each free user costs more than a pro user pays" is exactly the right mindset. Excited to see where Notie goes!