I rebuilt my LLM cost simulator with current pricing — trying to find out if anyone actually needs it
A while back I built modelindex, a simulator that estimates what an LLM feature will actually cost in production — you pick a scenario (support bot, RAG search, agent, etc.), set your token assumptions, and it shows monthly cost across models.
Problem: the prices went stale. New models, new pricing, and the numbers were just wrong — which for a cost tool means it's worse than useless. So I spent the last week refreshing it: every price verified against current provider pricing, consolidated into one source so it can't drift again, and I added the cost levers that actually move the bill (output being ~5x input, prompt caching, retries, context waste).
I'm not here to claim it's great. I'm trying to answer one question honestly: does anyone actually use a thing like this, or do people just eyeball pricing pages? I instrumented real usage so I can tell the difference between "people clicked" and "people used it."
If you run anything on LLM APIs: I'd genuinely value you trying a scenario and telling me what's wrong or missing. The "this doesn't model X" replies are the most useful thing I can get right now.
visit: www.modelindex.io
I think I'd use something like this before launching a feature or comparing providers, but probably not day -to-day . The bigger for me would be answering " what happens to my costs if usage doubles or i switch models? " rather than calculating today's bill . Curious if you've seen people use it more for planning than monitoring.
This is a useful distinction and honestly closer to how I use it myself, I reach for it at decision points (new feature, switching providers) way more than as a daily dashboard. The "what if usage doubles / what if I switch models" framing is interesting because that's really asking the tool to compare two shapes side by side, not just price one. Right now you'd do that by running it twice and eyeballing the diff, which isn't great for exactly the planning use you're describing.
Genuinely curious on your end: when you hit that "what if we switch models" question, is it usually cost driving the decision, or cost as a tiebreaker once quality's close? Trying to figure out whether the useful thing is a cost comparison or something that weighs cost against quality.
For me it's usually a tiebreaker . If two models meet my quality bar , cost becomes the deciding factor. I'd always find it useful if the tool could compare quality assumptions alongside cost , since the cheapest model isn't always the cheapest once retries or failure are factored in .
That last point is exactly the thing I keep coming back to the "cheapest" model stops being cheapest the moment retries and failures are in the picture, and that's invisible if you're just comparing sticker prices. That's basically why I built the cost side the way I did.
The quality-alongside-cost piece is the harder half, and I'll be honest about why I haven't built it: cost I can compute from real pricing, but quality is squishier it depends on your task, and I don't want to slap on fake benchmark scores and pretend that's rigorous. The version I'd trust would let you set your own quality bar per model from your own testing, then show cost among the ones that clear it. Would that be useful to you, or would you want some baseline quality signal to start from rather than defining it yourself?
Useful tool. I've been deep in AI tooling myself — building a directory that compares 500+ AI tools including pricing. The "does anyone actually need it" question is one I think about constantly too. What's been your validation signal so far — usage, feedback, or just gut feel?
Honestly, mostly the first two, I tried hard to avoid gut feel because that's how I've fooled myself before. I instrumented real usage events (scenario runs, not just pageviews) and wrote down a specific success threshold before I posted anything, so I can't move the goalposts after the fact.
The idea was to separate "people landed" from "people actually modeled something," which are completely different signals.
Too early to read it yet just started sharing. The qualitative feedback like yours is honestly the part I'm weighting most right now.
Directory sounds like a big undertaking the pricing-accuracy problem alone must be brutal at 500+ tools.
I think one subtle insight here is that accuracy isn't just a product feature—it's the product. A cost simulator only creates confidence if people believe the assumptions stay current. I also like that you're measuring whether people actually model scenarios instead of just landing on the page, because those are very different signals of value.
Yeah, you put it better than I did — "accuracy is the product" is exactly the bet. A cost tool with stale numbers isn't a worse tool, it's an actively misleading one, so the whole thing lives or dies on whether people trust the assumptions are current.
The "stay current" half is the part I'm still figuring out. I consolidated everything to one pricing source this week so it can't drift across the app the way it had been — but that just makes correcting it a one-place job, it doesn't solve actually keeping up as models ship. Still an open problem for me.
Since you clearly think about this — if you've shipped anything on LLM APIs, does the simulator actually match your cost shape? Or is there an assumption it's missing? That's the feedback I'm most after right now.
That's exactly why I wanted to continue the conversation.
Reading your reply, I think there's one strategic business decision sitting underneath that open problem which becomes much more significant as the product evolves, but I don't think I can do the reasoning behind it justice in a thread.
Happy to explain what I mean if it's useful. What's the best email to reach you?
Thank you, you can email me ravibuildswork@gmail.com
Just sent it over by email.
Looking forward to hearing your thoughts once you've had a chance to read it.