Another challenge is LLM's pricing
Even before building my app, I had many ideas of AI-based products and the infinite unsolvable problem was comparing LLMs pricings. I found several pages/websites on the Internet but they all lack either models or filtering options.
So, I decided to make it on my own. Found several free sources on the Internet, then built a searchable / filterable table. So, now it has 300+ LLMs to compare. Please note: it allows to compare LLMs against each other, not the providers (this is coming).
https://llmknights.com/llmprices
I hope you enjoy and let me know if you lack something :)
Per-token sticker price is the wrong unit of account, and it is why every comparison table quietly misleads. At SocialPost.ai our cheapest model per token was not our cheapest per finished output, because verbosity and retry rates differ so much between models. Since you are already building a testing playground, publishing cost per completed task from your own test runs would make this table something nobody else has.
This hits close to home. I'm building a tool that turns podcast transcripts into visual content using AI. The generation cost per piece is real, and it makes pricing the product much harder than I expected.
Yes, I can see it. Another thing that is probably a challenge is choosing a model that fits your needs and shows the best ration price/quality. This is why I created my tool - you will able to test different models, compare the result and metrics like cost, latency, token amount.
Seems right now that routers are big business, they're in demand as a tool and as an acquisition target.
I wonder if there's some obvious tie-in here for you to monetise.
https://www.justwatch.com/ offers availability of films across different providers - could you have a view like that? Or another analogy skyscanner.net
Hi, thanks for your question but unfortunately I don't get what you mean. I'm planning to add different providers information as well as make them available for testing on our platform. Is that what you mean?
LLM cost is definitely one of the biggest challenges for AI products.
I think the key question is whether the AI experience creates enough value that users are willing to pay more than the inference cost.
Hi, I think every providers has some features that make them valuable more than paying to providers directly.
That's a good point. I think the challenge is becoming "which model/tool should I choose for this specific business use case". There are too many options now. Do you use any tools to compare AI products or track new AI startups?
the “sites either miss models or can’t filter” frustration is painfully real. Shipping a searchable table of 300+ LLMs with price, context, and modality filters is the version of “I’ll just build it myself” that actually helps people.
Provider compare coming is the right next layer — model-level is how you shop; provider-level is how you commit.
What’s the first filter or column people ask for that you still don’t have — latency, rate limits, or something messier like “same model across APIs”?
Thanks!
What’s the first filter or column people ask for that you still don’t have — latency, rate limits, or something messier like “same model across APIs”?
If your question is about the tool itself, we already have all of them. Multimodel testing is coming soon.
For the table, not sure whether latency and such can be integrated at all there. Thoughts?
latency in a static price table is tough: it swings by region, time of day, and provider route, so a fixed column goes stale and can mislead.
I'd keep price/context as the core, then a measured badge linking to p50/p95 elsewhere — or an optional filter with a clear caveat that numbers are snapshots, not SLAs.
When multimodel testing lands, are you ranking same-prompt quality first, or cost/latency across providers of the same model?
Thanks for sharing your thoughts, definitely will think about it!
There are 2 modes: single and multi-model. In the single model mode, you create param sets (different temperature, reasoning effort and such) and run the test with the same system prompt and user input against them.
In the multi-model mode, you select different models (with their own parameters) and run the same system prompt + user input against them. In both cases an each individual test comes with latentcy, input/output token numbers, and calculated price (based on an original provider's price). Later, I may provide the calculation for other providers (based on the same token usage).
great, think it's helpful
This is actually a problem a lot of AI builders run into.
LLM pricing changes so quickly that comparing models becomes almost a product decision by itself.
The filtering aspect is probably the most valuable part — founders usually don’t just need the cheapest model, they need the best tradeoff between cost, latency, quality, and use case.
Would be interesting to see filters around things like:
best models for agents
coding
RAG/search
long context
structured outputs
Great resource. Keeping this updated is probably the hardest part.
Hi, thanks for your comment!
What you suggest is already in my mind! The whole point of the service is allow people to test LLMs and of course, we will do testing too and will publish the results of our testing. So, stay tuned!
This is a real gap, glad someone built it. We run multi-provider AI features and the pricing comparison part was way more annoying than it should've been, every provider formats it differently and cached/prompt-caching pricing especially gets left out of most comparison pages, like the other commenter mentioned.
Would be genuinely useful if "LLM Knights" eventually tracked cache pricing too, since that's often the difference between a model being cheap or expensive in practice, not just the sticker price per token.
Thanks for your suggestion!
Noted!
The column I would want is cached input price. Anything with a stable system prompt pays the cache-read rate on most calls, and that can be a fraction of base input, so two models sitting next to each other in your table can be far apart on a real bill. Do the free sources you are pulling from expose it?
Thanks for your suggestion, we definitely will provide this information soon.
the only person worth competing is the person you were yesterday
I agree 100%. Thanks for your comment.
llm pricing comparison is one of those things you assume exists properly until you actually need it and every page is missing half the models. building it because you kept hitting the wall yourself is usually how the useful ones start.
Exactly. It's still not perfect and not full, but I just tried to water to see if somebody else except me needs it :)