I Tested 10 Real AI Workflows Across Multiple LLMs for CoTel. My Biggest Surprise Wasn't the Winner
I spent the last few weeks working on what I thought would be a fairly boring problem:
rebuilding the pricing system for my AI SaaS that analyzes Telegram chats and communities.
When I launched, pricing was simple:
X requests per day, Y requests per month.
At the time it seemed reasonable. I had one primary model, very few users, and no real production data.
Then I started testing multiple LLMs on real user workflows.
That's when I realized that the concept of "one request" doesn't really mean anything.
One request might consume 1 unit of compute.
Another might consume 50.
Yet both look exactly the same from the user's perspective: they just typed a question into a text box.
To understand what was actually happening, I built a testing framework and ran the same workflows across different models using real Telegram chats and channels.
The workflows included:
- finding recommendations in discussions
- weekly discussion digests
- sentiment shifts over time
- conflicting opinions and debates
- source and link analysis
- cross-chat analysis across multiple communities
- practical planning and checklist-style tasks
I expected to find the best model.
Instead, I discovered that there isn't one.
Some surprising observations:
• The model that produced the best action plans and checklists was not the model that produced the best summaries.
• Some premium models were 10–15x more expensive while providing only marginal quality improvements.
• Reasoning-focused models performed well on structured tasks such as "Position A vs Position B", but not necessarily on messy human conversations.
• The strongest model on long-context synthesis unexpectedly failed one of the source credibility tasks.
• On several workflows, cheaper models performed so close to premium ones that using the expensive option simply didn't make economic sense.
My biggest takeaway was this:
The problem wasn't pricing.
The problem was that I was trying to build a product around a model.
What I actually needed was to build it around the type of task being solved.
As a result, instead of simply redesigning billing, I'm now building a routing layer that decides which class of model is best suited for a particular request.
It's funny how this started as a pricing exercise.
And ended as an architecture exercise.
Has anyone else gone through a similar evolution?
Did you start with a single-model AI product and eventually realize you needed model routing?
Also, would it be interesting if I shared a follow-up post with the actual models I tested and some of the more surprising findings from those benchmarks?
