Single model mode is roughly done
Today, I finished this part of coding: now, I can select a model and then add test runs to test this model against different parameters.
For example, I can change the temperature (if this model supports it), or level of verbosity or reasoning efforts.
Now, it looks this:
It still lacks some important features like the parameters that the tests were run with. But I will add it, soon.
The feature that would make this a daily tool for me is saved test suites I can re-run when a provider silently updates a model. We run LLMs in production at SocialPost.ai and the pain isn't finding the best parameters once, it's catching quality drift after we've shipped. Logging params per run is table stakes; re-runnable suites with diffs against the last run is the killer feature.
Wow, this is an interesting feature. So, by re-running do you mean just rerun the same combination of parameters, system prompt, and user input? Or are you talking about a multi-message conversations?
If the former one, it's already implemented! You can always click the re-run button to start a new test run, and it will grab the system prompt, user input, and all the param sets (which can contain different params for a single-model test set, or different models for multi-model test sets).
This is how it looks now:
Image — Postimages
For the conversational test sets, probably creating a copy of it could be a solution.
Hope, I answered your question :)
Every time I assign a task, it’s never fully completed. Something always gets skipped or ignored, even when I provide a detailed spec. I have to keep adding things later, which means more costs. It’s frustrating—it honestly feels like it's done on purpose just to bill more hours.
Are you talking about your experience with what?
One thing I'd add to the config-preservation idea: track cost per run alongside speed. I ran a multi-stage LLM pipeline in production (different models for different stages — a cheap/fast one for extraction, a stronger one for planning and a final quality check), and the parameter combo that scored best on quality sometimes cost 4-5x more per call than one that was ~90% as good. That gap only became visible once we logged cost per job with hard caps, not just latency/quality.
On the "why is this better" question: instead of trying to get the model to explain its own reasoning, we got more value out of a deterministic validator sitting outside the LLM entirely — plain rules that check the output's structure/constraints regardless of which model or params produced it. It won't explain the model's behavior, but it gives you a repeatable, non-probabilistic "is this run actually good" signal you can log and trend over time, even as the underlying models change.
Thanks for your suggestions!
We already show the cost, see here:
https://postimg.cc/r0Hgbz4K
As for validator, this is great idea, and I have it in my mind, too. I'm considering to implement it in 2 ways:
just with plain rules (like length of text, presence of specific words, json schema validation, and so on)
AI-based validator that for more complicated validation: like check the facts, or presence of the specific information, profanity check etc.
So, do you mean if the model changes, you will see it by running the same test again the validator, right?
"Unified playground" suggests comparison — but who's comparing? Researchers who need benchmarks, or product teams who need to choose a model? The researcher wants metrics and reproducibility. The product team wants speed and cost. Same playground, two different scoreboards.
Thanks for your question! Mostly, it's for founders and developers - which you call "product team", for choosing a right model. Speed is already among the metrics (you can see it on the screenshot), and cost is coming. Researchers can use it too, we can add more metrics in the future.
“Got it — makes sense focusing on product teams first.
One thing we’ve seen: even when speed + cost are clear, teams often struggle to decide because the ‘best’ model shifts by use case.
Would be interesting to test how different builders actually choose — happy to share what that surfaces.”
Totally! The sharing user feedback is already on the roadmap (with the test cases if they shared them too). We also will do tests different models x different test cases to demo what someone can even do with modern AI, should be fun!
I like the direction. Being able to compare the same model across different parameters feels more useful than just comparing different LLMs side by side.
Have you thought about helping users understand why one run is better than another for a given task, rather than leaving them to judge the outputs manually? That feels like it could be the real differentiator.
Hi, thanks for the question!
Interesting idea, but does someone really need to know why the result is better if they can just reuse the set of paramters?
Another thing is that I suppose it's not always easy to explain, especially with AI which is probabilistic and "deep mind" why a specific set of params gives the better results than others. But may think this direction, too.
That’s a fair point. I wasn’t thinking so much about explaining the model’s reasoning, but helping users build confidence in the result. If I keep coming back to the same task months later, I’d find it useful to know why I saved this configuration or what kind of tasks it consistently performs well on, even if the explanation comes from patterns in previous test runs rather than the model itself. Have you thought about preserving that context alongside the results?
Hmmm can you elaborate a bit more on "context"?
Right now, I store everything that can be useful: messages, prompts, params (like temperature, reasoning efforts etc.), latency, cost at the moment. Do you mean something else?
If you are talking about some user's comments, I will implement it soon as feedback - from it, you can see later, what you liked and how.
Btw, as an example of situation when it's hard to explain the result, look at this: today I was testing the params and reasoning efforts. The user input:
If I have 2 apples and John has 3 apples, and John gives me one apple, how many apples are there?
Model: o3
Test 1
reasoning {"effort":"low"}
Before any apples are exchanged, the total number of apples is:
• You: 2
• John: 3 Total = 2 + 3 = 5 apples John gives you 1 apple. This only moves an apple from John to you; it doesn’t create or destroy any apples.
After the exchange:
• You: 2 + 1 = 3
• John: 3 − 1 = 2 Total = 3 + 2 = 5 apples So, there are still 5 apples in all.
Latency 2.27 s
Input tokens 36 - 0.000072
Output tokens 198 - 0.001584
Total cost 0.001656
Test 2
reasoning {"effort":"high"}
There are still 5 apples in total. After John gives you one apple, you have 3 apples and John has 2, but the overall number of apples hasn’t changed.
Latency 2.06 s
Input tokens 36 - 0.000072
Output tokens 251 - 0.002008
Total cost 0.00208
Honestly, I can't explain this result. The only guess is the temperature is not 0, may be 1, and therefore, this result is not related with reasoning effort, but rather with randomness.
was thinking less about explaining why the model generated a specific answer and more about explaining why I saved that experiment.
For example, imagine I save a configuration with a note like:
“Works well for short customer support replies.”
or
“Reliable for structured JSON outputs, but slower and more expensive.”
Months later, I don’t have to remember why I bookmarked it—I immediately know when to reuse it. That kind of context comes from my own testing patterns rather than the model itself.
Your planned feedback feature actually sounds like it could evolve into something very close to this.
Ah, this makes sense. But I guess, it's already on the road map as the feedback feature. You will be able not only rate an every single test result by different criteria, but also leave the note to older yourlself!
Let me know if you mean something else :)
That sounds like exactly what I had in mind. I like that the notes and ratings become part of the experiment itself rather than something users have to remember separately. Over time, I could see that becoming a personal knowledge base of what works for different tasks. Looking forward to seeing how it evolves.
By the way, are you active on X, Discord, or Telegram? I’d love to follow along as you keep building.
Sure, you can find me here https://x.com/SeaCatWiz I'd be glad to connect!
Also, don't forget to leave your email because... Finally, decided on pricing and offer a free gift to all the early birds (read to know how) - Indie Hackers
balogunobalabiabdullahi@gmail.com
Awesome, thanks! Just followed you on X. Looking forward to seeing LLMKnights evolve. And thanks for the heads-up about the early bird offer I’ll check it out.