Building with AI? Here's the problem:
Prompt A + Model B = Output C… until it doesn’t.
AI models are non-deterministic, which makes API-driven apps feel like Russian roulette.
So I built https://PromptPerf.dev
A prompt testing playground for serious builders.
🔍 Run prompts across multiple models
🌡️ Test across temperature levels
🔁 View consistency across multiple runs
📊 Compare outputs with your "ideal" answer
Because vague outputs break trust.
And in AI, trust is your product.
Waitlist’s open. Feedback appreciated :)
Interesting
Thank you
Cool!
Thank for that. Any particular aspect you really liked?
Cool idea !
Thank you :). Is this something you relate to?
yeah, we’ve definitely seen the non-deterministic gremlins creep in with AI outputs, it’s like rolling dice sometimes! For us, it’s a balancing act: guiding the model tightly while keeping the vibe coding flow.
Your testing playground looks like a killer way to tame that chaos !
That's a cool landing page 🔥,
Did you hard code it or used any no-code builder tool?
Ive been a developer for 4 years so I put that into AI agent to get me the base done and manually tweaked other bits via code.
ohh, makes sense. Nice one dude
very cool idea
Thank you for validation. This is really inspiring as it helps me understand:
This is key for me as Im building and solving this using my innovation skills seeing the problem first hand working in Innovation and AI
Integrability is key, if you can integrate this tool with other tools users already use, you'll get 100x the amount of users.
find your existing workflows and see where this tool would nicely fit, doesn't have to be its own saas could be an MCP for example.
Interesting concept. Haven't explored offering the service as API. Probably need to add to the potential scaling options
yah! This is cool - so it's more about matching the best model given your prompt structure then?
Yes, Each model can be configured to use different temperatures (0.1 means its more straight to the answer, and 2.0 for more creative outputs) but this means each setting can change the output. Now do you test which setting and on which model gives you the output desired.
Plus added complexity of will the output be consistent?
This is why the tool im building is aimed to resolve. It will run your tests for you:
-> User wants to test 3 AI models at 3 different temperature settings and has 3 prompts to test. They want to ensure and find out which settings will get them the most consistent answers -> so the model will return expected answers everytime and not just the first 3 times so they want to run each setting 10 times to test.
Problem: 3 Models x 3 Prompts x 3 temp settings x 10 runs = 270 runs and thats 270 API calls and manually recording outputs and calculating which one to select. You do this today but then you have a new feature so you do it again, now the model is no longer available and Google or OpenAI want you to use the new model so you do it again. Repeated process alot of manual testing.
you can try to enforce guard rails by asking it to only return its result in json for example. or parse the output and re prompt until it gets it right.
You can only do this a couple of times but to test across multiple different config and models and also running the same setting multiple times to ensure consistency is a pain. Why not automate prompt engineering like unit tests for software.
wow，so interesting idea
thanks mate. I believe as more Ai model get deprecated this issue will become more prominent. I work in Innovation and R&D and believe it will take 6-12 months for this to be a problem faced by every organisation who has built AI powered apps in the last 1 year
Cool!
Thanks, any feedback on the landing page?
Looks solid!
Cool idea. I'm looking forward to htis
Looks good
Thanks. Have you been exposed to this problem?
The link isn't working.
https://promptperf.dev/
Just checked it works.
Hmm, alright. It's probably my wifi or something. So what is the tool about? It's for developers, right?