So I finally got the main feature working
I had shipped a MVP version of my product https://promptperf.dev I launched prompt testing with users API Key and only upload csv/json for the test cases.
Then I pivoted and made it so users can enter test cases on the app and also do bulk upload AND BIG PIVOT was to remove user API Key and allowed direct usage so I bear the API costs.
Now Im launching multi model runs. Heres a sneak peak of the dashboard. Please provide feedback if this looks good.
Going from “please paste your API key” to “I’ll pay for it myself” is wild founder energy. Somewhere Stripe is crying… but users are rejoicing.
After the launch the key metric for me changed from number of signups to number of evaluation performed so had to make the flow smooth to increase that.
looks good! Sounds I need it, I'll try it later.
I have official launch on uneed for this launch3 but the code is shipped to PROD if you want to check-it out early
Looks like a solid evolution — removing the API key barrier is a big win for UX. We’ve just launched an eCommerce starter kit ourselves, so I totally get how even small pivots feel massive behind the scenes. Dashboard’s looking clean too!
thanks mate. I just launched initially to get feedback and all this was then iteration to improvise so definitely slowly seeing users use the tool more
This is quite interesting
Thank you. Do check it out. I am launching the multi model testing in couple days.
Alright, I will be waiting for it too..
Why not having two options, bring your own key AND you offer the key.
Of you would offer a bring your own (ideally openrouter) key lifetime deal, I might have a closer look.
Or offer a plan like "official API cost plus 30% markup for you"?
Just thinking loud.
I tried OWN API key however from the 43 users who had signed up none ran evaluations. Once I shipped the removal of API key 43-60 so 17 users who signup 12 of them ran evals.
I do plan to introduce API Key but as for the current versions I think its too early. Its for technical advanced users who might have their custom models or dont want to route it through my API Key.
I still offer 3 test cases on unlimited evaluations so I can't see a clear reason a user might want their API key, As currently the costs are all on me.
Why am I taking the costs? I am just trying to ensure the product strategy and concept is validated I have calculated my costs and can bear 100-150 users running free evals on the site Im at 63 users as of now so still half way.