Couldn't sleep, Its Live. Launched 10 days early.
I just couldn't go to bed thinking about not being able to launch just because the payment button doesn't work.
So I launched with this promptperf.dev
This will allow users to signup and use the free testing which allows 3 test cases per run unlimited runs. So this is a good starting point. So i shipped it.
Its 12:30 now and just sent out the emails to all the early adopters, I will be manually upgrading each of these to Paid lifetime access for free forever. This is just a way to say thank you to all who supported by signing up on the waitlist.
Next steps:
- Get the payment button to work
- Gather feedback from early users
- Implement more AI models and increase price
- Only follow the roadmap to build a new feature for every new 100 users to ensure we get enough feedback and traction before building more features.
Nice project! Congrats!
I wonder what's the differentiator of this tool, compared to Anthropic's prompt improver.
It has some nice features, like generating the test cases—in my opinion, that was the hardest part of the job—and an evaluation tool that allows you to improve the prompt based on the evaluation of the test result.
I've been using that recently for my project. It's credit-based, so I pay only for the credits I use.
Now, what's anthropic is missing that I would be interested in having:
- It only supports anthropic-owned models, so that's really a blocker, and I don't think they ever will support OpenAI's models there, so something you are already planning to offer
- a CI/CD tool that will be easily integrated with the existing workflow of the codebase, so when someone from my team updates a part of the prompt, I want it to be tested in the Merge Request as a CI stage
- more advanced prompt optimization tool specially for input/output token optimization and smart suggestions, like "if you change your input/output format from this to this, in the long run you will save this many tokens"
- more robust test case generation tool, where I can have more control over what test cases will be generated, so as to not have trash/similar test cases that do not give value
- Model comparison tool, to test the same prompt with different models, and compare the results
- Probably the most important differentiator you can offer: An API so the system will push the real user interaction cases as a test case, and the ones which has failed or been reported by the user of a failed result, where your app will analyze those cases and make suggestions on what to change in the prompt to cover this cases, and also will track the success rate, and share a analytics dashboard of success rate over time.
Good luck!