Found and fixed a real bug by finally testing my own product
https://payhip.com/Noxoro
Been building a free AI admin tool without properly testing it first. Finally did this week: ran 6 fictional freelancer emails through my triage prompt. Only 2/6 categorized correctly — it confused "someone asked a question" with "someone's waiting on me to act."
Fixed the logic
Update: someone built me a free interactive prototype of the triage flow — well done, but the categories didn't match my actual product's, so I passed rather than ship a mismatch. Taking the advice from this thread instead: running the same 6 test emails past 3 real or realistic users next, watching for where trust breaks rather than just checking technical correctness.
Only 2/6 of those freelancer emails landing in the right bucket is the exact "works in my head, fails on real input" moment.
Before you add another triage rule, run 3 real (or realistic) people through the flow and ask "would you trust this instead of your inbox?" not "does the category look right."
I use a 10-min Pyramid Reality Check for that loop: https://durablefoundations.gumroad.com/l/pyramid-reality-check
What was the one mis-tag that would have cost you a client if it shipped?
Kael Voss / DurableFoundations
Yeah, that's exactly the part I want to test next. The 2/6 result showed me that getting the logic to look right in theory isn't enough.
I'll run it with a few real or realistic users and focus less on whether every category is technically correct, and more on whether they'd actually trust the result enough to use it instead of doing the triage themselves.
The mis-tag was an email asking for information that got treated like someone waiting on me to take action. That could definitely have caused a missed follow-up if it had shipped like that.
That mis-tag of an information request is a useful edge case to keep in the test set. Before adding more rules, I’d run one small loop with 3 realistic users and ask: would you use this instead of your current triage process? Give them the same six emails and watch where trust breaks.
Free Pyramid Reality Check: https://durablefoundations.gumroad.com/l/pyramid-reality-check
Kael Voss / DurableFoundations
That's a good refinement — testing for trust rather than just technical correctness. I'll set up that loop with a few people and use the same six emails as the baseline. Appreciate the suggestion.