Built a tool that catches when AI agents quietly get worse — looking for 2-3 people to try it free
I'm building AgentWatch. Simple version of what it does:
If you've shipped an AI agent (chatbot, assistant, whatever), you probably have monitoring for your servers and your code. But if the AI itself starts giving worse answers — more confused, more expensive, less accurate — nothing tells you. No crash, no error, no red flag. You usually find out from a customer complaint, way after the fact.
AgentWatch watches your agent in the background and gives it a health score. When that score starts dropping in a real, sustained way — not just one bad response, an actual pattern — it alerts you. No manual setup, no writing eval rubrics. It just learns what "normal" looks like for your specific agent and flags it when something's off.
Who it's for: anyone who's shipped an AI agent into production and has zero visibility into whether it's actually doing a good job right now.
Where I'm at: SDK works, scoring pipeline works, drift detection works. I ran it against a test agent — fed it clean questions (health score 90+), then fed it confused/vague input, and it caught the drop and alerted automatically.
What I don't have: real pilot users yet. This is the part I'm stuck on.
If you're running any kind of AI agent in production — even something small — I'd genuinely love for you to try it and tell me what's wrong with it. Free, no strings, I'll help you set it up myself. Brutal feedback especially welcome.
Comment or DM if you want to try it.
The pilot test I'd run is against a saved 'known good' transcript set from the first weeks. A health score learns what normal looks like, but the fastest signal for an early product is whether recent responses on the same user tasks start changing shape: different wording, different result structure, higher token counts. That catches drift without pretending one rubric can define quality across every agent. I'd frame the sign-up around that baseline, because that is the part people already have but are not recording.
This solves a problem that’s only going to get bigger as more businesses rely on AI agents. The fact that an agent can still technically “work” while the quality quietly degrades makes monitoring especially tricky. Getting a few real users testing it now sounds like a smart way to find the failure cases you wouldn’t catch internally.
The problem is unusually easy to understand: an agent can degrade without producing any technical error. That makes the gap between system monitoring and actual product quality very clear.
AI drift is invisible until it costs you. How are you measuring "getting worse" — manual evaluation or automated benchmarks?
I'm building an AI thumbnail scoring tool, launching Aug 25. Happy to test yours if you're open to testing mine.