AgentWatch
Catches when your AI agent quietly gets worse
I'm building AgentWatch. Simple version of what it does:
If you've shipped an AI agent (chatbot, assistant, whatever), you probably have monitoring for your servers and your code. But if the AI itself starts giving worse answers more confused, more expensive, less accurate nothing tells you. No crash, no error, no red flag. You usually find out from a customer complaint, way after the fact.
AgentWatch watches your agent in the background and gives it a health score. When that score starts dropping in a real, sustained way not just one bad response, an actual pattern it alerts you. No manual setup, no writing eval rubrics. It just learns what "normal" looks like for your specific agent and flags it when something's off.
Who it's for: anyone who's shipped an AI agent into production and has zero visibility into whether it's actually doing a good job right now.
Where I'm at: SDK works, scoring pipeline works, drift detection works. I ran it against a test agent fed it clean questions (health score 90+), then fed it confused/vague input, and it caught the drop and alerted automatically.
What I don't have: real pilot users yet. This is the part I'm stuck on.
If you're running any kind of AI agent in production even something small I'd genuinely love for you to try it and tell me what's wrong with it. Free, no strings, I'll help you set it up myself. Brutal feedback especially welcome.
Comment or DM if you want to try it.
1 Comment
The gap between infrastructure monitoring and agent-quality monitoring is very clear here. A system can be completely healthy technically while the thing users experience is quietly getting worse.