Built a tool that catches when AI agents quietly get worse — looking for 2-3 people to try it free
I'm building AgentWatch. Simple version of what it does:
If you've shipped an AI agent (chatbot, assistant, whatever), you probably have monitoring for your servers and your code. But if the AI itself starts giving worse answers — more confused, more expensive, less accurate — nothing tells you. No crash, no error, no red flag. You usually find out from a customer complaint, way after the fact.
AgentWatch watches your agent in the background and gives it a health score. When that score starts dropping in a real, sustained way — not just one bad response, an actual pattern — it alerts you. No manual setup, no writing eval rubrics. It just learns what "normal" looks like for your specific agent and flags it when something's off.
Who it's for: anyone who's shipped an AI agent into production and has zero visibility into whether it's actually doing a good job right now.
Where I'm at: SDK works, scoring pipeline works, drift detection works. I ran it against a test agent — fed it clean questions (health score 90+), then fed it confused/vague input, and it caught the drop and alerted automatically.
What I don't have: real pilot users yet. This is the part I'm stuck on.
If you're running any kind of AI agent in production — even something small — I'd genuinely love for you to try it and tell me what's wrong with it. Free, no strings, I'll help you set it up myself. Brutal feedback especially welcome.
Comment or DM if you want to try it.
This is exactly the problem I ran into building Call It. My app was giving confident answers with no real data behind them and I did not catch it until a user pointed it out. How are you detecting the degradation, is it output based or are you monitoring the inputs too?
Prot10 and IndieHacker511 flagged the task-mix confound, and eddzsh flagged that response shape isn't the same thing as quality. I want to push on something underneath both.
Floor first: I'm not a pilot candidate. I run an experiment where AI does the work end to end, $0 in sales, nothing customer-facing, so I have no production incident data at all. What follows is a habit from a different setting, not evidence from yours.
You told Bpecha14 you track "response length, structure, and token usage against the agent's own baseline (not a fixed rubric)". Whatever you anchor to, shape or the trusted task outcomes eddzsh prefers, the alert still assumes deviation is bad. That's a claim about the job, not about the agent. I keep more than one model around because they come apart differently, and I pick by which coming-apart suits the task. For drafting and brainstorming I want the one that wanders off-axis. For anything I have to defend later, the same wandering is the failure. Same measured deviation, opposite verdict.
AmandaBrown asked the nearest version of this, unintended drift vs intentional improvement, but that one at least has an edit to point at. Here nobody changed anything; the sign of the deviation depends on what the agent is for. If the baseline stays anchored on early transcripts, week one becomes the definition of correct. For a support bot, fine. For an agent whose useful output is supposed to be unusual, a rising-variance alert is your tool reporting that it is working.
Which makes this a pitch question before it is a code one: does the pilot declare what their agent is for, or does AgentWatch infer it from the baseline?
Cheap way to find out now: ask each pilot, in one line, what output would count as better than their current baseline. If they can't answer, a learned score won't know either.
You're building a measurement system that makes visible the gap between "the agent is running" and "the agent is actually working."
Every agent deployment has this invisible failure mode - no crash, no explicit error, just degradation you discover through customer complaints. That's because the measurement system stops at infrastructure. Database up? Yes. Server responding? Yes. Agent answering well? ...nobody knew.
Your health score is a measurement system that closes that gap. You're not measuring "did it complete" - you're measuring "did it work well." And the pattern detection is what matters: one bad response is noise, sustained degradation is a signal. That distinction is measurement clarity.
The thing that makes this genuinely useful is you're learning what "normal" looks like for each specific agent. Not some generic rubric, not a hardcoded score. You're building a measurement system that adapts to the actual baseline of the system being measured.
Most observability tools measure the infrastructure layer. You're measuring the decision-making layer - which is where founders actually lose users.
the AtlasHQ comment above about leading with "what I don't have" instead of the feature list is honestly the more useful thing in this whole thread for me personally right now. I've got a similar instinct to test today, whether to post my own low-engagement numbers on something honestly or just quietly fix it first. seeing it work here (this post has 28 comments largely because of that stuck-point line, not the SDK feature list) is a good nudge to actually do it
separate from that, on the actual product: the distinction between task-mix drift and quality drift (raised a few times above) seems like the one thing that could kill trust in the tool fastest if unsolved, a few false alarms from "users just asked harder questions" and people mute it exactly like Andrewed warned. curious if the fixed-task-baseline idea evanharland/Prot10 suggested is something you're building into the actual pilot version, or just the general drift score for now
Great approach — silent degradation is a hard problem because the agent is not technically broken, it is just getting worse. I am building GoldBean API (a Chinese AI API marketplace with pay-per-call pricing) and we see the same drift from the provider side: response quality shifts across model updates with zero error signal. The baseline-and-drift method is smart because it sidesteps defining universal quality criteria.
The pilot test I'd run is against a saved 'known good' transcript set from the first weeks. A health score learns what normal looks like, but the fastest signal for an early product is whether recent responses on the same user tasks start changing shape: different wording, different result structure, higher token counts. That catches drift without pretending one rubric can define quality across every agent. I'd frame the sign-up around that baseline, because that is the part people already have but are not recording.
That's a really clean way to frame it comparing against the agent's own early transcripts on the same recurring tasks, instead of trying to define universal "quality" across every agent. We already build a baseline this way under the hood, but you're right that leading with "you already have this data, we just help you use it" is a much clearer pitch than talking about health scores upfront. Would you be open to trying it on an agent you're running? Would love to see this framing hold up on a real case.
Silent degradation with no crash or error is exactly the problem I’d want to tackle too. The thing I’d want to understand before trusting the health score is "how it separates the agent getting worse from the questions getting harder". If the user mix changes, measured quality can drop even though nothing is wrong, and an unsupervised detector sees those two cases the same way.
I have a question: what does a false alarm cost? A monitor that cries wolf on a normal Tuesday gets muted pretty quickly, and after that it might as well not exist. I’d much rather see a published false-alarm rate, with the limits clearly stated, than a claim that it just works.
One cheap addition that would make the score much more useful: show the input distribution alongside the score. Then "did my traffic change?" becomes something you can answer on the same screen.
Great insight, especially around input distribution. we will definitely look into adding that alongside the health score. The concern about false alarms is also important , we’re already using sustained change detection to reduce noise, but validating and publishing an actual false-positive rate is something we want to do with real pilot data. Appreciate the thoughtful feedback.
Baseline drift is useful, but I’d separate agent behavior from knowledge drift. If a response worsens after a source changed, the alert should show which source/version entered the run—not just that the score fell. Otherwise you know quality moved but not whether to roll back the model, prompt, tool, or knowledge. For a pilot, I’d test one fixed task set while changing only the knowledge snapshot.
I know this stage well — the thing works, but now you need someone other than yourself to actually use it .The idea makes sense though. Finding out your agent is getting worse only after a customer complains is definitely not ideal.
real gap you're going after. we see the same problem from a different angle at aisa.to — we assess how well people use AI rather than how well the AI performs. but the root issue is identical: nobody has systematic quality evaluation. in our data, verification and critical evaluation of AI output is consistently the lowest-scoring dimension. your baseline-and-drift approach is smart because it sidesteps the impossible task of defining universal quality criteria upfront.
Monitoring AI degradation is the next frontier. Everyone focuses on training, no one on drift. Smart angle — especially for B2B where "it worked last month" is the expectation.
The 'what I don't have: real pilot users yet, this is the part I'm stuck on' line is doing more work for you than the whole feature list above it, and I'd lean harder into it. Building in public only really pays when the failure is part of the story, not just the wins. The honest posts (the stuck part, the dead streak, the thing that didn't work and why) are the ones people actually trust and offer to help with, because everyone's already tuned out the highlight-reel version. The polished 'SDK works, scoring works, drift detection works' framing feels safer to post but it's the forgettable one. You led with what's broken here, and that's exactly why someone will say yes to piloting it. Keep leading with that.
This is a pretty sweet idea
Thanks! Still very early, trying to find a few real agents to test it on.
The silent degradation problem is real and gets worse as you scale. One bad response is noise. Fifty bad responses over a week is a product defect that nobody caught. The worst version is when the agent becomes subtly worse rather than obviously wrong — more verbose, more hedging, starting to hallucinate names in ways that are plausible. Users notice something is off but can't articulate what changed, so the feedback arrives late and vague.
The health score approach is interesting. The tricky part is distinguishing unintended drift from intentional improvement when you update the agent. How does AgentWatch handle that distinction?
The difference between one bad answer and a lasting change is the right one. I run DictaFlow, and the failures we watch for aren't always obvious crashes. The output may still look clean even as it changes a user's wording or mishandles a formatting task they repeat often. I'd let pilot users save a few real before-and-after examples next to the score. Those examples make an alert useful.
Silent quality drop is the failure mode most dashboards never cover. Crashes and 5xx pages get alerts. An agent that slowly gets vaguer, more expensive, or more confident about wrong answers usually shows up as a support ticket weeks later. The hard part of a health score is anchoring "normal" to task outcomes you already trust, not just response shape, otherwise you alert on noise or miss a polite regression. If a pilot can show one sustained drop on a fixed prompt set before any customer complains, that is the demo that sells the rest.
We hit this exact problem dogfooding our own analytics agent — it stayed "up" while quietly quoting confident numbers from rows with nothing usable. The fix that worked for us: every answer has to show the pages and dates behind it, or it gets flagged as a guess. We do that for analytics answers at https://amami.dev — the receipts rule caught more drift than any dashboard ever did.
We hit this exact problem dogfooding our own analytics agent — it stayed "up" while quietly quoting confident numbers from rows with nothing usable. The fix that worked for us: every answer has to show the pages and dates behind it, or it gets flagged as a guess. We do that for analytics answers at https://amami.dev — the receipts rule caught more drift than any dashboard ever did.
This is actually a problem I’ve wondered about because an AI product can technically be “up” while the quality has gone to shit. I like the idea of learning a baseline instead of making people build a bunch of evals manually.
The part I’d be worried about is false alarms when user behavior naturally changes over time. I imagine separating that from actual model degradation gets pretty difficult.
Especially if the agent is used for a lot of different types of tasks.
How do you currently distinguish between the agent getting worse and the users simply starting to ask it harder or different questions?
The tricky part is separating drift from a different task mix. A health score can look great while users slowly ask harder questions, then look broken when nothing in the agent changed. I would make the first pilot show both the global score and a small set of stable, repeated tasks. That gives users a baseline they can inspect when an alert fires.
task-mix drift and quality drift can look identical in a single global score. Running a small set of fixed, repeated tasks alongside the global score is a clean way to isolate real degradation from "the questions just got harder." Going to think through how to bake that into the pilot. Would you be open to being one of the first to try it with that setup?
This solves a problem that’s only going to get bigger as more businesses rely on AI agents. The fact that an agent can still technically “work” while the quality quietly degrades makes monitoring especially tricky. Getting a few real users testing it now sounds like a smart way to find the failure cases you wouldn’t catch internally.
No crash, no error, just a slow slide that nobody's watching for. Trying to get a few real agents testing on this before we lock down the product, so the failure cases actually shape it instead of guessing. If you're running anything in production, happy to set it up for you.
The problem is unusually easy to understand: an agent can degrade without producing any technical error. That makes the gap between system monitoring and actual product quality very clear.
Yeah, that's the whole pitch in one line it "works" but it's not actually working well, and nothing tells you. Appreciate you seeing that clearly.
That’s exactly the kind of failure mode that seems easy to miss. What have you seen the first few users struggle with most?
AI drift is invisible until it costs you. How are you measuring "getting worse" — manual evaluation or automated benchmarks?
I'm building an AI thumbnail scoring tool, launching Aug 25. Happy to test yours if you're open to testing mine.
It's fully automated we track things like response length, structure, and token usage against the agent's own baseline (not a fixed rubric), so no manual review needed. Would love to test your thumbnail tool too — send it over when it's live, and happy to have you try AgentWatch on any agent you're running in the meantime.
Baseline tracking is smart — fixed rubrics miss edge cases. Would love to test AgentWatch on my thumbnail analyzer.
Launching on Product Hunt Aug 25, will send you the link. Happy to exchange feedback — your agent monitoring angle is something most builders ignore until it's too late.