We pointed an AI monitoring agent at one question and left it running for six weeks

Watching Agents by Inithouse is a platform where you deploy an AI agent to watch any question about the future. The agent builds hypotheses, tracks evidence in real time, and alerts you when something changes. We have 117 public agent pages live today. Here is what one six-week run actually looked like.

The setup

We picked a question with a clear binary outcome and enough public signal to keep an agent busy: a regulatory deadline with known milestones and active press coverage. The agent started on day one with a probability estimate of 62% and a confidence score of 34 out of 100.

Both numbers matter. Probability is the agent's best guess. Confidence is how much evidence it has behind that guess. A fresh agent with no data starts at low confidence, even if its initial probability is already directionally right.

Week 1-2: Calibration phase

The first 14 days produced 11 evidence items. Most were news articles, official statements, and policy updates pointing in the same direction. Probability barely moved (62% to 65%). Confidence climbed from 34 to 51 as the agent accumulated corroborating evidence without contradiction.

This phase taught us something about monitoring design: an agent collecting evidence in one direction looks convincing fast. Confidence rises not because the answer gets more certain, but because nothing contradicts the initial read. That is both a feature and a failure mode.

Week 3: The false alert

On day 19, the agent flagged a probability drop from 65% to 48%. The trigger was a news article about a proposed delay in enforcement timelines. The alert hit our inbox within an hour.

Root cause: the article discussed delays in a specific subset of provisions, not the broader scope the agent was tracking. The agent weighted keyword overlap too heavily against the narrower context. We saw the same pattern in two other public agents tracking regulatory questions. Broad question plus narrow evidence equals noise.

We did not patch it. Hand-tuning relevance thresholds defeats the purpose of autonomous monitoring. We watched whether the correction happened on its own. Within 48 hours, probability was back at 61%.

Week 4-5: The plateau

Between days 22 and 35, only 4 new evidence items appeared. Probability settled at 67%. Confidence reached 71 and sat there.

This is the phase most one-shot prediction tools skip entirely. A single forecast would return "67% likely" and stop. The monitoring agent sat there, checking, finding nothing new, and not changing its mind.

That absence of change carries information. When nothing moves for two weeks, the question is cooling down. For anyone watching this regulatory space, the plateau itself was the signal: no new implementation news, no surprise amendments, no enforcement pilots. Silence, reported as silence.

Week 6: The late alert

On day 38, an official press release confirmed the outcome. The agent updated probability to 89% and confidence to 84. The alert arrived 11 hours after the press release went live.

Eleven hours. A human analyst scanning RSS feeds would have caught it in minutes. The delay came from the evidence collection cycle (running every 6 hours at that point) plus processing time. We later shortened the cycle to every 2 hours for high-confidence agents, but the tradeoff is compute cost per agent.

The public vs. private decision

Watching Agents by Inithouse supports both public and private agent pages. Public pages are indexed, shareable, and visible to anyone. Private agents are visible only to the creator.

We made this agent public from the start. Public agents build the evidence trail in the open. Anyone searching for the topic can land on the agent page and see the full hypothesis, every evidence item, the probability curve over time, and the alerts. The page is a living document that updates itself.

The cost: missteps are public too. The day-19 false alert is visible on the page. Anyone can see the agent temporarily dropped probability on a misread article. We kept it because hiding the miss would make the track record less trustworthy.

Of the 117 public agent pages live today, roughly 40% started as private and were switched to public after the first two weeks of stable behavior. That is the pattern we recommend: private for calibration, public once the agent shows it can self-correct.

Prediction vs. monitoring: what the run made concrete

| | One-time prediction | Monitoring agent |
|---|---|---|
| Output | A number (67% likely) | A number that moves over time |
| Evidence | Whatever the model knew at query time | Collected continuously, timestamped |
| False signals | You never see them | Visible and self-correcting (day 19) |
| Silence | Not reported | Reported as a plateau (weeks 4-5) |
| Delay | None (instant, stale) | Real (11 hours in our case) |
| Record | None | Full audit trail on the page |

The six-week run produced 19 evidence items, 3 probability alerts (1 false, 1 self-correction, 1 genuine), and a clear answer. The useful insight was not the final 89%. It was the two-week plateau that said the question had gone quiet, and the 11-hour delay that showed where our collection cycle was too slow.

What we would change

Three things, in order of how much they would have mattered during this run.

First, shorter evidence cycles for agents that cross 70% confidence. The 11-hour delay in week 6 is the direct cost of a 6-hour collection window. For agents approaching a resolution, 2-hour cycles are worth the compute.

Second, a prominence weight on source specificity. The day-19 false alert happened because a narrow-scope article matched enough keywords to move a broad-scope question. Scoring how specific the source is to the question scope would have caught it.

Third, a "quiet period" indicator in the UI so users can see at a glance that an agent has been stable for N days. Right now you have to scroll through the evidence timeline to notice a plateau. A visual marker for "nothing has changed in X days" would make the silence legible without reading every item.

If you want to try it: watchingagents.com. Pick a question, deploy an agent, and leave it running. Some questions resolve in days. The ones worth monitoring are where you need to know when something changes, not just whether it will.