Why we built Watching Agents around evidence, not predictions
At Inithouse, a studio shipping a growing portfolio of products in parallel, we keep running into the same question from users: "So does this thing predict the future?"
No. And that distinction shaped every architectural decision in Watching Agents.
The prediction trap
Early in the build, we prototyped what everyone else ships: a confidence score. The agent watches a question ("Will competitor X raise prices this quarter?"), processes signals, and returns a number. Ours showed 0 to 100. Clean, simple, exactly what users expected.
It also meant nothing.
Here is what happened. One of our test agents tracked a SaaS competitor's pricing page. It picked up two blog posts mentioning "delivering more value" and a job listing for a pricing analyst. The confidence score jumped to 74%. A user screenshot that number and shared it internally as justification for a preemptive price change of their own.
The competitor did nothing. The blog posts were generic content marketing. The job listing was a backfill.
We had built a tool that manufactured certainty from noise. That was the moment we ripped out the score.
What replaced it: evidence, not numbers
The core model now has three parts, and none of them output a prediction.
Hypothesis state machine. Every agent starts with a hypothesis the user defines: "Competitor X will raise enterprise pricing before Q4." That hypothesis moves through states: proposed, tracking, supported, weakened, confirmed, refuted, or expired. No probability attached. The state reflects what the evidence says so far, not what we think will happen.
Evidence log. Each piece of evidence the agent collects gets timestamped, sourced, and tagged with direction (supports or weakens the hypothesis). A pricing page screenshot from June 3 that shows no change is evidence. A job posting for "Head of Monetization" is evidence. Neither gets collapsed into a score. The user sees the trail and judges weight themselves.
Change detection over scoring. The agent alerts when something changes, not when a number crosses a threshold. New evidence that shifts the hypothesis state triggers a notification. A week of silence is itself a signal: "still tracking, nothing new." We found users made better decisions when they saw a timeline of evidence than when they saw a number tick up or down.
The decision tree behind this
We had three options on the table during the architecture sprint:
Option A: Probability scoring. Standard approach. Feed signals into a model, output confidence %. We tried it, saw the false certainty problem above, killed it after 3 weeks.
Option B: Sentiment aggregation. Score signals as positive/negative/neutral, show a running tally. Better than raw probability, but still collapses nuance. A "negative" signal about a competitor might be negative for them and positive for the user's hypothesis. Context matters. Tally doesn't capture it.
Option C: Evidence timeline with hypothesis states. What we shipped. More complex to build (the state machine alone took longer than the entire Option A prototype), but it matches how analysts actually think. You gather evidence, you weigh it in context, you update your view. The tool tracks; the human decides.
We picked C. It cost us roughly 2 extra weeks of build time, but the first cohort of users stayed engaged 3x longer than our Option A testers. They came back to check the evidence log. With Option A, they checked the number once and forgot about it.
What we learned from our other "watching" products
This pattern (track, show evidence, let the user decide) showed up across our portfolio before we named it. At Be Recommended, we built an AI visibility report that tracks how chatbots recommend brands. Early versions tried to score visibility as a single number. Same problem: users fixated on the score and ignored the evidence underneath (which citations appeared, which models mentioned them, what context triggered the mention). We restructured to show the evidence trail first, score second. Engagement went up.
At Audit Vibe Coding, the tool watches AI-generated code for issues. The first version flagged "high/medium/low risk." Users ignored medium and low entirely. When we switched to showing the specific evidence (this function has no error handling, this API key is hardcoded on line 42), users fixed more issues across all severity levels.
The pattern: when you show people the evidence, they act on it. When you show them a score, they either panic or ignore it.
Architecture details for builders
If you want to build something similar, here is what cost us the most time:
State transitions need to be auditable. Every move from "tracking" to "supported" has to reference the specific evidence that triggered it. Without this, users lose trust fast. "Why did it change?" is the first question everyone asks.
Evidence deduplication is harder than it sounds. The same news article gets picked up by 5 sources. The same pricing page gets screenshotted weekly with no changes. We spent a full week on dedup logic alone.
Expiry matters. A hypothesis without a time bound ("Will X ever raise prices?") never resolves. We force a time horizon on every agent. When it expires without resolution, the hypothesis state moves to "expired," not "refuted." That distinction matters to users.
Where this sits now
Watching Agents has public agents anyone can follow and private agents users deploy for their own questions. The public agents double as content (each one produces a readable evidence timeline that search engines index). We observed that the public agent pages get crawled within 48 hours of evidence updates, which validates the "living content" thesis we had going in.
We are still iterating. The biggest open question: should the system ever surface its own assessment, or should it stay purely as an evidence collector? Right now we lean toward evidence only. Every time we tested adding interpretation, users deferred to the machine instead of thinking through the evidence themselves.
At Inithouse, a lab building many products at once, that is the thesis we keep landing on: tools that collect and organize, not tools that decide for you. Watching Agents is the purest expression of that so far.
If you are building monitoring or tracking tools, we would be curious what you have found. Does showing raw evidence work for your users, or do they demand a score?
The part I’d be careful with is not “never summarize,” but “never summarize without a path back to the evidence.”
A score creates false certainty because people treat it as the answer. But raw evidence alone can also become heavy once the timeline gets long. The useful middle layer might be a short state explanation like: “Supported because X changed, Y repeated, and Z stayed unchanged,” with each reason linked to the exact evidence item.
That keeps the human in charge, but saves them from rereading the whole trail every time. The auditability point feels like the real product principle here: every state change needs to be arguable, not magical.
We ran into the same tension building NeuralCore. Our first instinct was "just give people a trust score," but a bare number without context creates the same false-certainty problem you describe.
What's worked for us is showing both: a consensus/agreement score from querying 7 models as the quick filter, plus the raw individual model answers underneath so people can see exactly where they diverge when it matters.
Most users skim the score, but the ones asking real questions (medical, legal, financial) always drill into the raw answers — so I'd say build for both audiences, not one or the other.