My AI agent could "see" the fact — it just never wrote it down

That's the same direction I'm already moving in — minimum gates per criterion, not just an overall number. What made this one persuasive is that the summarized score genuinely read as "seems fine," and it took slicing by question instead of by the total to find the one gate failing almost every run. Curious how Agiloop decides which criteria get a hard gate versus which stay folded into the summary score — is that a fixed list per assessment type, or tuned per engagement?

Not memory/context — nothing was truncated or fell out of a window. It's squarely the second one: how the agent decides what's worth recording in the first place. The minutes-summarization step had an extraction schema tuned for numbers and named entities, and this fact (no in-state competitor, stated without naming one) didn't match either. It wasn't dropped after being noticed — it was never flagged as recording-worthy to begin with. Same story one step downstream, too: the competitor-analysis node's own classification only had two buckets (industry averages / named-competitor data), and a fact about relative position without naming a competitor fit neither. So less a memory failure, more a taxonomy failure, twice in a row.

This is a great comment, going through each point:

The extract-everything-into-a-pool-then-classify-with-an-uncategorized-bucket approach is better than what I've been doing. My current method is additive — widen the schema each time a gap surfaces — which means anything that doesn't fit yet gets silently dropped, and I only find out by manually walking the intermediate state after something already looks wrong. An uncategorized queue makes the gap visible immediately instead of requiring me to go looking for it. That's a real fix for the "there's probably a 5th pattern" problem, not just a patch. Reworking extraction around this instead of hand-adding a 4th and 5th category.
Fair catch, and sharper than the doubt I already had. I flagged the English-vs-Japanese trip rate as "not sure if it's language-specific or noise," but hadn't isolated that I wrote the English case myself — that's a real confound, separate from sample size. The grader doesn't see my prompt or intent, but I honestly can't say right now whether it's blind to case authorship or pre/post-fix status the way it should be. Need to check that before I put any weight on the 3-of-5 number.
Agreed on n=3. "Full marks all 3 times" reads more conclusive than it is — should be reporting the range next to the mean by default, and probably shouldn't have called it "fixed" off three runs.

And appreciate the note on the matrices — that's the whole point of publishing them, glad it's landing as something to actually check rather than just a claim to take on faith.

Tracing the drop, honestly — not fixing it, finding it. The fact wasn't just misclassified somewhere visible, it vanished before it ever reached the node whose job was to handle it. Had to walk the intermediate state output node by node until I found the first point where it was already gone, which turned out to be the very first summarization step. If I'd only looked at the node "responsible" for competitor analysis, I'd have never found it, because by the time execution got there the information simply didn't exist anymore.

Tracing the drop, honestly — not fixing it, finding it. The fact wasn't just misclassified somewhere visible, it vanished before it ever reached the node whose job was to handle it. Had to walk the intermediate state output node by node until I found the first point where it was already gone, which turned out to be the very first summarization step. If I'd only looked at the node "responsible" for competitor analysis, I'd have never found it, because by the time execution got there the information simply didn't exist anymore.

This is a side project around a full-time day job, so realistically it's evenings and weekends — not a fixed number, more like whatever's left after work on a given week. Some weeks that's a few focused hours on one bug (like the one in this post), other weeks it's mostly writing up what I found rather than building. Not sustainable at startup-hours pace, but enough to keep the validation loop moving.

Yeah, "fail loudly" is the key phrase — right now a disappearing fact fails silently, it just quietly lowers an average three questions later. Invariants at every node instead of one end check is the fix. And glad it's not just me — the hardcoded downstream list is the kind of bug that makes you feel dumb for an afternoon and then relieved when you realize it's a common shape of mistake, not a personal one.

Per-node contracts is a sharper version of what I've been doing by hand — tracing intermediate state after I already suspect something's off, instead of asserting it automatically at each step. That's exactly why the minutes_ingest drop sat unnoticed as long as it did: nothing flagged it, I only found it because I went looking.

And splitting coverage / fidelity / final usefulness into separate scores instead of one blended number is a real correction. I've been treating "did the required fact survive" and "was it represented correctly" as basically the same signal, and they're clearly not — the mystery-shopper fact failed on coverage, but a fact that survives and gets subtly distorted downstream is a different failure mode entirely, and right now I'd have no way to tell those apart. Building the fixture set with that split in mind from the start, not bolting it on after.

The Genie example is a really clean parallel — 12% of real traffic silently failing because the category wasn't even in the eval breakdown is the same shape of bug, just found from the production side instead of the eval side. Good reminder that this isn't a "my pipeline" problem, it's a "aggregate metrics" problem.

And yes, hit that exact hardcoded-formatter trap — fixed the upstream extraction, reran, output didn't change, spent longer than I'd like to admit assuming I'd made an error before tracing it down to a downstream node silently dropping the new category. Invisible until you manually walk the intermediate state, agreed.

On the coverage test — that's exactly where I'm heading next: a canonical case per fact type, asserted explicitly, so a new gap shows up as a red test instead of a quietly-lower average three weeks later. Don't have it built yet, but this thread is basically talking me into prioritizing it over the next validation round instead of after it.

Honestly, "since launch" undersells where I'm at — it's still invite-only, no public launch yet. So the biggest challenge so far isn't a launch-day fire, it's the gap this thread already poked at: getting from "passes my own rubric" to "actually useful for someone who isn't me." Everything's been validated against exam scoring criteria, not against a real person using it for their own work, and I don't yet know how those two diverge.

Thanks for reading it — glad the numbers made it easier to follow rather than just more noise to wade through.

You're right, and this is actually the direction I'm already moving toward — tracking pass/fail per required element instead of a summed score, so a 90 with one hard zero doesn't get averaged into "looks fine." The CAC analogy is a good one; that's exactly the kind of blending that let this sit unnoticed for longer than it should have. Going to make the gate explicit rather than something I have to remember to check manually each time.

Not with real users, no — everything so far has been tested against exam rubrics and scoring criteria, not against someone actually using it to do their job. That's the gap I'm aware of and haven't closed yet. The early-access program is partly meant to fix that: get it in front of a few people who'd actually use this kind of report, see where it breaks in practice rather than on paper.

Good question — honestly the part I'm least confident about. The 3 post-fix runs were on the same case I used to find the bug in the first place, so this doesn't rule out just overfitting to that one pattern. What would actually convince me is running the same fix against a completely different case (different industry, different competitive structure) and checking the miss rate holds up there too — haven't done that yet, it's next on the list, along with building a proper held-out test set instead of eyeballing individual runs. Will post an update once I have real numbers instead of "seems better."

Thanks, glad it was useful! Honestly still very early on the user side — no public launch yet, just reaching out directly to a handful of people who actually write this kind of report for a living (mostly independent consultants) and asking them to try it. Wanted the failure modes ironed out first before pushing on distribution. This post is partly that too — seeing if the story holds up with people outside my own bubble before I put more effort into finding users.