My AI agent could "see" the fact — it just never wrote it down

Probably the rubric element that turned out to be satisfied by the wrong fact. We went looking for whether a specific competitor-benchmarking detail showed up in the report, expected a clean pass/fail, and instead found the grading criterion was being satisfied by an entirely different fact that happened to also qualify. So the score said "pass," the criterion was technically met, and the actual thing we cared about had never shown up at all. It wasn't hiding a failure so much as finding a legitimate-looking substitute to hide behind — which is a sneakier failure mode than I expected going in.

Second most surprising: that fact still doesn't show up even after two separate rounds of fixing the extraction pipeline. Each fix closed the specific gap we found, but the fact kept dying somewhere else. That's the part that convinced me point-patches weren't going to cut it and pushed us toward the bigger rework that's happening now.

Two of your three points are already shipped, one is exactly what we're working on right now — which is a good signal that this is the right shape of fix, not just our own pet theory.

"Fail the run when a required element is missing instead of letting a strong average mask it" — done. We shipped per-element pass/fail gating a few days ago specifically because we found a run where the total looked fine but a specific element had failed on every single attempt, hidden entirely by the average.

"Keep the missing fact as a regression fixture" — also done, same batch of work. We registered the exact fact from our root-cause writeup (a specific competitor-benchmarking practice that kept getting dropped) as a permanent held-out check. It still fails today, on purpose — the fixture's job is to stay red until the underlying extraction is actually fixed, not to get quietly patched away.

"Source span, category, and confidence as an explicit handoff contract" — this is the piece we don't have yet, and it's next. Right now our fact-check is closer to a boolean (keyword present or not) than a structured contract with confidence. We're mid-way through pulling the grounding-check logic into its own module, and your phrasing of "handoff contract" is a cleaner way to describe where that's headed than what we had in our own notes. The confidence dimension specifically isn't something we'd built into the design yet — worth stealing.

Same shape, different layer. Our version: a rubric element would show as "satisfied" in the aggregate score, but when we dug into why, it turned out to be satisfied by a completely different fact than the one we thought we were testing for. The score measured "did a qualifying fact show up somewhere" rather than "did the specific fact we cared about show up" — so a real gap sat underneath a passing grade for who knows how long, exactly the way your tool-call example sat underneath a "looks fine" answer.

Your framing of the fix is basically identical to where we landed too: you had to log the actual call per turn instead of trusting the final answer; we had to start tracking claims against the specific source span that grounds them instead of trusting the final rubric score. Both are the same move — stop scoring the artifact at the end and start requiring the intermediate step to be provably real. We're mid-way through pulling that grounding check into its own module right now, partly because "trust the aggregate score" is exactly the failure mode that bit us here.

Curious how you're logging the tool calls — full request/response per call, or just a call-happened/call-didn't-happen boolean? We went with the richer version (keep the actual source span, not just a pass/fail flag) because a flag alone doesn't tell you which claim it validated, and that's usually the thing you actually need when you go looking for what broke.

Fair hit on the fixture format — right now the entry just points back at the full source document, not a minimal isolated repro. Six months from now that's exactly the "wall of booleans nobody dares delete" problem you're describing, because nobody will want to re-derive why that line exists. Pulling the minimal input + expected fact into the fixture itself, versioned together, is a straightforward fix and we'll do it that way going forward.

The ledger idea is the sharper version of something we'd already sketched (tracking facts as retrieved/cited/omitted instead of a binary), but your framing closes a gap ours didn't: we were going to track three states internally, you're requiring every row to resolve to either "in the report" or "in an exclusions appendix," with nothing allowed to just not show up. That's the part that actually makes it diffable — a three-state internal tracker can still fail silently if nobody looks at it, but a mandatory exclusions appendix means an unaccounted-for fact is a visibly missing artifact, not a log line someone has to go dig for.

Worth being honest about where this leaves us: what we have today is closer to "spot-check known facts we went looking for" (the holdout check from this thread) than "exhaustive ledger over everything extracted." Getting to your version means the ingest side has to retain a fact inventory in the first place, not just classify into fixed buckets — which is the same rework already on our plate for generalizing this past two cases. So it's less "great idea, filed away" and more "this is now the acceptance criteria for that rework."

Ours died even earlier than both of your candidates: at ingest, before any research or strategist agent got a chance to consume or summarize it. The extraction schema pulled numbers and named entities from the source text, but a competitive-positioning fact that wasn't tied to a specific number or name never made it into structured form at all — so there was nothing downstream to truncate or fail to quote. It wasn't lost in translation between stages, it was never translated in the first place.

What's uncomfortable is that fixing that (widening the extraction schema, then fixing a hardcoded category list one node downstream) didn't close the gap entirely. We have a second fact — a specific benchmarking practice — that still doesn't survive to the final report even after both fixes, and we haven't pinned down exactly which layer eats it this time. That's actually forced our hand toward your "structured handoff" direction rather than "cite it or lose it" alone: patching node-by-node every time we find a dead fact doesn't scale, and each fix so far has only caught the specific failure mode we went looking for, not the general one. We're now building toward per-node contracts (a defined set of facts/source-spans each node is required to pass forward, checked automatically) plus stable fact IDs so a missing one fails loud instead of just quietly not showing up in the output. Basically your two fixes aren't an either/or for us — the source-span requirement is the "cite it or lose it" part, and it only works because it's enforced through a structured contract instead of a prose summary.

Both land, but differently for what I'm building. The provenance-block-as-schema idea is basically where this whole thread has been pushing me — treating "no source span, no valid output" as a hard constraint at generation time rather than something to catch downstream. Framing it as "the artifact is invalid without it" is a sharper way to say it than what I'd been calling a coverage check.

The session-start state file is a different shape of problem for what I have, though — this pipeline runs one report per input, not a long-lived agent with sessions to pick back up. But there's an analog: I've been accumulating known failure patterns as I find them (this thread talked me into turning that into a proper regression fixture), and that's functionally the same thing — a small file the system gets checked against every run so past failures aren't silently repeated. Different trigger (per-run vs. per-session), same underlying idea: don't rely on memory or habit, make the constraint something the system re-reads.

Interesting reframe — if I'm reading it right, this sidesteps the ingest problem entirely rather than fixing it, because there's no separate lossy summarization step deciding what's "worth" passing to a specialist. The coordinator sees the raw material and decides both what specialists are needed and what each one should look at.

Tradeoff I'd want to think through: right now I can test a fixed node in isolation with a fixture and know exactly what it's supposed to do with a given input. If specialist allocation itself becomes a model decision made per-case, debugging "why did this pipeline drop a fact" gets harder, because the failure could now also be "the coordinator decided a competitor-analysis specialist wasn't needed here," which is a different and less visible failure mode than the one in this post.

Does your version have the coordinator working off the full raw minutes, or a pre-processed version? That distinction seems like it determines whether this actually fixes the class of bug I found, or just moves it up a layer.

Honestly, right now I don't have a good way to tell the difference — that's exactly the gap you're pointing at, and it's the same one a few other comments in this thread converged on independently. The answer I'm landing on isn't "get better at judging when a category is real," it's to stop doing category-based inclusion at that node entirely — default to keeping everything from the minutes and define a small denylist of what's safe to drop, instead of trying to positively enumerate what matters. If I'm not classifying to decide what survives, the "is this genuinely a new pattern" question mostly goes away, because nothing has to earn its way into a bucket to avoid being dropped.

The three-state version — retrieved / cited / omitted — is more granular than what I've been tracking, which has basically been binary (did the fact make it into the report or not). That distinction would've told me something extra here: whether the competitor fact was pulled out at minutes_ingest and then dropped later, versus never retrieved at all. Turns out it was the latter (dropped at the very first step), but I only know that because I happened to trace it manually — a ledger with that granularity would've told me immediately, and it would also flag the "retrieved but not cited" case, which right now I have zero visibility into.

Both, actually — widened what minutes_ingest extracts (from numbers/named-entities to causal statements and third-party structure generally), and added a new category at the competitor-analysis node specifically for "relative position without naming a competitor." Widening ingest alone wouldn't have helped, since the downstream node still had nowhere to put the fact once it arrived. Same experience on the per-criterion logging, for what it's worth — the total genuinely looked fine until I broke it down.

The stable-ID piece is the detail I was missing from "assert facts survive" — without an ID, "did this fact survive" is fuzzy and hard to check automatically, since paraphrasing makes exact-text matching brittle. Tying a fact to an ID at extraction and failing the next node on a missing required ID turns this from "periodically eyeball whether it's still there" into an actual pipeline gate. And getting the regression fixture as a byproduct of that, instead of having to build it as a separate project, is a real bonus — makes it much more likely I actually do this instead of just agreeing it's a good idea and not getting to it.

That last line is a good way to put what actually happened — finding the missing fact was an afternoon of manually walking the intermediate state output, and it would've been minutes with must-survive checks at each node instead. The aggregate score told me something was off; it took a lot of manual tracing to find where. This is roughly the fourth version of "add intermediate checks" this thread has converged on, so it's clearly the first thing to build, not a nice-to-have I get to later.

That distinction is really the whole crux of it — the model technically "saw" the fact in the sense that it was right there in the input, but seeing something and doing something durable with it turned out to be two different capabilities, and I'd been implicitly assuming the first implied the second. Wouldn't have put it that cleanly myself.

Yes, and that's a good, concrete way to apply what this whole thread converged on — instead of asking "what's the 5th category," the benchmarks-7-competitors fact becomes the first real test of the retained-but-unclassified approach. If it survives in an unclassified lane instead of getting silently dropped again, that's a much better signal than "I added a category and it worked on the one case I built the category from." Actually going to use that specific fact as the first entry in the holdout set, since I already know the current pipeline fails it.

Prioritizing the end-to-end fixture over the per-node assertions makes sense as the first piece — it's the one that would've caught the original bug outright, no manual intermediate-state walking required. And the held-out set is the part I was actually missing: everything I've tested so far, including the "3 clean runs post-fix," was against cases where I already knew the failure mode existed. A fix that only works because it memorized the specific gap isn't really a fix. Adding a couple of genuinely unseen cases before I call anything resolved.

(Also — good username to be giving me pipeline architecture advice under.)

This is the third comment in this thread converging on the same fix, which is a pretty strong signal I should stop resisting it. Framing it as a "safe to drop" denylist instead of "keep everything with no filter at all" is the piece that makes it concrete — a short list of boilerplate/restatement/procedural noise is something I can actually write and audit, whereas "classify what matters" turned out to be an open-ended list I kept discovering the hard way, one production gap at a time. Inverting that node from allowlist to denylist is the actual change, not just "add more categories."

Fair, and it's a real tension with where this thread has been pushing me — a lot of good architectural suggestions here (retention over classification, per-node contracts, promotion rules) that I want to implement, but none of them matter much if nobody outside my own testing ever uses the output. Probably makes more sense to get a couple of real users on the current version before I rebuild the ingest node, not after.

Not with real users, no — everything so far has been tested against exam rubrics and scoring criteria, not against someone actually using it to do their job. That's the gap I'm aware of and haven't closed yet. The early-access program is partly meant to fix that: get it in front of a few people who'd actually use this kind of report, see where it breaks in practice rather than on paper.

This reframes it correctly — 3→4 wasn't "found two more patterns," it's evidence that anything I bucket at ingest will leak, no matter how many buckets I add. "Retention beats classification" at that specific node is the fix I hadn't considered: don't ask ingest to judge relevance at all, just keep every fact with its source line and let the downstream node that actually knows the question do the filtering. That also makes the uncategorized-bucket idea from earlier in this thread unnecessary at that stage — nothing to bucket if nothing gets dropped there in the first place.

One thing I'm not sure about: does full retention at ingest just push the same lossy-filter problem one node downstream, onto whichever node does the question-aware filtering? Or is that fine specifically because a node that knows the question is a fundamentally better place to filter than one that doesn't? Leaning toward the latter, but want to think it through before I rip out the ingest schema.

And agreed on per-element being the default view — burying it as a drill-down is basically saying "trust the summary unless you go looking," which is the exact behavior that let this slip past me in the first place.

That's the same direction I'm already moving in — minimum gates per criterion, not just an overall number. What made this one persuasive is that the summarized score genuinely read as "seems fine," and it took slicing by question instead of by the total to find the one gate failing almost every run. Curious how Agiloop decides which criteria get a hard gate versus which stay folded into the summary score — is that a fixed list per assessment type, or tuned per engagement?

Not memory/context — nothing was truncated or fell out of a window. It's squarely the second one: how the agent decides what's worth recording in the first place. The minutes-summarization step had an extraction schema tuned for numbers and named entities, and this fact (no in-state competitor, stated without naming one) didn't match either. It wasn't dropped after being noticed — it was never flagged as recording-worthy to begin with. Same story one step downstream, too: the competitor-analysis node's own classification only had two buckets (industry averages / named-competitor data), and a fact about relative position without naming a competitor fit neither. So less a memory failure, more a taxonomy failure, twice in a row.

This is a great comment, going through each point:

The extract-everything-into-a-pool-then-classify-with-an-uncategorized-bucket approach is better than what I've been doing. My current method is additive — widen the schema each time a gap surfaces — which means anything that doesn't fit yet gets silently dropped, and I only find out by manually walking the intermediate state after something already looks wrong. An uncategorized queue makes the gap visible immediately instead of requiring me to go looking for it. That's a real fix for the "there's probably a 5th pattern" problem, not just a patch. Reworking extraction around this instead of hand-adding a 4th and 5th category.
Fair catch, and sharper than the doubt I already had. I flagged the English-vs-Japanese trip rate as "not sure if it's language-specific or noise," but hadn't isolated that I wrote the English case myself — that's a real confound, separate from sample size. The grader doesn't see my prompt or intent, but I honestly can't say right now whether it's blind to case authorship or pre/post-fix status the way it should be. Need to check that before I put any weight on the 3-of-5 number.
Agreed on n=3. "Full marks all 3 times" reads more conclusive than it is — should be reporting the range next to the mean by default, and probably shouldn't have called it "fixed" off three runs.

And appreciate the note on the matrices — that's the whole point of publishing them, glad it's landing as something to actually check rather than just a claim to take on faith.

Tracing the drop, honestly — not fixing it, finding it. The fact wasn't just misclassified somewhere visible, it vanished before it ever reached the node whose job was to handle it. Had to walk the intermediate state output node by node until I found the first point where it was already gone, which turned out to be the very first summarization step. If I'd only looked at the node "responsible" for competitor analysis, I'd have never found it, because by the time execution got there the information simply didn't exist anymore.

Tracing the drop, honestly — not fixing it, finding it. The fact wasn't just misclassified somewhere visible, it vanished before it ever reached the node whose job was to handle it. Had to walk the intermediate state output node by node until I found the first point where it was already gone, which turned out to be the very first summarization step. If I'd only looked at the node "responsible" for competitor analysis, I'd have never found it, because by the time execution got there the information simply didn't exist anymore.

This is a side project around a full-time day job, so realistically it's evenings and weekends — not a fixed number, more like whatever's left after work on a given week. Some weeks that's a few focused hours on one bug (like the one in this post), other weeks it's mostly writing up what I found rather than building. Not sustainable at startup-hours pace, but enough to keep the validation loop moving.

Yeah, "fail loudly" is the key phrase — right now a disappearing fact fails silently, it just quietly lowers an average three questions later. Invariants at every node instead of one end check is the fix. And glad it's not just me — the hardcoded downstream list is the kind of bug that makes you feel dumb for an afternoon and then relieved when you realize it's a common shape of mistake, not a personal one.

Per-node contracts is a sharper version of what I've been doing by hand — tracing intermediate state after I already suspect something's off, instead of asserting it automatically at each step. That's exactly why the minutes_ingest drop sat unnoticed as long as it did: nothing flagged it, I only found it because I went looking.

And splitting coverage / fidelity / final usefulness into separate scores instead of one blended number is a real correction. I've been treating "did the required fact survive" and "was it represented correctly" as basically the same signal, and they're clearly not — the mystery-shopper fact failed on coverage, but a fact that survives and gets subtly distorted downstream is a different failure mode entirely, and right now I'd have no way to tell those apart. Building the fixture set with that split in mind from the start, not bolting it on after.

The Genie example is a really clean parallel — 12% of real traffic silently failing because the category wasn't even in the eval breakdown is the same shape of bug, just found from the production side instead of the eval side. Good reminder that this isn't a "my pipeline" problem, it's a "aggregate metrics" problem.

And yes, hit that exact hardcoded-formatter trap — fixed the upstream extraction, reran, output didn't change, spent longer than I'd like to admit assuming I'd made an error before tracing it down to a downstream node silently dropping the new category. Invisible until you manually walk the intermediate state, agreed.

On the coverage test — that's exactly where I'm heading next: a canonical case per fact type, asserted explicitly, so a new gap shows up as a red test instead of a quietly-lower average three weeks later. Don't have it built yet, but this thread is basically talking me into prioritizing it over the next validation round instead of after it.

Honestly, "since launch" undersells where I'm at — it's still invite-only, no public launch yet. So the biggest challenge so far isn't a launch-day fire, it's the gap this thread already poked at: getting from "passes my own rubric" to "actually useful for someone who isn't me." Everything's been validated against exam scoring criteria, not against a real person using it for their own work, and I don't yet know how those two diverge.

Thanks for reading it — glad the numbers made it easier to follow rather than just more noise to wade through.

You're right, and this is actually the direction I'm already moving toward — tracking pass/fail per required element instead of a summed score, so a 90 with one hard zero doesn't get averaged into "looks fine." The CAC analogy is a good one; that's exactly the kind of blending that let this sit unnoticed for longer than it should have. Going to make the gate explicit rather than something I have to remember to check manually each time.

Not with real users, no — everything so far has been tested against exam rubrics and scoring criteria, not against someone actually using it to do their job. That's the gap I'm aware of and haven't closed yet. The early-access program is partly meant to fix that: get it in front of a few people who'd actually use this kind of report, see where it breaks in practice rather than on paper.

Good question — honestly the part I'm least confident about. The 3 post-fix runs were on the same case I used to find the bug in the first place, so this doesn't rule out just overfitting to that one pattern. What would actually convince me is running the same fix against a completely different case (different industry, different competitive structure) and checking the miss rate holds up there too — haven't done that yet, it's next on the list, along with building a proper held-out test set instead of eyeballing individual runs. Will post an update once I have real numbers instead of "seems better."

Thanks, glad it was useful! Honestly still very early on the user side — no public launch yet, just reaching out directly to a handful of people who actually write this kind of report for a living (mostly independent consultants) and asking them to try it. Wanted the failure modes ironed out first before pushing on distribution. This post is partly that too — seeing if the story holds up with people outside my own bubble before I put more effort into finding users.