My AI agent could "see" the fact — it just never wrote it down

I've been building AnchorStrategy, a multi-agent pipeline that takes meeting notes and turns them into a strategy report: it reads the minutes, has several AI agents split up research on competitors, customers, and cross-industry analogies, then a strategist agent pulls it into a SWOT and a set of strategic options.

I'd been validating it against Japanese business-school case studies (past exams from a Japanese small-business consultant certification), which worked, but left me unable to tell whether a weakness I was seeing was specific to Japanese-language input or a real flaw in the pipeline itself. So I built an English case to re-run the same validation and find out.

Building a clean case

Copyright was the first snag. Harvard Business School cases are cheap to buy for personal use (under $10), but publishing AI-generated output based on one needs separate licensing. I ended up using data from NIST's Baldrige National Quality Award — a US quality-award program. NIST's own 2-page profile of a winning company is clean to use but way too thin for a case study. The company's own ~50-page award application has plenty of detail, but it's the company's copyrighted work.

So I pulled only the facts out of that application — numbers, org structure, the business issues — and wrote an entirely new case narrative from scratch, using none of the original wording, headings, or product names. Facts aren't copyrightable; only the specific expression is, which is the same principle newsrooms and textbook writers rely on. The result: a fictional metal-nameplate and label manufacturer, "Company A," Dallas, Texas — family-owned since the 1940s, won a quality award in the late '90s.

The setup

Generation and grading run on different vendors (Gemini for generation, Claude for grading) so the grader isn't just validating its own house style. The grader never sees my prompt or intent — only the question, the official scoring rationale, a checklist of required elements, and the full generated report. It's judging one thing only: does the report touch the facts it needs to, not whether the strategy itself is any good.

The first result looked fine

Five runs, same case, same settings: average 78.0/100, range 73.5–86.5. That's roughly the same run-to-run variance I'd seen on the Japanese cases — nothing that looked alarming.

Until I broke it down by question. Question 1 (a 3C analysis plus a call to identify a structural weakness, worth 40 points) averaged around 30 — not bad on the surface. But that question has four separate scoring elements (E1–E4), and one of them — "references a competitor-related fact" (Q1-E2, worth 10) — scored zero in 4 of 5 runs. Everything else scored near-full marks, so the total buried a failure that was happening almost every time.

I nearly missed it myself. If I'd only tracked the total, this would have filed under "decent."

What was disappearing

The case narrative states the company's competitive strengths plainly:

Company A is a founding member of its industry's national trade association... To the company's knowledge, no other chemical-etching business operates in Texas; the nearest direct competitor is in a neighboring state. Unlike most of its competitors, Company A carries essentially no debt, maintains its own on-site metal inventory... It formally benchmarks seven of its leading competitors every year through a mystery-shopper program.

(No in-state competitor / debt-free, in-house inventory and tooling / benchmarks 7 competitors annually.)

But in the first run's output, the section that should have covered this read:

[HYPOTHESIS VALIDATION] The President & CEO's suspicion that reliance on spreadsheets is a major barrier to efficiency is supported by the operational bottleneck created by manual data entry between production, sales, and accounting...

No trace of the competitive picture — just a validation of the CEO's offhand comment about spreadsheets. In other runs the report went further and stated outright, "No individual competitor data is available in the input," when the data was very much there.

Tracing it back

I walked the pipeline's intermediate state, node by node, to find where this dropped out.

At the very first node — the one that summarizes the minutes (minutes_ingest) — this competitive information was already missing from the extracted key facts. It never even reached the dedicated competitor-analysis node (market_competitor); it was gone before that.

Digging further: the competitor-analysis node's prompt sorted incoming information into two buckets — "industry averages" and "named-competitor data." But "a fact about our own relative competitive strength, stated without naming a specific competitor" (no in-state competitor, debt-free, in-house capabilities) fit neither bucket. It fell through the gap in the classification scheme and was dropped before extraction even happened.

The fix

I widened what the minutes-summarization node extracts — from "numbers and named entities" to include causal statements and third-party business-structure context generally. I added a new category to the competitor-analysis node: "relative competitive position, without naming a specific competitor" — constrained to only fire when the input explicitly states it, so it doesn't start inventing unsupported claims.

Midway through, I found a side effect I hadn't planned for: a downstream node that formats the final report had the category list hardcoded, so adding a new category upstream didn't actually surface it in the output. Fixed that too.

Re-ran the same case 3 times after the fix. Competitor analysis scored full marks all 3 times.

Company A is the only chemical-etching business in Texas; the nearest direct competitor is in a neighboring state.

That line held, every time. Average total jumped to 89.3.

What's still broken

Not a clean sweep. One more fact — "benchmarks 7 competitors annually via a mystery-shopper program" — stayed missing in all 3 post-fix runs. It didn't affect the score, since that question only required any one of three facts, but it doesn't fit the three categories I just expanded (numbers/named-entities, causal statements, third-party structure), so there's likely a fourth pattern I haven't classified yet. One data point isn't enough to fix it properly, so I logged it and left it for next time.

Patching the classification scheme every time a gap surfaces is inherently reactive — I went from 3 categories to 4 this round, and I'd bet a 5th pattern turns up eventually.

Separately: the guardrail that blocks fabricated content (citation fidelity) tripped in 3 of 5 runs on this sample — a higher rate than I've seen on the Japanese cases. Not sure yet if that's an English-specific weak spot or just noise from a small sample.

The actual takeaway

The bug wasn't the interesting part. What stuck with me was that I almost didn't catch it, because I was reading the average. Sum several scoring elements together and one of them can fail systematically, every single run, and still get buried under everything else scoring well. The agent looks fine on the surface metric while consistently missing one specific thing — which is exactly the pattern AnchorStrategy is supposed to catch in a human's business analysis. Finding it in my own grading pipeline first was not lost on me.

Try it: I'm running an early-access program. Registration takes just an email address — apply here, and I'll follow up with access.

Request access

Full score matrices, the complete generated reports (before/after the fix), and the grading rationale for every checklist item are public here, if you want to check my work:

Validation archive