2
3 Comments

I built a public dashboard that shows almost nothing, on purpose

I shipped a two-agent sales kit a couple weeks ago. Two agents, one shared attribution layer, sold as a self-deploy kit for $349. And the dashboard that's supposed to prove the thing works currently shows the honest empty state, because no deals have closed yet.

That's not a launch problem I'm hiding. It's the design.

Here's the thing about selling anything that claims to measure AI's contribution to revenue: everyone claims it, almost nobody can show it, and a buyer who's paid attention knows the difference between a vendor number and a real one. So before I worried about what the dashboard would eventually show, I spent real time on whether the thing generating those numbers could be trusted not to lie to itself.

The two agents, InboxCopilot for inbound email and EnrichmentMessenger for CSV enrichment, both write to the same database as capped, conservative attribution credit. Before shipping either one, I ran a cold verification pass: the skill file exactly as written, fed to the model as its only instructions, output checked against a strict schema, no retries, no hand-holding. Six test cases total. Five passed clean.

The sixth one was the interesting failure. A CSV row with basically nothing usable in it. Correct behavior: skip it, log nothing. What actually happened: the agent invented a plausible-sounding "yes, this became a real lead" result out of a row that had no basis for one. If I hadn't caught that before launch, the sourced pipeline number, the single cleanest metric this whole product is built around, would have been quietly wrong from day one. Not dramatically wrong. Just wrong in a way nobody would have noticed until they went looking, which is worse.

The root cause wasn't the model being dumb. It was my own instructions letting ambiguity resolve toward "sure, I'll call it something" instead of toward "I don't have enough to call it anything." I rewrote the decision logic as a single ordered check that rules out "not enough information" first, before anything else gets evaluated. Re-ran the same six cases. Six for six.

I bring this up because it's the actual answer to the question every attribution product should get asked and mostly doesn't: how do you know your own tool isn't lying to you? Mine failed a test once, in a way that would have mattered, and I have the commit where I caught it and the commit where I fixed it. That's a better answer than a clean track record I couldn't prove.

The dashboard being empty right now is just the same discipline pointed at the next question. I'm not going to publish a number I can't defend, and I'm not going to borrow someone else's case study to fill the gap while I wait for my own data. It fills as the real deals close. If you'd rather watch a real number arrive slow than trust one that showed up too fast, the empty dashboard is at stack.chadtdyar.com/proof and the kit itself is at stack.chadtdyar.com.

on August 19, 2026
  1. 1

    Caught the same class of bug dogfooding our analytics agent (amami.dev): it didn't fail loudly, it produced a confident, plausible-sounding number from a row with nothing usable. Ordering "not enough information" first is the right call - we shipped the same rule, and now every answer has to show the pages/dates behind it or it gets flagged as a guess. The empty dashboard is the honest version of that discipline.

  2. 1

    The attribution hallucination failure you caught is exactly the right failure to worry about. "Confidently wrong" is worse than "obviously wrong" because it does not trigger alarm bells until someone goes looking - and in sales attribution, by the time someone goes looking, you have already made decisions off it.

    The fix (ruling out "not enough information" before anything else gets evaluated) is a pattern worth noting anywhere you are asking a model to make a credibility claim. Models are trained to complete tasks, and that pressure beats "admit I cannot" in ambiguous cases unless you explicitly structure around it first.

    The empty dashboard as design choice rather than gap is the thing I would actually pitch to buyers. Any vendor can show you a number. Almost none of them can show you the commit where they caught their own tool lying and the commit where they fixed it. That is a more compelling proof-of-methodology story than a full pipeline chart.

  3. 1

    The failed sixth test is more compelling than a clean 6/6 would have been. You found a failure mode that could have produced believable but indefensible attribution, which is exactly the kind of failure worth exposing before customers depend on the number.