On one topic, my sourcing step found 2 usable figures. That's the whole problem.
When I counted last week, Fiverr had over 6,500 gigs offering to rewrite and fact-check AI-generated content. Fifteen to two hundred dollars a piece.
People are already paying to repair what these tools produce.
Read the negative reviews of any AI writing tool and you'll see what they're paying for. It's rarely the prose. It's the statistics. The complaint is almost always the same: I had to verify every number, and the ones I checked were wrong or invented.
Here's the part I don't see discussed.
I built a step that searches and fetches sources before the article is written, so the writer can only use figures it can trace back to a real page. Then I ran it on a cannabis marketing topic.
It came back with two usable figures. Two.
Not because the search failed. Because the English SERP for that topic is agency blogs citing agency blogs. There was almost nothing to cite.
That reframes the whole thing. We talk about models hallucinating statistics like it's a flaw in the model. On a lot of commercial topics there is simply nothing citable, and a system required to produce a number will produce one. That isn't hallucination. That's what happens when saying nothing counts as failure.
So I made saying nothing an acceptable output. When the sourcing comes back thin, the article writes qualitatively instead of inventing a figure to fill the gap.
It still slips. Prompt rules are probabilistic, not guarantees. An article I generated this morning had six bare figures in it. So I score how often the rule is followed rather than assume it is, and I read the output before it goes anywhere near a client.
Reads less impressive than a chart going up. It's also the only version I'd put a client's name on.
Two usable figures may be the correct dataset. I wouldn’t automatically fall back to purely qualitative copy, though. I’d expose evidence tiers: primary source, independent secondary source, vendor claim, nothing found.
Then the article can say, “Two public sources report X; I found no independent estimate,” instead of either inventing precision or hiding the missingness. In structured datasets, absence is useful data—especially when twenty pages all inherit the same unsupported number.
"I score how often the rule is followed rather than assume it is" is probably the part most people building on LLMs skip.
It’s basically instrument calibration: the spec says the rule should hold, but the only way to know your actual error rate is to measure your own output continuously. Models drift, prompts change, and edge cases accumulate.
The other habit that transfers well is treating "no figure available" as a legitimate output. Being explicit about uncertainty is part of the product. A precise-looking number with unknown provenance is often worse than having no number at all.
Interesting point. Finding only 2 usable figures really shows how difficult sourcing can be. Sometimes the challenge isn't finding options, but finding the right ones that actually meet the requirements. Hope you find a good solution!
I’ve run into the same issue. Stopping AI from making up data is only part of the problem. You also have to verify whether the data it finds online is actually trustworthy. At the end of the day, it’s really an information reliability problem.
The two usable figures are probably more revealing than the sourcing system itself. It exposes how much of the problem comes from the underlying information available, not just the model producing the article.