I found my own AI humanizer was quietly changing what the source text actually said
Ran a real test on a feature I built (an AI tool that rewrites stiff text to sound more natural) and found something worth sharing.
Source text: a study result, explicitly called "preliminary," 18% improvement, needs more data to confirm. Team of four engineers, no timeline given.
Humanized output: "This proves the routing system permanently fixed response time issues." Team of six engineers, in two weeks.
Read that second version alone and it sounds completely fine. Confident, clean, no grammar issues. But the hedge quietly became a certainty, and two specific details were just invented a team size and a timeline that weren't in the source at all.
Nothing about that trips a grammar checker. It happens exactly when you're making text sound more trustworthy, not less.
The fix I ended up building: a second, optional step that checks the rewritten output against the original source document, claim by claim. Flags anything unsupported, contradicted, or that shifted from a maybe into a definitely. It only checks against the document you give it (not the open web), and it's built to say "unclear" rather than guess when something's genuinely ambiguous — a false accusation felt worse than a missed one.
Full writeup with the complete example: letsflw.com/blog/humanize-plus-verify-against-source
Tool itself, if useful: letsflw.com/tools/humanize
Curious if anyone else has run into this same silent-drift problem in AI-assisted content, writing or otherwise feels like it gets easier to miss as outputs get more fluent.
Four engineers becoming six, with a two week timeline invented out of nothing, is the part that would have slipped past me too. Fluency hides the drift.
The move that keeps me honest on work like this is a weekend slice: pick one narrow claim type, say numbers only, and ship the check for just that in two sittings. Small enough to finish, real enough to tell you whether the flagging is trustworthy before you widen it to hedges and dates.
More on how I cut those slices: https://durablefoundations.gumroad.com/l/pyramid-reality-check
Which claim type drifts most often in your logs?
Kael Voss / DurableFoundations
Honest answer: I don't have a clean breakdown by claim type yet haven't logged flags at that granularity in production, so anything I said here would be closer to impression than data. What I can say from testing so far: hedges turning into certainties ("preliminary" → "proven") and invented specifics that were never in the source at all (a team size, a timeline) seem to slip through review more easily than outright number changes a wrong number is often something a person half-remembers and double-checks, but an invented detail that "sounds plausible" doesn't trigger the same instinct to verify.
The narrow-slice approach is the right instinct generally I did something similar building this, actually: started with just numbers (percentages, counts, dates) since that's the most mechanically checkable claim type, got that solid, then expanded to hedges/qualifiers once the numeric case was reliable. Ran a real test on that exact narrow slice recently and it held up well, including on subtle cases like a dropped "approximately" or "at least" being correctly caught, not just outright number swaps.
Good prompt to actually start logging by claim type going forward instead of relying on impression that's a real gap.
This is exactly why text transformation tools need semantic diffing, not just before/after comparison. The dangerous changes are not grammar fixes — they are meaning shifts that look like improvements. An AI humanizer that turns maybe into definitely is not humanizing, it is hallucinating confidence. The fix is to flag any factual claim that appears in the output but not in the input, and treat those as potential fabrications.
Exactly this, and it's basically why we ended up building it. The "flag any claim in the output not clearly in the input" approach is precisely what we shipped it checks the humanized text claim by claim against the original source, flags anything unsupported, contradicted, or where a hedge quietly became a certainty, and treats uncertain matches as "unclear" rather than guessing (a false accusation felt worse than a missed one).
One nuance we ran into: it's not quite a generic diff, since "meaning changed" and "text changed" aren't the same signal a lot of legitimate rephrasing changes the surface text without changing the claim. What matters is comparing the claims, not the wording. Wrote up the actual mechanics plus a real example (a fabricated team size and timeline that slipped through, caught before publishing): letsflw.com/blog/humanize-plus-verify-against-source
Curious if you'd approach the claim-extraction step differently that's the part I'd expect to be hardest to get right at scale.