I spent a day red-teaming my own LLM scanner. Here's what it got wrong.

Instead of shipping a feature, I pointed my own CLI at my reference targets and watched where it failed.

1. Canary matching was case-sensitive. A flipped-case leak walked right past it. Switched the scan path to case-insensitive, kept the default value behavior intact. Committed.
2. The probe set was narrower than its own target. The same probes that fired on one surface missed leaks on a doc-summarizer surface. (I'd earlier called the detection "form-factor independent" — turns out that's true for the mechanism, but not for how the probes are worded.) I made that target a worst-case fixture and widened the trigger patterns until those probes landed. Committed.
3. The one I couldn't fix. If a leak is transformed — base64, split tokens, spacing, reversed, zero-width chars — a deterministic substring matcher simply doesn't see it. That's a structural limit, not a bug I can patch away. So I put an explicit warning in the scan output and the --canary help: encoded/split leaks are NOT detected.
   The tempting move is to quietly fix the easy ones and stay silent on the hard one. The honest version is to ship the warning. A green scan means "no Category-1 verbatim leak found" — not "your app is safe."
   Curious how others draw the deterministic-vs-fuzzy line in security tooling before false positives start eroding trust.
   If you want to poke at it, the code and the demo target are here: https://github.com/ghkfuddl1327-wq/rojaprove