Users don't attack your content filter. They negotiate with it, one word at a time.

I run an image-to-video tool. You upload a photo, describe the motion in a sentence, and a model animates it. Motion prompts are the whole product.

That one detail broke every assumption I had about content moderation.

The thing I got wrong first

The obvious way to ship moderation is to call a classifier and trust its flagged boolean. I did that. It was wrong within a day.

Omni-moderation flags low-confidence "violence" on perfectly benign action prompts. "She bursts into a sprint down the pier, camera tracking." "He slams the door and walks out." Those aren't edge cases for me, they're the bread and butter of what people type into a motion box. A boolean gate would have blocked a chunk of my legitimate traffic on day one.

So the gate isn't the flagged field. It's a mapping from specific flagged categories to a decision, with the categories I actually care about treated differently from the ones that fire on ordinary verbs.

What the logs actually showed

Once the first organic wave hit, I expected to see attacks. Prompt injection, encoded text, that kind of thing.

I saw negotiation instead.

The same user, same session, submitting a sequence of prompts that were each a slightly softer version of the last. Delete one explicit noun, resubmit. Still blocked? Delete the next one. Rephrase the verb. Keep going until the classifier lets it through.

Nobody was breaking the filter. They were binary-searching it. And a stateless per-prompt classifier loses that game every single time, because each individual attempt is, in isolation, a slightly-less-bad prompt than the one before it.

Three changes that actually helped

1. State. The filter has to remember.

Count a user's recent denials in a short rolling window. A couple of denials and their threshold tightens. A few more and everything gets denied for the rest of the window.

That last one is the important one. It doesn't block bad content, the per-prompt check already tries to do that. It blocks the probing loop itself. "Delete a word and resubmit" stops being a viable strategy when attempt number five gets denied regardless of what it says.

2. Different thresholds for different risk profiles.

Users who have never paid get screened at a stricter score than paying users, even on prompts the classifier didn't flag.

This sounds unfair until you look at the numbers. In my data, the people burning free credits on soft-NSFW attempts convert to paid at zero. Not low. Zero. Meanwhile they're the ones putting my provider and payment accounts at risk. Aiming the strictness at that specific cohort cost me nothing I wanted to keep.

3. Screen the prompt and the image together, but only as a second pass.

This is the part I'd have gotten backwards if I hadn't measured it.

I screen the text alone first. If it passes, I screen text plus source image in one request. Either pass blocking blocks.

Why not just do the combined check once and skip the text-only pass? Because a benign photo dilutes the text score. Measured, same day, same prompt: a borderline request scored 0.57 on text alone and 0.38 when paired with an ordinary portrait. The classifier averages across the modalities.

So if I had only run the pair check, uploading a completely innocent photo would have been a reliable way to soften a bad prompt. Running text-first and treating the pair pass as additive only means the second check can add a block but never lift one.

The fail-open exception that took me longest to see

The whole stack fails open. Missing API key, network error, upstream 5xx: the request proceeds and I log it. That's deliberate. The model providers run their own NSFW checks as a last line, failed generations auto-refund, and failing closed would brick the core feature every time a vendor hiccups.

But there's one failure mode where fail-open is a hole: a deterministic 4xx.

If the moderation request itself is rejected, say an oversized image payload, then retrying the identical bytes will fail identically, forever. Fail-open on that means "pad your image until the moderation API 400s" becomes a reliable, repeatable bypass. That one case has to fail closed, and it's the only one.

Network blip: fail open. Vendor down: fail open. Request permanently malformed: fail closed. The distinction is whether a retry could ever succeed.

Two smaller things

The blocklist is deliberately full of holes. There's a fast local string blocklist that runs before any network call. Free, instant, and it never fails open. The temptation is to stuff it. I don't, because single words that look unsafe often have completely legitimate uses in a visual tool: "intimate" (typography), "playroom" (kids' content), "boudoir" (a photography style), "passion" and "climax" (a color name and a song title). Multi-word phrases can be aggressive; single words mostly can't. A false positive on someone's real prompt costs more than a marginal catch.

Allowed prompts are stored as hashes. The audit table keeps a snapshot of the raw prompt only for denials, because that's what you need to review a false positive. Everything that passed is a hash plus the score. I still get threshold-tuning data; I just don't accumulate a database of everything my users have ever typed.

Where I landed

Moderation for a generative product isn't a filter you install. It's an adversarial loop with a user who is patient, iterating in real time, and getting feedback from you on every attempt.

The classifier is maybe a third of it. The rest is state, cohort-aware thresholds, and being deliberate about which failures are safe to wave through.

Happy to go deeper on any of it. The tool is stivio.ai if you want to see what the prompt box looks like in practice.