Our AI pipeline attributed products to companies that never made them. So we made evidence mandatory.

I run PromptZone, a 22K-member AI community. Last year I built a content

pipeline that drafted articles with AI assistance. It worked, until it didn't.

For a while the pipeline generated article intros from a reference example.

The model copied the example's opening sentence verbatim into articles it had

nothing to do with — attributing unrelated products to a company that never

made them. Published. Indexed. Live for a while before anyone caught it.

The obvious lesson is "add a check." That's the wrong lesson. The right one is

that generated output needs evidence attached before it's presented as true.

A check catches the error you thought of. Evidence catches the ones you didn't.

So I rebuilt the prompt directory around one rule: a prompt doesn't get

published as usable unless it has been run, and you can see the run.

Every published prompt is executed in a locked sandbox against its target

model — no tools, no network, no credentials, synthetic inputs only, an exact

model identity with no provider fallback. The complete output of that run is

stored and shown on the prompt's page. Not a summary of it. The output.

Curated prompts additionally pass review on usefulness, accuracy and safety:

three models from three different vendors, one per dimension, and all three

have to pass. That review is done by AI agents, not human editors. I say that

plainly because implying an editorial team that doesn't exist is exactly the

failure mode I just described.

Two implementation details that turned out to matter more than expected:

1. The evidence is bound to a SHA-256 hash of the prompt text. Edit one

character and the badge is invalidated until the pipeline runs again. A

badge that survives edits means nothing.

2. Review is unanimous, not majority. With no human anywhere in the loop,

"two of three liked it" is too weak to put a badge on.

The honest limit: machine review catches a prompt that doesn't run, produces

nothing, or produces something obviously wrong. It doesn't have taste and it

doesn't know your customers. What it gives you is a floor — the prompt works,

and here's what it did.