Week 3: our first pre-registered measurement came back against us

Short version: on Sep 3 I wrote here that a tracked brand looked like it had lost ~98% of its AI visibility in two weeks, that I thought it was a name collision, and that we had pre-registered a test and would publish Sep 7 whatever it said. It said: partly wrong.

The qualifier cleared the collision (0 of 42 answers about the markdown editor). Visibility did not come back: 2 of 37 clean answers cited a directory (5.4%), against the 18–31% our other "alternatives"-shaped prompts get. And a second collision appeared in Gemini within the week (Otter.ai, the transcription tool). Pre-registered verdict: gate does not hold; our "prompt shape decides directory citation" claim is now "correlates, with exceptions".

The vendor-vs-third-party follow-up I promised: of the 147 comparison-shaped pages cited, 78% were competitors' own "X alternatives" posts, 15% third-party, G2 under 3%. The answer slot is real; directories do not own it, vendors do.

What it changes in the product: (1) a "qualifier" is not a fix — we are building answer-text off-category detection instead of prompt tweaks; (2) every score now ships with its n and its interval, because 2/37 and 20/37 look the same on a chart without one; (3) we are adding a holdout set of prompts so we can tell an engine change from a brand change.

Being honest about it cost about a day and one claim I liked. Thread with the pre-registration and the full numbers: reddit.com/r/aeo/comments/1w3690m. Still €0 MRR, still building in public.