I set up a clean CTR experiment. 20 days and 0 clicks later, I realized I designed it wrong from the start.

Aug 23 I rewrote a review page's title/description as a single-variable CTR test (kept everything else on the page untouched, per a suggestion from someone here on IH about isolating variables properly). Said I'd check back mid-September. Position held steady the whole time, roughly 4-6 range, so the rewrite didn't hurt ranking. But clicks stayed at 0 for the full 20 days.

When I actually ran the numbers instead of just eyeballing "still 0, still 0, still 0" week over week: cumulative impressions on that one page never broke 20 over the entire 20-day window. At a typical position-5 CTR (5-7%), expected clicks from 20 impressions is under 1.5. Zero actual clicks from that is just Poisson noise, not a signal - a process with an expected value around 1 lands on zero roughly a third of the time.

I'd built an experiment before I had enough traffic to ever get a real answer out of it. The rewrite might be working, might not be - I genuinely can't tell yet, and won't be able to until impressions on that page get into the hundreds instead of tens.

Two things I'd do differently, for anyone setting up something similar on a young/low-traffic domain:

1. Calculate the minimum impression volume needed for your expected effect size to be statistically distinguishable from noise before starting the clock, not after.
2. Pick the page with the most existing impressions for a test like this, not just whatever page happens to rank decently - "already gets some traffic" and "ranks well" aren't the same thing on a domain this size.

Anyone else run a page-level experiment that turned out to be underpowered from the start? How do you usually gut-check the traffic volume before committing to a test?