I spent a day debugging a coin flip that was working fine

I had a homepage A/B test running for a couple of weeks. Two arms, 50/50 split. The counts came back 178 and 57.

So I did the obvious thing and went after the randomizer. Read the middleware, read the cookie logic, eventually wrote a script that called the assignment function ten thousand times just to watch it bucket. It came back 4,981 / 5,019. The coin was fair. I had just spent a day proving it.

The problem was on the other side. I had one function that both attached the arm as a sticky dimension to future events and fired a one-time "this person entered the experiment" event. The first job has to repeat on every page load. The second must not. I was calling that function from the root layout, so every page load anywhere on the site fired both. My exposure count was really a pageview count.

What actually cost me the day was that the two arms rendered through different layout trees, so they inflated at different rates. If they had inflated evenly I'd have seen something like 178 and 174, thought "denominators look high", and found it in ten minutes. Uneven inflation made it look like a broken split instead. The bug was hiding behind a symptom that pointed somewhere else.

The lesson I keep coming back to: a number that contradicts your system is not automatically evidence about your system. It's evidence that one of the two is wrong, and the measurement is the half nobody writes tests for.

Conversions were counted correctly the whole time. Only the denominators were junk, which means every rate I'd looked at for two weeks was wrong while every input to it was right.

What's your check for making sure the thing doing the counting is actually right?