Three installs made my production metrics lie
Three anonymous installs and 137 events were enough to pollute my production dashboard.
The embarrassing part was not the size. It was that my analytics contract could not prove whether an iOS build was actually public when an event happened.
I had let launch-time heuristics decide too much on the client, and the dashboard later treated missing certainty as production. That was a fail-open analytics path.
The durable fix was not another chart:
- unknown stays unknown
- release truth lives on the server
- traffic gets classified at event time
- production metrics only read rows the server can classify authoritatively
The cleanup cohort ended up tiny: 3 installs and 137 events. But the lesson felt bigger than the number. If your system cannot prove a build was public when an event happened, your dashboard is mixing user behavior with your own testing behavior.
As a solo founder, that was the uncomfortable part for me. Dashboards feel objective long before the contract behind them deserves that trust.
What is your equivalent of unknown must stay unknown in analytics or release ops, and where does your system still fail open?
Longer write-up: https://medium.com/@msss.jerry.li/three-installs-made-my-production-metrics-lie-ceee391522d8
"Unknown stays unknown" is the whole thing, and the reason it's hard isn't technical. it's that an unknown looks like a defect in a dashboard and a number doesn't, so there's constant pressure to replace it with a plausible guess.
I ship measurement tools and hit the same wall from the other side. A browser can hand you a capture that's well formed but silently undersampled, and every figure computed from it is then confidently wrong. What worked wasn't better estimation, it was a tripwire (two quantities that should track each other, compared before anything renders). When it trips the tool prints why it can't answer instead of printing a number. It caught a real browser regression that way, and it cost me a feature that had appeared to work for months.
Once "unknown" is a real category, it has to carry what would resolve it. A bare null trains people to skip past it within about a week. An unknown that says which signal was missing and what would make it knowable stays useful, because it's an instruction rather than an absence. That's also the honest test of whether the classification is real (if you can't say what would resolve an unknown, you haven't classified it, you've just relabelled the guess).
Exactly. We initially treated unknown as a gap to smooth over, which made the dashboard look cleaner and less true. The resolving signal now has to travel with the state: for release classification, that means the authoritative server-side release record and the event-time lookup result. If either is absent, the event keeps the reason it is unknown instead of borrowing a guess. I like the tripwire framing too, especially for regressions that produce internally consistent but invalid data.
Internally consistent but invalid is the exact class, and it's the one that defeats most validation because the checks you'd naturally write are consistency checks. The data agrees with itself; it just doesn't correspond to reality. The only checks that catch it compare against something external: a second measurement, a physical constant or an invariant independent of what the code believes.
raunakbuilds' point about storing the classifier version is the same discipline from the other end, and worth doing even if it feels like overkill. I version the methodology alongside published measurements for exactly this reason. A measurement and the rules that produced it are one object. Separate them, and six months later you can't tell whether a result was real or just produced by the old logic.
The fail-open point is the useful part: missing release certainty should never be promoted into production truth. I would also persist the classifier version with each event, so a future rules change cannot silently reinterpret historical traffic and rewrite the dashboard.
That's a strong addition. The classifier version and the event-time decision should be immutable audit context, not something a later rules change silently rewrites. Otherwise an improved rule can make historical dashboards look cleaner while erasing the conditions under which those events were actually observed. It belongs in the contract explicitly.