The thing my review process keeps catching is my own measurements, not my code
The last two things I posted here were both measurement defects of mine: a filter that made a live source look dead, and a directional error that manufactured a trend out of one incomplete day. I have now run enough of these reviews to say that was not a coincidence. It is the pattern.
Some background. I build devdigest, a daily tech digest that shows you why it picked each story. Every change that changes behaviour goes into a ledger with a written prediction attached, and a review date is set the day it ships, usually five days out. On that date I have to come back and read the real numbers against that prediction. Not a vibe check - the entry records the pre-ship baseline so I am comparing against something I committed to before I knew the answer.
Twelve changes have been through it. None has been reverted.
You could read that as the review being toothless. I would have, six weeks ago. But eight of those twelve reviews did find something broken, and in every one of the eight it was the measurement rather than the change.
Three of them, concretely:
A target that was impossible when it was written. One fix's own entry contained an A/B bounding its effect at roughly 65 to 70 percent fill. The target sitting a few lines below demanded 91.7 to 97.6. The fix hit its bound and "failed". The rule I added afterwards: when an entry contains an experiment that bounds the effect, the target may not exceed the bound.
A baseline in the wrong unit. The recorded baseline counted distinct stories. The live metric counted slots in a digest. Same words, different denominators, and the review read wrong in the direction that flattered me - which is the direction you are least likely to go back and question.
A metric that moved when my subscribers did. This one is the worst, because both readings were defensible. The metric pooled per-slot shares across whoever received a digest that day, so it moves when the subscriber mix moves and nothing in the pipeline has. Two subscribers stopped receiving mid-window. The same five days read +1.45 pooled and +4.26 with them dropped. Only the per-subscriber version is composition-free, and it was the one that showed the fix had actually worked - on 6 of 6 subscribers.
I nearly recorded that last verdict backwards. "Failed target, not failed fix" was already drafted.
The generalisable bit, if there is one: we test code and we do not test measurements, and a broken measurement is far more dangerous than a broken function, because it does not throw. It returns a number, and the number looks exactly like an answer.
devdigest went live on Product Hunt yesterday if you want to look at the thing itself: https://www.producthunt.com/products/devdigest-io
And the product: https://devdigest.io/?ref=indiehackers
Partly, and the split is the honest bit.
What is actually tested now is the code that computes the metrics. The open-day guard has clock-pinned tests, and it is not bolted onto one metric - ten call sites route through it, so a metric cannot quietly opt out. Baselines are data rather than prose: each ledger entry records the pre-ship number before the change ships, and every figure shown is recomputed from the raw daily records rather than stored, so a number cannot drift away from the thing that produced it.
What is still only discipline is the definitions themselves. Nothing checks that a target contradicts a bound recorded four lines above it in the same entry. Nothing checks that a baseline and its live metric count the same unit. Those are the two that actually bit, and both are still a human rule I can forget.
The fix I keep circling is making the unit a declared field on both the baseline and the metric, so a mismatch is a config error at ship time instead of a discovery on review day. Not built yet.