If you filter data on the way in, your metrics are counting survivors
I have a filter that strips podcast roundups out of one of my sources. Last week it nearly made me delete the source.
devdigest, the daily tech digest I build and run, pulls from more than 100 sources. New ones go into quarantine: collected and scored normally, but never allowed into anyone's real digest until I've seen a week of actual data and made a call.
Review day came for a batch of 28. One of them, a well-known security writer, read like this in my own tooling:
Stories published in the review window: 0
Stories in the last 30 days: 0
Most recent post: a month ago
Dead feed. Obvious reject. I nearly deleted the entry on the spot.
Then, out of habit rather than suspicion, I opened the actual RSS feed first.
He had published that morning. And weekly before that, with no gaps, going back months.
Here is what happened. When I added the source I also added a title filter, because roughly seven of every eight of his posts are "Weekly Update NNN" podcast roundups rather than articles, and a roundup is not something a digest can use. That filter runs at collection time. Anything it excludes never gets written to disk at all.
So my review tool was never reading what the source published. It was reading what survived my own filter, and displaying it in a column labelled "published in the last 30 days".
The number wasn't wrong exactly. It answered a different question than its label implied, and I had been reading it for a week without noticing.
The general version, which is why this is worth writing down: if you filter data on the way in, every metric downstream is measuring survivors. That is fine while you remember it. It stops being fine the moment someone uses that metric to make a delete decision, because at that point "filtered out" and "never existed" look exactly the same.
The fix was not in the filter. The filter is correct and I kept it. The fix went into the thing that reads the data: any source carrying a filter now says so, in the tool, next to the numbers, with an instruction to check the live feed before calling anything dead.
The source went back into quarantine with a proper window, to be judged on the right question. Not "does he publish", because he clearly does. The real question is whether what remains after the roundups are stripped out earns a permanent slot.
Four days later it caught me again, and that is the part I did not expect. I was about to apply the same kind of fix to a second source, on a number that read "1 story affected". Same trap: that 1 came from records saved before I added that source's filter, so it described a version of the source that no longer exists. The warning I had just shipped is the only reason I checked. Against what the source can actually publish today the real number was 0, and the change would have done nothing at all.
If you run a pipeline that drops records before storage: go and find out what your dashboards have actually been counting.
If you want to see what it actually produces: https://devdigest.io/?ref=indiehackers
The fact that the metric was technically correct but still misleading is a nasty one. Especially when the label makes the distinction invisible.
That is exactly the part that got me. A wrong number you argue with. A number that is correct under a definition you have forgotten is one you act on.
It caught me again today in a different shape. A test in my suite passed while making a live API call and cancelling a real subscription. Green run, because the assertion was satisfied by the side effect it should never have caused. Same structure as the metric: technically true, and answering a different question than its label implied.
What I have taken from both is that the fix is almost never in the mechanism. The filter was correct and I kept it. The test was correct about what it asserted. What was missing both times was something making the gap between the label and the question visible at the moment someone reads it.
That’s a pretty strong pattern — the number can be correct while the decision it drives is wrong. The live API example makes that especially clear. Curious whether you’re thinking about this as a broader product principle now, rather than just a metric/testing issue?
Yes, and it happened a third time since I wrote this.
Today I reviewed a fix over a five day window. The target said "roughly 10 to 25 stories demoted per day". Two of the five days came in at 4 and 5, which reads as the fix underperforming. The eligible pool on those days was 27 and 40 stories instead of the usual 200. As a rate it was flat across all five days.
The count was correct. The conclusion I would have drawn from it was wrong, for the same reason as the filter: the label carried an assumption the number did not.
So the principle I work to is narrower than a product philosophy. When a number is going to drive a decision, write the denominator and the assumption beside it, in the tool, where the person reading it will see them. Not in documentation, which nobody opens at the moment they act.
That’s a very useful distinction — especially the idea of putting the denominator and assumption where the decision happens. I’d be interested in continuing this conversation. What’s the best email to reach you at?