Four ways mining Reddit for startup ideas went wrong for me, and one disclosure
Everyone mines public complaint threads for ideas. Almost nobody writes down where the method fails, so here are the four that cost me the most, and the disclosure at the end that I think matters more than the technique.
One: I was hunting for novelty. Wrong target. A novel complaint is usually novel because it belongs to one person with an unusual setup. What is worth finding is the boring complaint that keeps reappearing in different words from people who do not know each other.
Two: I did not write findings down as sentences with links. This sounds like bookkeeping and it is not. If a problem cannot be written as one sentence with a link, it is a memory of a thread rather than a finding — and by the fourth thread you already know what you are hoping to see, and you see it. That is not a discipline failure, it is what reading is.
Three: I read complaint volume as demand. Threads sample who was annoyed enough to type, not who would pay. Satisfied users never post, so the loudest problem in a community is often the problem of its most vocal minority.
Four: I assumed a niche forum represented its market. A community selects its members, so whatever you conclude generalises to people already there. That is a real segment. It is not the market.
The disclosure, because a guide about mining discussions should say what its author's own tool actually reads: we do not source Reddit. Our published run pulled 56 documented problems from 41 sources and read 24 pages in full, and the hosts were review and alternatives sites, revenue comparables, job boards and Hacker News threads. Reddit sits in a slot we have not built.
Naming that felt worse to write than it reads. It also makes the article true, and the technique does not depend on anyone's connector list.
Of those problems, 32 carry a link you can open and the rest are labelled as our estimate — which is the same discipline as step three, applied to ourselves: https://whittleos.com/guides/startup-ideas-from-reddit
The sentence-with-a-link rule is useful, but I reckon the next trap is treating linked findings as independent findings.
Five pages can repeat the same original complaint, quote one another, or reflect the same commercial incentive. That creates the appearance of persistence without five separate people encountering the problem.
I’d want each finding to carry a little provenance: source type, date, user or buyer role, current workaround, and whether it is genuinely independent of the other evidence. A link proves that somebody said it. The surrounding record helps show whether the problem repeats across different people and contexts.
Do you currently deduplicate repeated claims across those 41 sources?
Straight answer: at URL level yes, at claim level no — and the distinction is exactly the one you're pointing at.
What runs today: the same page found by several queries merges into one record, and a URL that shows up under two sub-markets is kept once. Across runs there's a shared store of problems keyed on a hash of the normalised theme plus the problem text, so a re-surfacing bumps an occurrence counter instead of adding a row. That's an exact-after-normalisation string match, which means two paraphrases of one original complaint do not merge, and five pages restating one source are five records. So what my output calls persistence is closer to "this phrasing recurred" than "five people independently hit this" — your criticism holds, and the counter is the part most likely to be read as stronger than it is.
Against your five fields: source type I have, through the host and the provenance tier. The current workaround is captured inside a run but not persisted, so it never contributes to the repeat count. Date, user role, and independence I don't carry at all. Independence is the one I don't know how to do honestly — quoting is sometimes detectable when a page links or attributes, but a shared commercial incentive between two review sites almost never is from the page itself. My instinct is that the achievable version is a weaker claim rather than a better detector: count distinct hosts and distinct dates, and say that's what the number means.
This is really the story of four different measurement failures. You're describing how the absence of a clear filtering system led you astray - you measured novelty instead of persistence, you stored findings in memory instead of writing, you measured volume instead of behavior change, and you measured forum-size instead of market-size.
The disclosure here is the upshot: write findings down with links so you can apply signal discipline yourself. It's the only way to stop measuring the wrong thing.
Four measurement failures is a tighter frame than the one I wrote, and it names the common cause better: each one measures something adjacent to what I cared about, because the adjacent thing was the countable one.
One caveat on the write-it-down half, because I got that wrong for a long time in a way that felt rigorous. A file of findings is still a survivor list — it records what you kept and silently loses what you looked at and discarded, which is the half that tells you your filter is miscalibrated. I only started catching my own bad filters after I wrote down the rejects with the reason beside each one; reading the kept pile always felt fine. Second caveat: a link buys less than it looks like. It proves you read a page, not that someone re-deriving the number from it would land where you did. Worth carrying those as two separate labels rather than letting the link imply both.