I built a startup-idea scanner. It just told me none of my 3,400 ideas are easy wins.

I really like that you tested your own assumptions instead of treating the score as unquestionable. One thing I'm curious about: what's the biggest factor that consistently separated the higher-scoring ideas from the rest? Was it distribution, willingness to pay, or something else?

3,400 ideas and none flagged as easy wins is actually useful signal in itself. What criteria is it using to judge 'easy win'?

You asked for the argument that scoring winnability is impossible in principle. I don't have that one — I think it's scorable. But there's a weaker link earlier in the chain: the 3,400 ideas you scored were surfaced by your own scanner, so the corpus and the scorer share an origin.

That's what makes "the new model is stingy... maybe because reality is stingy too" a coin flip with a third side. If the generator leans on what's already documented, it lands in well-covered space a lot of the time, and a near-zero ceiling is then partly a fact about where the ideas came from rather than about the market. You can't separate those right now: everything in the pile came out of the same machine.

Where I'm standing, so this doesn't read as advice from above: I run an experiment where an AI does the operator work end to end and I only touch the irreversible levers. I let it run hands-off for a stretch and it produced a steady supply of plausible, unwinnable directions. Sales are $0. What eventually moved it wasn't better scoring — it was an angle a person brought that the model wasn't going to generate.

The control I'd want is scoring ideas that didn't come from the scanner — whatever your first users brought you in conversation rather than ran through it — and seeing where those band. But something else comes first: you mentioned the same topic re-run swung the ceiling between 1.5 and 6.0. That's wider than the gap such a control would be trying to show. Freeze the model, get the per-topic noise floor, and the comparison becomes readable; run it before, and you get a number that supports whichever story you already believed.

Which is the call I'd actually put in front of you: not how to tune the weights, but whether the 75%-below-2 distribution is evidence about reality yet. Until you know the noise floor, I don't think it is.

Fascinating project, Eli. The fact that your tool told you the hard truth about your own ideas is the ultimate validation of its utility. Most tools just flatter the founder.

On the debate about whether scoring winnability is impossible: I think it is possible to estimate, but the variables (especially 'build cost' and 'distribution') fluctuate wildly based on the founder's hidden advantages. For example, a 6.1 idea requiring 1,200 build hours might be a 9 for a solo dev who uses enterprise-grade boilerplates to cut 80% of the build time, but a 2 for someone starting from absolute scratch.

I run into this as an engineer. I build production-ready Next.js templates (HadiKits) precisely because founders constantly underestimate the 'build cost' variable in their winnability score. You can have a winning idea, but if the execution drains your resources before launch, the math was wrong.

Curious how you factor 'build hours' into your model—is it a flat estimate, or does it adjust based on the founder's technical stack and existing assets?

3,400 ideas and none are easy wins? That's actually a good thing. It means the hard ones have moats. The easy wins are all taken by now.

nice!

"That makes sense — picking ideas that extend what you're already using is probably smarter than starting from scratch. I'm learning that the hard way with Rallynex — trying to build something that fits into existing workflows rather than forcing a whole new one. Which tools/platforms are you augmenting? Curious how you decided which ones were worth building on top of."

The 6.1 ceiling might be the most honest output a tool like this can give — but I'd flag what it can't see: the version of the idea you discover three weeks in. The thing I'm building now would have scored miserably on paper (crowded space, giant incumbents), and its actual wedge only showed up after watching real users do something I didn't design for. Scanners rate your starting position; most of the winnable-ness gets created mid-game by whoever stays in it. Fun test would be scoring the ORIGINAL ideas behind famous pivots and counting how many 3s became 9s.

And if you think scoring winnability is impossible in principle, tell me why. That argument is the one I most want to have.

Building any sort of model (machine learning, or even LLM/prompt-driven) is really hard unless you have a good training/test set: input data with the associated outputs so you can build a model and test whether the model, or your tool, does a good job.

You're never going to have a perfect experimental data set, but I think you can try to prove whether your tool works or not--and then it doesn't matter whether people agree in principle or not.

So pick a few ideas that you've seen work in real life, for some definition of "work" (successful write-ups here, funding announcements, acquisitions, etc.). You'd want to avoid anything sufficiently well-known that the LLM would know about the actual company. Any single idea's success is of course subject to all sorts of randomness and factors that are hard to measure. But if ideas that have worked tend to produce high winnability scores, then you'd start to convince me it's doable--and I think that would be a stronger argument than any argument you could make in principle.

Nice

"3,400 ideas is wild. What made you finally pick one to build? I'm building Rallynex and struggling with that focus myself."

The insight about "distribution gap" is what really separates validation from validation theater. Most idea tools amplify the fantasy because their business model depends on keeping you building. Your tool does the opposite - it says "stop until your audience is ready."

That's harder to commercialize (nobody wants to hear it) but it's actually the most valuable signal. The 5/6 willingness-to-pay failures probably teach the model more than the successes - what matters is whether founders with distribution problems can now recognize the pattern before burning runway on a problem they can't solve with code.

Did you place successful acquisitions in it to test it? :)

The strongest product insight here is that "winnability" probably has to include one very boring variable: can this founder reach 20 qualified buyers this week without borrowing an audience?

Demand, competition and build cost matter, but for tiny founders distribution access is often the real constraint. I would score each idea against an actual founder-owned path: existing list, communities they already participate in, customers they can email, niche expertise, or paid channels they can afford to test. If the path is vague, the idea should stay low even if the market looks big.

Treat winnability as a ranking model, not a truth score. The next useful test is prospective: freeze the model, take a holdout set of ideas it has never seen, and run the same low-cost validation sprint on the top, middle, and bottom bands.

Then compare precision at the top: what share of the top 10% produces a qualified conversation, a concrete pre-commitment, or a paid pilot? Also test a few low-ranked ideas to estimate false negatives. If high-ranked ideas consistently outperform low-ranked ones, the model is useful even if no score reaches 7. If everything fails equally, it is only producing persuasive numbers.

The product's moat may end up being calibration data from these validation outcomes, not the scoring formula itself.

75% of the corpus scoring below 2 is the part that lands hardest — most idea tools tune their output to feel useful, so keeping the model stingy is what makes it a decision tool instead of a validation machine. The 5/6 willingness-to-pay failures are fascinating: did that check separate "won't pay" from "won't pay yet — no audience"? Curious whether the new scoring weights distribution enough to catch that gap.

The 5/6 willingness-to-pay failures are the interesting part to me.

You changed the scoring model after seeing that result — what would you need to see now to know the new “winnability” score is actually better at choosing which ideas deserve customer conversations, rather than simply being more conservative?