We published the weights our product scores ideas with. Here is what publishing them cost, and the one number I wish I had two years ago.
The setup: 7 weighted categories, top three tied at 20 each, plus one category that is rated, shown, and weighted at zero on purpose.
The zero-weight one is founder understanding. It belongs on the page because it changes what YOU should do about a verdict, and it stays out of the arithmetic because an idea does not become a better business by being handed to someone who understands it.
The number worth having: the heaviest area carries 4 times the leverage of the lightest. Take an idea rated 7 everywhere except distribution at 3 — total 62. Put three rating points into distribution and you get 70. Put the same three into the lightest category and you get 64.
8 points against 2, for identical work. One of those changes the grade and the recommendation; the other changes nothing at all, and feels exactly as productive while you are doing it. That is the argument for publishing a rubric, and I did not believe it until I had the ratio.
What it cost: a reader can now tell us our priorities are wrong, and can check whether our own scores follow our stated rules. Both are the point, and both were uncomfortable to sign off on.
The part I insist on printing next to the number: the arithmetic is fixed, the ratings are not. They are model judgements sampled at temperature one, so the same idea run twice can move a couple of points. If you are choosing between ideas the letter grade is enough; if you are deciding what to fix, the per-area ratings are the right instrument and the total is the wrong one.
Every weight and the worked example: https://whittleos.com/guides/how-to-score-a-business-idea
Nice, this makes a lot of sense. What's been the most surprising part of it so far?