85
59 Comments

Don't validate your ideas, invalidate them.

It's a simple fact that most startup ideas are not going to work out. Just like most tweets never get a retweet, most GitHub projects have no users, and most Hacker News posts never get an upvote. That's the simple truth.

If most ideas are no good the most efficient way to find good ideas is to discard the bad ones as quickly as possible. People talk about "validating" their business ideas by doing market testing, but "validating" has a bias towards a positive outcome. In reality the outcome is usually negative. So don't validate your startup ideas, invalidate them.

I've found it useful to take a scientific attitude. When you have a business hypothesis you need to run experiments to see if your hypothesis matches with reality. In the words of Richard Feynman, "if it disagrees with experiment, it's wrong." Just like Newton poking his eyeballs with knitting needles it helps to be a little detached. Not your retinas though. You'll need those.

Newton pokes his eye

This is the story of how I invalidated my greatest ever micro-SaaS idea.

A couple of weeks ago on my morning run, high as a kite on endorphins, I came up with a magnificent idea. I was going to change the world by helping noble open source developers get funded. I'd do this by helping people grow their GitHub sponsors with perks for their sponsors.

I spent the whole day doing some deeper research. There was so much going for this idea. Here is the list of "pros" I wrote down and breathlessly emailed to my entrepreneur friends:

  • viral loop (people link to the URL to use it)
  • proven concept (existing competitors)
  • makes money for people
  • creator economy trend
  • i personally know the customers

Foolproof. I started fantasizing about what I would say in my Indie Hackers Podcast interview when it hits $10k MRR. How one has to stay humble and wait for the big idea. How one has to work hard and stay focused. How one must listen to the customer you idiots, listen to the god damned customer!!!

The idea was perfect. Except for one thing. Nobody actually wants this software. I know this because I invalidated it.

I set a concrete goal. If I can't convince 100 people to sign up for the launch in 2 weeks then I will give up on the idea. John O'Nolan got 30,000 sign ups from one blog post about his idea before building anything. Surely if my idea is any good I can get 100 signups.

Invalidation #1 - friends

The first thing I did was tell some of my developer friends. Their feedback was interested but luke-warm and skeptical. None of them wanted this product. Nobody said "take my money!"

yeah I like your gh idea too
but unsure if its good or bad

Ok, that's interesting. Maybe it's just my particular developer friends who don't want this? Friends try hard not to hurt or offend you so "your idea is ok" should really be downgraded to "your idea sucks".

I should also note that one of my friends is an open source developer actively taking donations. They should be the target demographic, but they were not very interested.

Invalidation #2 - search

It's always good to check search traffic. Is anybody out there looking for your solution already? You can check Google search volume, Reddit, and Twitter.

For Google search volume I use Ahref's free Keyword Generator tool and also the SurferSEO plugin.

Ahrefs said there are 80 searches for "github sponsors" per month. Not great. SurferSEO said 880 searches per month. That's better, but I trust Ahrefs more. The keyword is also very broad. Searches for more targeted keywords like "how to get github sponsors" were very low.

A Twitter search for "github sponsors" shows there is a lot of chatter. Out of all of the data I collected this Twitter chatter is probably the strongest pro-validation signal.

A Reddit search likewise shows a bit of traffic for people talking about "github sponsors" but the volume was lower than Twitter.

So I built a landing page

I created a simple signup page where people could get notified of the launch by signing in with GitHub. That should get high quality signups from real GitHub users. It could not be easier. All they have to do is click the "sign in with GitHub" button.

GHPerks screenshot

I wrote up the idea in the clearest way I could, explaining the benefits. This also allowed me to proof-of-concept the tech stack and GitHub API integration and make sure I could actually deliver the features. You can see the site at GHPerks.com.

Then I started the next round of invalidation testing.

Invalidation #3 - landing page

I posted about the site and the idea in a bunch of places.

  1. I wrote a tweet asking if I should pivot from my previous micro-SaaS idea to this.

Invalidation tweet one

161 impressions. 4 people clicked through. No signups.

  1. I posted to my local Linux users group mailing list. It has thousands of open source people on it.

1 reply. No signups.

  1. I posted on Indie Hackers with the title "Looking for devs who want to grow their GitHub sponsors".

33 views. 3 upvotes. No signups.

  1. I posted a final tweet as I felt like the first one didn't really communicate it well.

Invalidation tweet two

214 impressions. 12 people engaged. No signups.

In the end the MVP landing page had hundreds of open source developers visit and nobody signed up to hear about the launch.

This is a big warning sign that nobody wants this product.

Invalidation #4 - competitors

At the start when I first had the idea I did some competitor analysis. Did anybody else have this idea already and was it working for them? People often see competitors as a bad thing, but usually it just means there is a healthy market already where you can offer a differentiated product.

I re-discovered the story of Caleb Porzio who had grown his own GitHub sponsors. I found his MVP of a similar idea https://sponsorsyrup.com and remembered that I actually signed up for this thing.

It made me wonder why he hadn't posted any progress updates. Why had it not launched yet? I had never received any emails about it. I couldn't help thinking, is it because Caleb didn't get the interest he hoped for?

I also discovered https://onlysponsors.dev which is a similar idea. Eduardo has 84 sponsors on GitHub, but are they sponsoring him for this or for his work on Vue.js? From what I can tell there are not many people posting on the site.

This research helped me think about differentiators. What would I do differently? These projects both appear to be closed source but I would stay open source. I would also position it differently from Only Sponsors, and offer different features.

In the end this is an invalidation. Both of these people have huge followings on Twitter and GitHub and they have existing sponsorships. Only one of them has shipped and there doesn't seem to be much activity. With my small audience it would be a lot of work on the marketing and distribution side.

Invalidation #5 - target audience

I contacted some open source developers I know who have substantial sponsorship on GitHub. I even contacted devs who explicitly said they have this problem. There was some interest but not huge, and some strong warning signs:

Dev 1:

Currently I don't have any ideas for "sponsor" only content. I tried this for a while with my videos and secret links, but I gave up on that

When I tried something like this with the videos eventually I felt like I was spending too much time on pleasing sponsors with extra stuff rather than my OSS, so that would be one reason not to go there. When I asked my sponsors about this, most of them said: we are sponsoring for your projects, not for the perks, so you don't have to spend extra time on those perks.

As for special content: I did a few "hidden" videos but eventually I also just wanted to share that with everyone. I guess I'm just bad at keeping things a secret or away from people.

Dev 2:

I have done basically nothing to promote it. probably should be doing more on that front but its mostly organic

no rewards no. can't think of anything useful

Dev 3:

I’ve thought this through a few times and talked to a few other maintainers a while ago. My first impression was: The target group (open source devs) love to build things on their own and they don’t have money to spend. That’s both very tough to deal with. Also, I know some teams failed with similar products (probably for that reason).

This is some great feedback. These people are the exact open source maintainer target audience with existing GitHub sponsors. They are telling me no, we don't want this, and other people have failed at it already. Strong invalidation.

Conclusion: invalidated

The strongest message is the signups. Remember at the start I said I was aiming for 100 signups in two weeks? Only four developers signed up to hear about the launch. Three of them are friends and one is my brother.

My hypothesis was "this idea is so good I will get 100 signups in two weeks". Now I have strong empirical evidence to falsify it.

So this isn't the right app for me to build. It could still be a valuable idea for someone. The right person with a high level of passion and commitment might be able to make it work. Maybe Caleb or Eduardo will succeed with it (and I hope they do).

For me though it is nice to have given the idea a good chance and see it through to invalidation. Running this experiment was fun. Now that I have invalidated it I can skip the pain of building all the features only to discover nobody wants it. I feel very good about that!

Do you want help invalidating a business idea? Tell me about it and I'll help you come up with a plan.

  1. 10

    Wow one of the best posts I've read on IH. Really great job Chris.

    1. 2

      Thank you so much. I am a big fan of your work. Your research on marketing channels really helped me.

      1. 1

        Wow thanks! Glad to hear that.

  2. 1

    Can't agree more. I usually approach your concept like following:
    Arrange all reasonable conditions (market-wise) for target audience to not pass your solution of their problem. If they do pass it still -- that's it, try something different or find someone else.

  3. 5

    Chris this a really helpful and well written article. I guess all entrepreneurs will have to come up with their own ways to invalidate their specific ideas.

    I remember my thesis advisors had asked me to rewrite my hypotheses with a negative bias!

    1. 1

      For sure every idea will have different specific ways to [in]validate. There are some common techniques that are broadly useful though. Stuff like:

      • Checking search traffic.
      • Speaking to people who might be a potential user.
      • Landing page pitch and collecting emails.
      • Posting about it and looking at interest metrics.
      • Introspection to see if you will actually enjoy working on it.

      That's funny to hear about your thesis advisors, thanks for sharing!

  4. 3

    I started fantasizing about what I would say in my Indie Hackers Podcast interview when it hits $10k MRR.

    Haha! I start with clearing my wish list. I'll step up my game.

    All good points. imho;

    It's impossible to validate or invalidate via friends. Instead a retweet is a better indicator. Means a network of people backing the idea, the project in some way. If not it would a be a check mark for invalidation.

    Target metrics are vague, I've released something recently and got 3 visits in total :) In the meantime; https://twitter.com/ImSamThompson/status/1449518979387576323 Idea is simply to exchange emails for supportive conversations ($30/month) I believe clout what we all need. Without clout invalidating ideas is quite easy.

    Search volume is another metric to check yes, but I wonder how much volume there were for to come up with onlyfans or clubhouse idea, who asked for them.

    It's interesting with such low search numbers there were already competitors. This kinda tells me there is demand for it but no one come up with the right solution/approach. I used to work with a company that their motto was to "empowering open source" with cyrpto currency. We provided thousands of logo designs, translations, bug fixes etc. via paying $. There is also GitCoin recently raised another $11M totaling $37M.

    Paying for open source was an endless debate in my opinion perks is not the way to go. But it's a huge market with great people who deserves to be compensated for their hard work.

    1. 3

      I wonder how much volume there were for to come up with onlyfans or clubhouse idea, who asked for them.

      Great point. There are obviously products which do well despite no prior demand. They create a new market and then capture the whole thing. The question is, as an indie do you want to risk your own time and money on a product like that? Most business ideas don't work and I would say that is doubly true for "blue ocean" ideas without existing demand. I would say as an indie it is less risky to start with something smaller with more certainty about demand. Later once you know you can build and sell things, then it makes sense to try for a blue ocean idea. Basically a "Clubhouse" type of startup should not be your first attempt (VCs will tell you the opposite of this of course because of misaligned incentives).

      It's impossible to validate or invalidate via friends.

      Yep. It's dangerous because they don't want to make you feel bad (Mom test). However I note @1hakr does this, as he mentioned in his AMA:

      I see a potential problem, see if I can solve it. See if it's already being solved, if not I pitch the idea to few people. If atleast 50% got excited. I pick this idea to build

      So he uses it as a basic filter. If even friends don't get excited it's probably not going to be easy to sell.

      I believe clout what we all need. Without clout invalidating ideas is quite easy.

      True it's hard to tell without a sizeable audience if something is going to have traction or not. That's why I think it's important to run multiple lines of inquiry, and test for traction in multiple ways. Make sure you are empirically getting eyeballs on the idea. Maybe there's no search traffic, but maybe 100 people on Twitter say "take my money!" etc. So try both.

      it's a huge market with great people who deserves to be compensated for their hard work.

      It's interesting just how many people have tried to solve this problem over the years and none of them has worked out in a big way. Probably GitHub sponsors is the most successful so far. I guess there is something about the open source funding space that is inherently tricky.

      Thanks for your comments Anil, very interesting!

  5. 2

    Great story, hats off!!!
    My 2 cents as an open source dev about the idea - I think the biggest problem could be - the perks you offered. Open source devs want to see their work getting used more and more. "Private videos", "Private docs" ideas go completely against that philosophy.

    Maybe perks like "being able to meet the repo owner once in a month to discuss (video call)", "being able to view and influence the future roadmap", "getting premium support for issues" would be more exciting.

    1. 1

      That's a good point. It might be that different perks would generate more interest. However I would be careful about doubling down on an idea like this and iterating on details if people are so fundamentally disinterested as I discovered.

    2. 1

      Or perhaps getting a callout during the install process or on the webpage.

  6. 1

    What's your view on building a MVP and test it with beta tester first ?

    1. 1

      Seems sensible. More than one beta tester is good too.

  7. 1

    This is a super powerful post, thanks so much, Chris. I LOVE my idea and am very committed to it, but you are 100% right... I need to build certainty that there's a need out there. Well, er, I know there's a need (for forgiveness) but is there a desire? I appreciate your very clear advice.

    1. 1

      Thank you, I'm glad it was useful. Best of luck with your idea.

  8. 1

    Thanks for sharing this. Will go ahead and apply right away for an idea I'm working on

    1. 1

      Best of luck, I hope it works out!

  9. 1

    Chris, this was a super useful article, thank you for sharing !

  10. 1

    Great post explaining how to (in)validate an idea. Thanks for sharing 👌

    1. 1

      Thanks, I'm glad you enjoyed it.

  11. 1

    Love it, "Product Market Unfit."

  12. 1

    Thanks for sharing your invalidation! It's good that you didn't start building.

    1. 1

      Yes I saved a lot of time and effort! Thanks for reading.

  13. 1

    I love how put the title. Unique way of thinking 👍

    1. 1

      Thank you. I've seen your stuff on IH before. I hope your projects are going well.

      1. 1

        Thanks, yep I’m planning on building some website templates again :D

  14. 1

    Thanks for sharing this path Chris.

    1. 1

      No problem, thanks for reading.

  15. 1

    This is such a helpful post! It's so easy to fall in love with our ideas that we overlook validating our ideas or rather invalidating them. I can so relate to thinking your idea is so brilliant that you daydream about your success 😂

    1. 1

      Heheh yes I think we've all been there!

  16. 1

    Great post! This article should be considered as a checklist for what to do when someone is so excited about the million-dollar idea :P

    1. 1

      Thank you. I was thinking about making a check list like that. I will post on IH if I do it. Thanks for reminding me.

  17. 1

    Useful write-up, probably everyone with an idea should read it. I chuckled at

    Foolproof. I started fantasizing about what I would say in my Indie Hackers Podcast interview when it hits $10k MRR. How one has to stay humble and wait for the big idea. How one has to work hard and stay focused. How one must listen to the customer you idiots, listen to the god damned customer!!!

    I was also fantasising how I will ensure to hire great people in my unicorn. So typical! :)

    1. 2

      Heheh yes we've all been there. 🙈

  18. 1

    Worth reading. Thanks for sharing.

    1. 2

      Thank you it was my pleasure.

  19. 1

    This really set me off in the right direction. Thabk you so much Chris!

    1. 1

      No worries, I am glad it helped you.

  20. 1

    Great work Chris! This is the exact hustle required.

  21. 1

    truly great posts like this one are what I miss on indiehackers nowadays

    1. 1

      Thanks Nikolai, glad you liked it!

  22. 1

    Great post! Thank you for sharing. It’s nice to see stories with ‘negative’ outcomes :)

    1. 2

      Thanks, yes, in science they have this problem where people don't publish their failed studies often enough, so there is a bias towards studies that show effects. I think the same is true in the indie hacker space. Success stories are motivating and great but fail stories are useful too.

      1. 1

        Yes, there is more to be learnt from failures than success stories. I think people should write more about their failures!

  23. 1

    Great post @chr15m and my god do I wish I had read this 10 months ago! meetrhea.com.au started out as an auto follow-up tool over Facebook messenger that I got funding for and pivoted to an open home follow-up and lead qualification tool. The website has been up for months and we have had a handful of signups. We pivoted to proptech companies that do lead generation but feel although I might just be pro-longing it's death. I know there is something here ( automating sales and customer service via conversational AI ) but could really use some help invalidating ideas along the way!

    1. 2

      I think you should get systematic. Decide on some metric and pick a hard number that would make you go "yes, I will keep going with this" or "no, this is dead in the water". Then start running experiments:

      1. How would a customer describe the problem your app solves. What would they type into Google? Look at search volume and see if anybody is doing this.

      2. Make a list of places where your potential customer hangs out online. In those places is anybody talking about the problem you solve? Engage with people and ask about the problem space without mentioning your solution. Do they have the problem?

      3. I would try A/B testing some changes to your copy. Try to get some people from the target audience to explain the value proposition back to you and use their words in your copy instead. Then post your landing page up in the places you wrote down in 2 to see if you get more signups.

      4. Run some ads targeting your audience. Spend a couple of hundred dollars getting some clicks through on keywords closely related to the solution, and see if you can improve signups.

      If you still can't get enough signups to build a business on it then you should can it as quickly as possible and move on. Or don't. Do what you want. :)

  24. 1

    After my initial feeling of "aren't you inviting failure?" I get it. I think it is a realist approach even if explaining it sounds almost cynical. At the end of the day you very thoroughly ran it up the flag pole without writing a line of code. You truly saved yourself a ton of time to gun for the next idea because of it so that's the actual win here.

    1. 1

      Oh I wrote a lot of code don't get me wrong. The GitHub auth code is legit! 😁 I just didn't build the whole thing which would have taken weeks or months. "Saved yourself a ton of time" really nails it.

      I do worry about the "inviting failure" thing because building a thing that works also requires a lot of persistence. I have had some small success on other projects where I just kept working on it for a long time, so persistence can work for sure.

      These things are all multipliers, so ideally you want to apply your persistence and staying power to a business idea that is also very good from the start, not one that is going to be a grind.

  25. 1

    Great post. So easy to get caught up in an idea and focus on validation and overlook assumptions.

    I like your SEO insight too, that even with keyword research you can convince yourself a search query has buyer intent when it may well not.

    But yeah the competitors existing and then disappearing is fairly strong signal there's nothing there.

    1. 1

      Yeah absolutely. I made that exact search volume mistake with a previous startup idea. There were hundreds of searches per month on a term, but then it turned out none of them wanted to buy they just wanted information.

Trending on Indie Hackers
How I grew a side project to 100k Unique Visitors in 7 days with 0 audience 49 comments Competing with Product Hunt: a month later 33 comments Why do you hate marketing? 29 comments My Top 20 Free Tools That I Use Everyday as an Indie Hacker 16 comments $15k revenues in <4 months as a solopreneur 14 comments Use Your Product 13 comments