How to validate growth experiments?
I'm trying to think through how to validate/invalidate various customer acquisition strategies. Of course, I could lean on classical statistics, which would say that I need a very large sample size to conclude anything. As a startup, I don't have access to hundreds or thousands of users for every experiment. If I did try everything on all my users, they would probably get tired of it and opt-out :)
Has anyone run lots of experiments and used smaller sample sizes (ex. n=10 or so) to get a signal from the market? Any blog posts that you would recommend about running smaller marketing experiments as a startup?
Trending on Indie Hackers
I think in the beginning you just need to rely on your gut and talk to customers individually to get a feel for what is working and what isn't. Also, copy what seems to be working for others who are further along.
Always good advice!
Every stage requires a different approach to "get a signal from the market". For tens of customers - it's 1-on-1 interviews. Move to experiments and statistics only when interviews are too costly or don't give any new information. Good luck!
Definitely! Have been leaning on my customers, but the challenge is that I don't have that many of them (yet) so I can't ask them questions every few days :)
You don't want to run to your customers every time you're in doubt. :) They can't tell you what to do anyway.
Maybe theres some inspiration for you in PG's essay titled: "Do things that don't scale":
http://paulgraham.com/ds.html
Great suggestion! Had forgotten about that gem. Just reread it and it'll help me focus my marketing going forward
Hi Harris,
Been there before and totally get your struggle.
I’ve mentored hundreds of startups on experimentation in different accelerator programs and my advice boils down to:
Lastly, I’d challenge you on running any experiments on 10 users or so. Sounds like it could be better to focus on acquiring more customers and test things on the acquisition side?
Best
Johannes
Hey @joradig,
First, so sorry for my delayed reply! I appreciate all the depth you went into here.
I've got an Airtable sheet that I'm using to keep track of all of the experiments. I usually test them on about 10 people, just to get some signal into whether it's a good idea. They're all on the customer acquisition side, so I'm glad that's what you'd suggest I do!
Definitely can't waste my time working on buttons colors :) There are so many things that I would love to change about my UI, but they're not the most important things to do to move the business forward.
It's really hard to get significant metrics when the sample size n <=10.
However, if you can get those numbers up a little, I've found MixPanel to be the perfect tool to measure such experiments.
Nice...I'm on Amplitude, but similar technology. What sort of an n do you look for?