How are you doing A/B testing?
Hi Indie Hackers!
I'm looking to go into the space of A/B testing, which I always found interesting (the ability to come up with an idea and prove that it actually increases value, makes both the creative and analytical part of me happy).
It would be very interesting to hear from those of you who are doing A/B testing today. What tools and techniques are you using and what is your process when performing experiments?
As a bonus, it would be fun to hear good anecdotes (a test that worked much better than expected, one you were sure would work but didn't or that time you miscalculated the results and almost killed your business).
As someone who's full-time job used to be running A/B tests (300+ tests a year for a travel company) I don't think most Indie Hackers should be doing A/B testing. It's unlikely that the majority here will see sufficient traffic, in a reasonable period of time, to reach statistically significant results on areas of the funnel which matter.
Now, that doesn't mean you shouldn't be doing Conversion Rate Optimisation, but often the more qualitative data will be useful – particularly when starting out.
Ah okay. I was expecting many of the more established indie hackers to be big enough for testing. At which stage should one start to get into it?
For qualitative data, is user testing with recorded videos the way to go or more like heatmaps from all visitors?
Btw, that sounds like a cool job! I can imagine it feels good to see the results you make represented as clear numbers.