Do any indie hackers use A/B testing for their products?

I've done a lot of A/B testing on a previous job but never on my own side projects.

My projects have virtually zero users and assuming a lot of the people here are in a similar situation it is hard to achieve statistical significance.

That said, if you think it's fun, have at it!

A/B testing can be simpler than you think though. Depending on what you want to test, I've A/B tested designs using a presentation and spreadsheet survey.

I think in most cases at an early stage qualitative data is all you need, but don't think you need a thousand users and analytics to make qualitative informed decisions.

Yeap. Optimizely is the best by far

I have A/B tested ideas before, but not on my recent indie product. It comes down to the volume of users you have for the feature you are looking to test and how much of a delta you expect to see in the performance between your A and B versions. The larger the volume the better. But at the same time, if the difference in performance between your A and B versions is not much, even a relatively large sample may be insufficient. I would suggest A/B testing only high-impact ideas if your volume of users is small.

Do remember not to violate the basics of A/B testing, such as by stopping the test when you hit statistical significance, regardless of sample size.

Testing a product with a small sample size/volume will likely not produce any meaningful results - at least not in a way that would justify the amount of work required to build, analyze, and continuously test. 70%+ of A/B tests will likely fail, and those that win will be small.

I used to live in "AB testing world" for a company that had 4M monthly visitors. It certainly had its highs and lows, and it's very exciting to see a win when there is a revenue impact associated with it. It can be addicting (and distracting for some personalities).

My suggestion for side projects is to keep your focus on your site's analytics, engagement, and behavior without AB testing will be the most beneficial.

Ask for written feedback or provide a popup survey with a rating scale for your limited users. It may not only produce more meaningful insight, but it is also a lot easier to build and maintain.

I haven't used A/B testing for my side projects, but I employ it daily in my freelance work with clients, primarily through Amplitude. I'm very data-driven when it comes to usage and growth, and I've probably conducted over 300 A/B tests since I began. Although this isn't my main role, as I'm primarily a developer 😅.
I've experimented with various aspects such as funnels, onboarding, typography, UX, and more.

To address your specific question: you're correct that testing with a small user base often provides inconclusive results, as random effects can significantly influence the outcome. Even when this wasn't a factor, I seldom observed a x10 effect with A/B tests; it's usually a 2 or 3% improvement. When starting a project, it's crucial to aim for a x10 improvement. For instance, investing your time in writing blog posts, tweeting, or launching ad campaigns might be a more effective way to attract and convert users for your product.

In my opinion, A/B testing becomes more relevant when your business attains a certain level of revenue and user count. In such instances, a 2 or 3% improvement can equate to thousands of dollars.

Yes, A/B testing is FUN and we are dogfooding our own A/B testing tool and yes it is built for marketers.

For indiehackers, I do believe that no brand/sideproject/product is too small for testing and everyone SHOULD be doing it.

It might take a bit longer to gather enough data to reach the required sample size and statistical significance, but it can provide valuable insights that help you understand your customer's preference and test out your key messaging.

There's really no downside - just valuable information waiting to be discovered.