A/B testing, who, how, why, what?
The lean approach to startups focuses on testing your assumptions, validating, pivoting, etc. I wonder if you have any good examples on how you've set up, for example, an A/B test to check your ideas. What were you trying to validate, how did you make sure it was statistically relevant, no biases, what was your expected outcome before running the experiment, what happen afterwards?
Just an example, imagine you have a newsletter, and you would like to check whether people open it more if you send it on Tuesdays and not on Thursdays. The obvious approach would be to split your list into two groups, half gets it on Tuesday and half on Thursday and compare. The problem is, what happens if your opening rates change from one week to the next? May mean your content, your subject, etc. is more important than the day, and thus the A/B test would be meaningless.
Totally agree with @harrydry. IMO a/b testing is valuable when you already have a good product that people love and you want to test little optimisations on an already successful screen or funnel (the placement of a CTA or colour of a paywall screen aka low variable count). The example you gave about letters has too many variables so you're right a/b testing won't work here. Even if you get a result you won't know what learnings to take away from it.
In those cases you get way more bang for you buck by having open channels to get feedback from users and by dogfooding your own product.
yep - said better than how i said it
exactly. you've summarised the problem with a/ b testing at a small / mid level.
too many variables to get any statistical significance . I personally don't see value unless we're talking 60k / mo user size. follow your gut. and ask your users for their thoughts.
You could run the test over a few weeks?