2
5 Comments

A few questions about A/B testing

Hey all,

I had a few questions about setting up A/B testing. I can handle the dev side of things, just need help with the strategy.

  1. How do you handle a return customer?
    For example, say I'm testing a price change and a user gets one price based on a link they clicked, but then navigate to the website directly and then they don't see their original price/change.
    I could set up a cookie for them so that the site is the same way every time.
    Is there a standard way to handle this?

  2. How do you encode your unique URLs so that you can tell what you did in that version, but the user can't tell by looking at the URL? I'm assuming you can just use a random letter/number generator and make a note somewhere what that code means.

  1. 2
    1. You would need to stick the user to a specific version and as long as the test is running they should get the same variation of the page.
    2. I think you can do this in a ton of different ways. event 1 or 2 would work out because nobody knows what is what

    I would recommend using a tool that is made for a/b testing that takes care of all of this!
    Explainer: I'm working on Splitbee.io an analytics and A/B testing tool :)

    1. 1

      Once the test is over then all users get pushed to the "winner" or do some of the users stay grandfathered in the old price? I'm assuming everyone gets the update.

  2. 2

    Great questions - following to see if I can also learn the answers :D

  3. 2

    AB testing pricing is hard work.

    Do you have the scale to make sure the tests are significant?

    I think you would need 100s of paying customers during the test.

    1. 1

      Definitely not, but that's good to know that I need to wait a bit before I start testing that. Thanks

Recommended Posts