Why I stopped doing 50/50 A/B tests (and built a Multi-Armed Bandit engine instead)

Hey Indie Hackers,

I’m the founder of Zyro. Coming from an analytics background, I’ve always had a love/hate relationship with traditional A/B testing tools.

The "hate" part comes from the math.

If you run a standard A/B test (Frequentist model), you split traffic 50/50 and wait for "statistical significance" (usually P < 0.05). For a small startup, this might take 2–4 weeks.

The Hidden Cost: Regret

During those 2 weeks, you are knowingly sending 50% of your traffic to a losing variation. In data science terms, this is called "Regret"—the difference between the reward you could have gotten (by showing the best version) and what you actually got.

I realized that for bootstrapped founders, "Regret" is expensive. We don't have infinite traffic like Google or Amazon to waste on "data purity." We need revenue now.

The Solution: Thompson Sampling (The "Bandit" Approach)

I decided to ditch the static 50/50 model and build an optimization engine based on Multi-Armed Bandit algorithms (specifically Thompson Sampling).

For those not familiar with the math: instead of testing A and B equally, the algorithm updates the probability distribution in real-time.

• Start: 50/50 split.

• Day 3: Variation B converts slightly better. The engine automatically shifts traffic to 40/60.

• Day 7: Variation B is clearly winning. The engine shifts to 10/90.

This means we exploit the winning variation during the test, maximizing conversions immediately instead of waiting for the test to finish.

Adding "God Mode" Context

The other flaw with standard testing is the "Average User" myth. A user coming from a TikTok ad has a completely different intent profile than a user coming from a Google Search.

We combined the Bandit engine with our Traffic Source Detector. Now, instead of finding one "global winner," the engine runs separate instances for different sources.

• It might learn that Headline A works best for TikTok traffic.

• But Headline B works best for Google traffic.

• It then routes them accordingly in real-time.

The Tech Stack

We built the decision engine to run server-side to prevent the "flicker" effect. It uses a localized geolocation database (MaxMind) to keep decision latency near 0ms.

I’d love to hear how other founders are handling optimization. Are you sticking to standard A/B testing for statistical rigor, or are you moving toward dynamic/bandit models to move faster?