I built a tool because A/B testing cold email is broken — here's what happened when we tested it
I've been running cold email campaigns since I was 16 for a B2B HR software company. Five years of the same frustration: you test 3 email versions, one clearly gets more replies, but Instantly keeps splitting sends equally across all three. By the time you notice and manually switch, you've already burned hundreds of leads on your worst copy.
So I built Apex Overlay. It connects to Instantly.ai and automatically shifts more sends toward whichever email version is actually getting replies — from day one. No manual checking, no waiting days for A/B test results, no hard kill switch.
First beta campaign: StaffCircle, 1,053 HR leads, 3 email versions.
Best email had 96% of sends by day 12
6% more replies vs Instantly's equal-split approach
A curiosity-style subject line beat two direct pitches across every job title — a finding we never would have caught with manual testing
Second campaign (Notetaker):
14% more replies on day one
Best email identified within 24 hours
Got a qualified reply within 3 hours of the first optimised sends — that reply converted to a deal
The unexpected finding: 31% of clicks in the StaffCircle campaign were from security scanners, not humans. If you're picking winners based on click data, you're optimising on noise. Our system filters those out.
What I learned building this:
The algorithm is invisible and hard to explain simply. People understand "A/B testing" but glazing over when you describe continuous probability shifting. The positioning took longer than the code.
Getting beta testers is harder than building the product. I've messaged over 200 people on LinkedIn and Reddit. The conversion from "that sounds interesting" to actually signing up and connecting an API key is brutally low.
The product works but the go-to-market is the real challenge. The people who need this most (agency founders managing 5-10 client campaigns) are also the hardest to reach because everyone is pitching them something.
Currently:
In beta — first campaigns showing ~13% more replies vs equal-split testing
Free for 3 months, no card needed
Building in public — research and findings at apex-scale.com/research
Happy to answer questions about the Thompson Sampling implementation, the bot filtering approach, or honestly anything about what it's like trying to sell a technical product to people who don't care about the technology.
Results from live campaigns: apex-scale.com/results
Try it: app.apexscale.live
