Guessing what works in cold email is expensive. A/B testing replaces opinion with evidence: you send two versions, compare results, and keep the winner. Done right, it compounds, each test makes the next campaign better.
Here’s what to test, how to test it properly, and which metrics to actually trust.
What is A/B testing in cold email?
A/B testing (split testing) means sending two variants of an email, identical except for one element to comparable groups, then comparing how they perform. The goal is to isolate what actually drives replies so you can do more of it.
The golden rule: test one variable at a time
If you change the subject and the CTA at once, you won’t know which caused the difference. Change exactly one element per test. It’s slower, but it’s the only way to learn anything reliable.
What to test (in priority order)
- The opening line: your biggest lever; test different personalization angles.
- The subject line: affects opens (though opens are noisy; see below).
- The CTA: a question vs a meeting ask, soft vs direct.
- The value proposition: how you frame the benefit.
- Email length: shorter vs slightly longer.
- Follow-up timing and number of touches.
Which metrics to trust
Weight replies and positive replies over opens. Open tracking is unreliable now because providers pre-load images, so a “winning” subject by open rate may mean nothing. The metric that matters is whether more people replied, and replied positively.
How to run a clean test
- Pick one variable and create two versions.
- Split a comparable, large-enough segment randomly between them.
- Send under the same conditions (same time window, same list quality).
- Measure replies and positive replies, not just opens.
- Roll out the winner, then test the next variable.
Common A/B testing mistakes
- Testing multiple variables at once.
- Samples too small to be meaningful.
- Judging by opens instead of replies.
- Never acting on the result.
Frequently asked questions
What should I A/B test in cold email?
Start with the opening line, then the subject and CTA, one variable at a time.
How big does my test sample need to be?
Big enough that the difference isn’t random for small lists, a few replies’ difference isn’t reliable, so test on larger segments.
Should I test by open rate or reply rate?
Reply (and positive-reply) rate. Open tracking is unreliable due to image pre-loading.
How long should an A/B test run?
Until you have enough replies to be confident, accounting for follow-ups, not just the first email’s results.