Cold email A/B test calculator: significance and sample size
An email A/B test calculator tells you whether the difference between two variants' reply or open rates is bigger than chance. Enter delivered emails and replies for each variant: it runs a two-proportion z-test and returns the p-value, the lift and a confidence interval, then works out how many sends each variant needs to detect the lift you care about.
Use delivered emails (sent minus bounces) and unique counts: a person who replies twice is one reply.
Variant B wins at 95% confidence
- A reply rate
- 3.00%
- B reply rate
- 5.00%
- Relative lift (B vs A)
- +66.7%
- p-value (two-sided)
- 0.0225
The difference is 2.00 percentage points (z = 2.28). At 95% confidence the true difference is between 0.28 and 3.72 points.
Show the formula
Pooled rate p = (xA + xB) / (nA + nB) = 4.000%. z = (pB − pA) / √(p(1 − p)(1/nA + 1/nB)) = 2.282. The p-value is 2 × (1 − Φ(|z|)), where Φ is the standard normal distribution.
How many sends does each variant need?
To detect a change from 3.00% to 3.90% at 95% confidence and 80% power, send 6,455 delivered emails per variant (12,910 in total).
Next step
Variant B won on 2,000 delivered emails. Make sure the next test is measuring copy, not data.
A bounced email can never reply, so dead addresses shrink each variant unevenly and add noise to every comparison. Verifying the list before you split it means both variants go to real inboxes, and the sample sizes above are the ones you actually get.
- Drop your list as a CSV
- See its rows, duplicates and bad formats, free
- 100 free email checks when you sign in with Google, no card
Ready to verify now? Verify the list
Sample result
Variant B wins at 95% confidence: its 5% rate beat 3%, with a p-value of 0.022.
- Winner at 95%
- Variant B
- p-value
- 0.022
- Lift
- +66.7%
What this means for you
The 66.7% lift is unlikely to be chance, so B is the version to keep sending.
- P-value, lift and confidence interval
- Replies, positive replies, opens or clicks
- 90%, 95% or 99% confidence levels
- Sends per variant at 80% or 90% power
- Z-test
- Sample size
- Significance
By SimpleVerifier Team · Last updated · Runs in your browser; nothing you paste is sent to us
How do I check if my email A/B test is significant?
- Choose what you're comparing (replies, positive replies, opens or clicks) and the confidence level, usually 95%.
- Enter delivered emails and unique replies (or opens, or clicks) for variant A and variant B.
- Read the verdict, the p-value and the confidence interval for the difference. A range that includes zero means no winner yet.
- Before the next test, enter your current rate and the smallest lift worth detecting to get the sends needed per variant.
Example
- You enter
- Replies at 95% confidence: variant A got 30 replies from 1,000 delivered, variant B got 50 from 1,000.
- You get
- Variant B wins: 5% against 3%, a 66.7% lift with a p-value of 0.022. Detecting a 30% lift on a 3% rate needs 6,455 sends per variant at 80% power.
- Limits
- No limit; runs in your browser.
How do you calculate statistical significance for an A/B test?
For rates such as replies per email, use a two-proportion z-test, the test this calculator runs. Work out each variant's rate, then the pooled rate p = (replies A + replies B) ÷ (sends A + sends B). The test statistic is z = (rate B − rate A) ÷ √(p(1 − p)(1/sends A + 1/sends B)), and the two-sided p-value is the chance of a z at least that far from zero if the variants were really the same. The NIST/SEMATECH e-Handbook sets out the same test.
With 30 replies from 1,000 sends against 50 from 1,000, z is about 2.28 and p is about 0.023, so B wins at 95% confidence (p below 0.05) but not at 99%. The confidence interval says how big the win might be: here between roughly 0.3 and 3.7 percentage points, which is a wide range for a decision.
How do you calculate the sample size for an A/B test?
Decide three things first: your current rate, the smallest lift worth acting on, and how sure you want to be (confidence and power). The sends per variant are then n = (z₁₋α/₂·√(2p̄(1 − p̄)) + z₁₋β·√(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₂ − p₁)², where p₁ is the current rate, p₂ the target and p̄ their average. It's the two-group form of the normal-approximation method in the NIST e-Handbook.
Low rates need big samples. At a 3% reply rate, detecting a 30% relative lift (3% to 3.9%) at 95% confidence and 80% power takes about 6,450 delivered emails per variant. That's why most cold email tests with a few hundred sends per variant can't separate anything but very large differences, and why testing a subject line on opens is tempting, and misleading.
Should I A/B test cold emails on open rate or reply rate?
Reply rate, where you have the volume. Opens are counted by a tracking pixel, and since 2021 Apple's Mail Privacy Protection has stopped senders from using invisible pixels to see when someone opens a message, so Apple Mail loads them whether or not anyone reads. A difference in opens can be a difference in how many Apple Mail users each variant happened to reach.
Open tracking also has a deliverability cost in cold email; see open tracking and cold email deliverability. Positive replies are the best metric of all, and the hardest to power, so pick the test size from your positive reply rate when you can.
Why do bounces matter in an A/B test?
Because a bounced email can never reply. If one variant happens to get more dead addresses, its rate drops for reasons that have nothing to do with the copy, and bounces shrink the delivered sample, which widens every interval on this page. Enter delivered emails (sent minus bounces), not sent, and keep the bounce rate low and even across variants by verifying the list before you split it.
How many sends does a cold email test need?
Sends per variant for 95% confidence and 80% power, from the formula above.
| Current reply rate | Detect +20% | Detect +50% | Detect +100% |
|---|---|---|---|
| 1% | 42,693 | 7,750 | 2,319 |
| 3% | 13,914 | 2,518 | 749 |
| 5% | 8,158 | 1,471 | 435 |
| 10% | 3,841 | 686 | 199 |
Relative lifts: +50% on a 3% reply rate means 4.5%. Small lifts on low rates need volumes most single campaigns don't reach.
Who uses this tool, and for what?
- Cold emailers
- Check whether a new opener really beat the old one on replies before rolling it out to every sequence.
- Agencies
- Show a client whether a test result is a real win or noise, with the p-value and interval to back it up.
- SDR teams
- Plan how many sends a test needs before starting it, so a week of outreach isn't spent on a test that can't conclude.
- Newsletter and lifecycle marketers
- Test subject lines on click rate instead of opens, and see how much volume that takes.
What can’t this tool tell you?
- It assumes each variant went to a random half of the same list; a test split by date, domain or segment can't be fixed by statistics.
- The z-test is a large-sample approximation and gets unreliable below about 5 replies (or 5 non-replies) per variant; the calculator warns when that happens.
- Checking the result repeatedly and stopping when it turns significant raises the false positive rate above what the p-value says.
- Open rates include automatic opens from Apple Mail Privacy Protection and security scanners, so open-rate tests measure more than people reading.
- It compares two variants. Testing several at once needs a correction for multiple comparisons.
Common questions
- How do you calculate statistical significance for an A/B test?
- For rates like reply or open rate, run a two-proportion z-test: z = (rate B − rate A) ÷ √(p(1 − p)(1/nA + 1/nB)), where p is the pooled rate. A two-sided p-value below 0.05 means significant at 95% confidence.
- How do you calculate sample size for an A/B test?
- Choose the current rate, the smallest lift worth detecting, the confidence level and the power, then apply the two-proportion sample size formula shown on this page. The calculator does it and returns sends per variant.
- What is the A/B test formula?
- For conversion-style rates, z = (p₂ − p₁) ÷ √(p(1 − p)(1/n₁ + 1/n₂)), with p the pooled rate (x₁ + x₂) ÷ (n₁ + n₂). The p-value is 2 × (1 − Φ(|z|)).
- How long should you run an A/B test?
- Until each variant has reached the sample size you planned, not until the result looks significant. Stopping as soon as p dips below 0.05 inflates false positives. For cold email, that usually means a fixed number of delivered emails per variant.
- Is A/B testing the same as hypothesis testing?
- An A/B test is a hypothesis test: the null hypothesis is that both variants have the same rate, and the test asks whether the observed difference is too large to be chance. This calculator uses the standard two-proportion z-test.
- What is A/B testing in email marketing?
- Sending two versions of an email, differing in one thing such as the subject line or the call to action, to random halves of a list, then comparing reply, click or open rates to see which version performs better.
Sources
Authentication and capacity are half of deliverability.
The other half is who you send to. Every hard bounce counts against the domain and inboxes you are sending from, and nothing in DNS can tell you which addresses on your list are dead.
- Bulk verification with real SMTP mailbox checks
- Hard bounces removed before a send, not discovered after
- Catch-all, disposable and role detection on every row
One plan, cancel anytime, no contract. Sign in with Google and your first 100 email checks are free, no card. The free tools stay free either way.