Home / Writing and design / A/B test sample size calculator

Email A/B Test Sample Size Calculator

Find the minimum number of people each version of your email needs before an A/B test means anything, check whether your list is big enough, and test whether a finished subject line test has a real winner.

1. How many people does each version need?

What are you measuring?
How do you want to state the lift?

Your typical unique open or click rate for this list.

Example: 10 means a 20% open rate rising to 22%.

How many people you can send to.

2. Did your finished test find a real winner?

Version A

Version B

Settings

Opens or clicks? Why opens are unreliable now

Apple Mail Privacy Protection, used by many iPhone, iPad and Mac users, loads the images in every email in the background, whether or not the person opens it. In Apple's words, it "downloads remote content in the background by default" and does so "regardless of whether you engage with the email". Each of those background loads counts as an open, so open rates are inflated and the extra "opens" are spread across both versions of your test. That hides real differences between subject lines.

Use clicks as your test metric when you can. Click rates are lower, so a click test needs more people, but the result reflects what real readers did. If you must test on opens, test big, clear differences and treat small wins with suspicion.

The formula this calculator uses

Sample size per version, for a two-sided test comparing two proportions with equal groups:

n = ( zα · √(2 · p̄ · (1 − p̄)) + zβ · √(p₁(1 − p₁) + p₂(1 − p₂)) )² / (p₂ − p₁)² p₁ = your usual rate p₂ = the rate you want to detect p̄ = (p₁ + p₂) / 2 zα = 1.645 (90% confidence), 1.960 (95%), 2.576 (99%) zβ = 0.842 (80% power), 1.282 (90% power)

The result is rounded up. It is the standard normal-approximation formula used by statistics textbooks and by R's power.prop.test. We checked the calculator against an independent computation in Python (SciPy) for a range of rates, lifts, confidence and power levels.

Significance check: a two-proportion z-test with a pooled standard error. With rates pA = opensA/sentA and pB, and pooled rate p = (opensA + opensB) / (sentA + sentB):

z = (p_B − p_A) / √( p · (1 − p) · (1/n_A + 1/n_B) ) p-value = 2 × (1 − Φ(|z|))

The confidence range for the difference uses the unpooled standard error, √(pA(1 − pA)/nA + pB(1 − pB)/nB).

Typical sample sizes

People needed per version at 95% confidence and 80% power, for a relative lift:

Usual rate10% lift20% lift30% lift
2% (clicks)80,68221,1099,798
3% (clicks)53,21113,9146,455
5% (clicks)31,2348,1583,780
20% (opens)6,5101,683772
35% (opens)2,978758342

Small lifts on low click rates need tens of thousands of people per version. On a small list, test bold changes (a completely different subject line, not one changed word) so the difference is big enough to detect.

How to run a fair subject line test

  1. Change one thing at a time: subject line, or sender name, or send time.
  2. Work out the sample size first, with this calculator, and split people at random.
  3. Send both versions at the same time, so the time of day doesn't decide the winner.
  4. Wait long enough for most results to arrive (at least a day for clicks) and check once at the end. Stopping as soon as one version looks ahead produces false winners.
  5. Clean the list before you send. Bounced and dead addresses shrink both groups and add noise.

Related free tools

Related guides on EmailNa

What the A/B test sample size calculator does

An A/B test sends two versions of an email, usually two subject lines, to two random groups and compares the results. If the groups are too small, the "winner" is often just luck. This calculator tells you the minimum sample size each version needs for a statistically significant result, checks whether your list is big enough, and, after the test, tells you whether the difference you saw is real.

Benefits

  • Opens or clicksEnter your usual open or click rate. The tool reminds you why clicks are the more honest measure now.
  • Relative or absolute liftState the change you care about as a percent of your rate (10% better) or in percentage points (2 points higher).
  • Standard settings built in95% confidence and 80% power are selected by default, with 90% and 99% confidence and 90% power available.
  • Is your list big enough?Enter your list size to see what share of it the test uses, and how many people are left to receive the winner.
  • What a small list can detectIf the list is too small, the tool shows the smallest lift a 50/50 split of the whole list could reliably detect.
  • Significance checkerEnter the sends and opens or clicks for each version to get the p-value, the difference, a confidence range and a plain verdict.
  • Formula on the pageThe exact formulas are shown below, so you can check the maths or explain it to a client.

Who it is for

  • A small-business owner trying two subject lines on a newsletter for the first time.
  • A marketer deciding whether to test on 20% of the list or the whole list.
  • Someone whose email platform declared a "winner" and wants to know if it is real.
  • A freelancer explaining to a client why a 50-person test proves nothing.

How to use it

  1. Choose opens or clicks and enter your usual rate, for example 25 for a 25% open rate.
  2. Enter the smallest improvement worth knowing about, for example 10 (a 10% relative lift).
  3. Keep 95% confidence and 80% power unless you have a reason to change them, and add your list size.
  4. Read the sample size per version and whether your list is big enough.
  5. After the test, enter the delivered emails and unique opens or clicks for A and B in part 2.

Your privacy

Everything is calculated in your browser. The numbers you enter are never sent anywhere.

Frequently asked questions

What is the minimum sample size for an email A/B test?

It depends on your usual rate and the size of the difference you want to detect. At 95% confidence and 80% power, detecting a 10% relative lift on a 20% open rate needs about 6,500 people per version; a 20% lift on a 3% click rate needs about 14,000 per version. Use the calculator above for your own numbers.

What does statistically significant mean?

It means a difference this large would be unlikely to happen by chance if the two versions really performed the same. At 95% confidence, a result counts as significant when the p-value is below 0.05.

What is statistical power?

The chance that the test detects a real difference of the size you entered. At 80% power, a real lift of that size is found 4 times out of 5. Higher power needs more people.

Why shouldn't I use open rates for A/B tests?

Apple Mail Privacy Protection loads email images in the background for many Apple Mail users, whether or not they read the email, and each load counts as an open. That inflates open rates and blurs the difference between versions. Click rates reflect what real readers did.

Can I stop the test as soon as one version is winning?

No. Checking again and again and stopping at the first significant result makes false winners far more likely. Decide the sample size first, send to everyone in both groups, and check the result once.

Is this a one-sided or two-sided test?

Two-sided: it detects whether B is better or worse than A. That is the standard choice for email tests, and it needs slightly more people than a one-sided test.

My email platform says the winner is significant but this calculator doesn't. Why?

Platforms may use a different test, a different confidence level, or total opens instead of unique opens. Use unique opens or clicks and the number of delivered emails here for a standard comparison.