A/B Test Sample Size Calculator

Estimate sample size and runtime for an A/B or multi-arm conversion experiment using a fixed-horizon normal approximation.

Inputs

Percent of users who convert in control.

Relative lift percent, such as 5 for a 5% lift.

Use one-sided only when decreases are not decision-relevant and the direction is fixed in advance.

Percent of total daily eligible users included in the experiment.

Total arms including control.

Percent of experiment traffic allocated to control. Remaining traffic is split evenly across treatment arms.

Outputs

Absolute MDE -
Treatment rate at MDE -
Sample per variant -
Control sample -
Each treatment sample -
Total sample -
Runtime -
Control users per day -
Each treatment users per day -

This calculator uses a fixed-horizon two-sided test for differences in conversion rates with equal traffic allocation across variants.

What this is good for

  • Rough planning for A/B tests and multi-arm conversion experiments
  • Comparing how baseline rate and MDE change runtime
  • Checking whether your traffic volume makes a test feasible

This first version does not handle sequential testing, CUPED variance reduction, clustered assignment, heterogeneous treatment effects, or ratio metrics.

Formula and assumptions

Let $p_1$ be the baseline conversion rate and $p_2 = p_1 + \Delta$ be the treatment rate at the minimum detectable effect. Let $w_c$ be the control traffic share and $w_t$ be the traffic share for each treatment arm.

The calculator uses the normal approximation for a fixed-horizon test of two proportions with unequal allocation:

$$ \mathrm{SE}_0 = \sqrt{\bar{p}(1-\bar{p})\left(\frac{1}{n_c} + \frac{1}{n_t}\right)}, \qquad \mathrm{SE}_1 = \sqrt{\frac{p_1(1-p_1)}{n_c} + \frac{p_2(1-p_2)}{n_t}} $$

where $\bar{p} = (p_1 + p_2)/2$. For planning, it solves

$$ \Delta = z_{1-\alpha^\ast} \mathrm{SE}_0 + z_{\mathrm{power}} \mathrm{SE}_1 $$

for the required treatment-arm sample size $n_t$, using the allocation relationship $n_c / n_t = w_c / w_t$.

  • $\alpha^\ast = \alpha / 2$ for a two-sided test and $\alpha^\ast = \alpha$ for a one-sided test before any multiple-comparison correction.
  • When there are multiple treatment arms, the calculator applies a Bonferroni adjustment across treatment-versus-control comparisons.
  • Traffic is assumed independent and identically distributed across users.
  • Runtime assumes stable daily traffic and no ramp schedule.