A/B Testing Guide: Sample Size, Power,
and Sensitivity
### Overview of A/B Testing: Sample Size, Power, and Sensitivity
A/B testing is a statistical method used to compare two versions of a product, webpage, or
marketing strategy to determine which one performs better. The key elements include
sample size, statistical power, and sensitivity.
#### 1. Sample Size in A/B Testing
Sample size is critical for reliable and valid results. Too small a sample can lead to
inconclusive results, while too large a sample wastes resources.
Factors influencing sample size include baseline conversion rate, minimum detectable effect
(MDE), statistical power, and significance level (α).
#### 2. Statistical Power
Statistical power is the likelihood that the A/B test will detect an effect if there is one.
Typically, power is set to 80%, meaning there’s an 80% chance of detecting a significant
difference.
#### 3. Sensitivity (Minimum Detectable Effect)
Sensitivity, or Minimum Detectable Effect (MDE), is the smallest effect size that you
consider meaningful. Smaller effects require a larger sample size to detect.
### Concrete Examples of Sample Size Calculations
#### Example 1: Basic Sample Size Calculation
Example where baseline conversion rate is 10%, and MDE is 5% relative increase, requiring
around 57,780 per group.
#### Example 2: Testing a Smaller Minimum Detectable Effect
Example where baseline conversion rate is 10%, and MDE is a 2% relative increase,
requiring around 356,050 per group.
### Unequal Sample Sizes in A/B Testing
For unequal sample sizes, the control and treatment groups don’t have to be the same size.
Traffic allocation ratios can be different.
### Online Tools for Sample Size Calculation
1. Evan Miller’s A/B Test Calculator: https://www.evanmiller.org/ab-testing/sample-
size.html
2. Optimizely Sample Size Calculator: https://www.optimizely.com/sample-size-calculator/
3. AB Test Guide Sample Size Calculator: https://abtestguide.com/calc/
4. Statistical Solutions A/B Test Calculator: https://www.statisticalsolutions.net/ab-test-
sample-size-calculator/
5. VWO Sample Size Calculator: https://vwo.com/tools/ab-test-sample-size-calculator/
---
This guide provides an overview of key concepts in A/B testing and shows how to calculate
sample sizes with both equal and unequal group sizes.
