Free A/B test calculator

A/B testing is running two versions of a page, email or feature at the same time and splitting traffic between them to see which one performs better. Version A is usually the current design and version B is the change you want to try. Because both versions run over the same period with the same kind of visitors, any difference in conversion rate is down to the change rather than the day of the week or a campaign that happened to launch. It is the simplest way to find out whether an idea actually works instead of arguing about it.

Statistical significance means the difference you measured is unlikely to have happened by chance alone if the two versions really performed the same. Conversion rates wobble from day to day, so a small gap between A and B proves nothing on its own. A significance test weighs the size of the gap against how much random variation you would expect at your sample size. Significant does not mean important: a result can be significant and still too small to be worth shipping.

0.05 is the usual threshold, meaning there is a 5% chance of seeing a difference this large if the two versions were identical. Lower thresholds such as 0.01 make you more certain but need more traffic. The p-value is not the probability that your variant is better, and 1 minus the p-value is not your confidence in the winner. Pick the threshold before you start the test, not after you have seen the numbers.

It depends on your current conversion rate and the smallest lift you care about detecting. As a rough guide, a site converting at 3% that wants to detect a 10% relative lift needs somewhere around 50,000 visitors per variant. Smaller effects need dramatically more traffic: halving the effect you want to catch roughly quadruples the sample. Work the number out with a sample size calculator before you launch, because a test that can never reach significance is wasted traffic.

Run it for at least one full week, and ideally two, even if you hit your sample size sooner. Behaviour differs between weekdays and weekends and between paydays and the rest of the month, so a test that only covers a Tuesday to Thursday measures a narrow slice of your audience. Decide the end date and the sample size up front, then leave the test alone until you reach both. Stopping the moment a result looks good is the most common way teams ship changes that do nothing.

Power is the chance your test detects a real effect when one exists, and 80% is the usual target. A test with low power will often come back inconclusive even when the variant genuinely is better, so you throw away a good idea. Power rises with sample size and with the size of the effect you are looking for. Low power is why most tests on low-traffic sites read as flat.

The minimum detectable effect, or MDE, is the smallest change in conversion rate your test can reliably spot at the traffic you have. If your MDE is 10% and your variant delivers a genuine 3% lift, the test will almost certainly report nothing. Setting a realistic MDE up front keeps you from running tests that were never capable of answering the question. It also forces an honest conversation about whether the change is big enough to be worth testing at all.

Checking a running test repeatedly and stopping as soon as it crosses significance inflates your false positive rate, often to 20% or 30% rather than the 5% you think you are accepting. Every extra look is another chance for random noise to cross the line. Either fix the sample size and end date in advance, or use a sequential testing method built to be checked continuously. Peeking with a standard z-test is how teams end up with a pile of wins that never show up in revenue.