Sample size and test duration: how long to run an A/B test

Benchmyrk plans conversion goals with the standard sample size for a two-sided, pooled two-proportion z-test (Fleiss, Levin and Paik, equation 4.14, without continuity correction). It is the same test the results engine uses once the planned sample is reached, so the plan matches how the result is judged.

n = [ z(1 − α′/2) · √(2·p̄·(1 − p̄)) + z(1 − β) · √(p₁(1 − p₁) + p₂(1 − p₂)) ]² / (p₂ − p₁)²

With more than one challenger, the significance level is split evenly between the comparisons (a Bonferroni split): two challengers at 95% confidence plan each comparison at 97.5%. Results are later adjusted with the Holm correction, which is never stricter than this split, so the plan errs on the safe side.

Each variant needs 53,211 visitors. With every visitor entering the test and a 50/50 split, each variant gets 2,000 a day, so the test needs 27 days. Round that up to whole weeks: 28 days covers 4 full weekly cycles.

Adding a second challenger doesn't just add a third of the traffic. Each variant now needs 64,439 visitors (the significance level is split between two comparisons), 193,317 in total, and at the same traffic the test takes 49 days.

The MDE matters most: halving it roughly quadruples the sample, because the required sample grows with one over the effect squared.

A lower baseline needs more visitors for the same relative lift, because conversions are rarer. Checkout and purchase goals usually have the lowest rates; add-to-cart rates are higher, which is one reason teams test on them, but a lift in add-to-cart doesn't always reach orders.

days = visitors per variant / (daily visitors × share in the test × share of the smallest variant)

• Run at least 7 full days, and whole weeks where you can. Benchmyrk gives no verdict before 7 full days.
• Count only visitors who reach the tested pages. A test on product pages gets product-page visitors, not the whole store's traffic.
• Avoid starting during an unusual week (a big sale or a launch) unless that is what you want to learn about.
• If the plan comes out longer than about four weeks, Benchmyrk flags it as low traffic: cookies expire, shoppers switch devices and seasons change over that time.

• Test a bigger change. A new offer or a reworked page can move the rate by 20% or more; a button colour rarely does. A larger MDE needs far fewer visitors.
• Test where the traffic is. A change on every product page or in the site header reaches more visitors than one landing page.
• Use fewer variants. One strong challenger beats three weak ones on a small store.
• Reduce variance. For purchase and revenue goals, CUPED uses each visitor's orders before the test to tighten the intervals. It doesn't change the plan, but it can bring a result forward.
• Use auto-allocation for short campaigns. When the goal is to earn the most during a promotion rather than to learn a precise lift, an auto-allocated test moves traffic toward the leader as it runs.

The planned sample is not a lock on the results page. You can look as often as you like: until every variant reaches the planned size, Benchmyrk only calls a winner on an always-valid p-value, which allows for repeated looks. A large effect can be called early that way. A small one usually needs the full sample, and once it is reached the classic test applies. Read more in How to read an A/B test result.

Revenue per visitor has no single rate to plan from: most visitors spend nothing and buyers spend different amounts, so the sample depends on how spread out spending is (its variance). The usual formula for comparing two means applies:

n = 2σ² · (z(1 − α′/2) + z(1 − β))² / Δ²

Benchmyrk plans revenue goals with this formula once the test has data, using the control's observed variance. Expect revenue goals to need more visitors than the conversion rate of the same test.

The sample size calculator runs the same code in your browser: enter your baseline, MDE, variants and daily visitors, and share the link with your team.