How long should a Shopify A/B test run?

How long a test runs follows from how many visitors it needs. That number depends mostly on the smallest change you want to be able to detect (the minimum detectable effect): halving it roughly quadruples the visitors.

The 10% row is the trap. 36 days is past the four-week mark, so Benchmyrk flags the plan as low traffic. A 20% change is detectable in 2 weeks, and a 30% change in 1 week. On a store this size, test changes big enough to move the number: a new offer, a reworked product page, a different layout. A button colour won't get there.

Try your own numbers in the sample size calculator, and read the full guide for what each input means.

A classic A/B test assumes you look once, at the end. Check it every day and stop at the first significant result, and the chance of calling a winner that isn't real climbs well above the 5% you signed up for: our simulation of 1,000 tests shows how far.

Sequential tests are designed for repeated looks. Their p-value stays valid however often you check it, so you can stop as soon as it crosses the line. The cost: they need somewhat more evidence than a classic test to call the same effect, so small effects still take the full plan.

A 30% lift is strong enough to call after a week, a fraction of the planned sample. At five days the same rates give no verdict at all, because Benchmyrk waits for 7 full days whatever the numbers say.

A 10% lift may well be real, but after two weeks there isn't enough evidence to say so. The test keeps going toward its plan, or you decide the change isn't worth 36 days.

• The plan is reached and nothing is significant. That's an answer: the change, if it does anything, probably does less than the effect you planned for. Keep the simpler version.
• The test passes four weeks with no end in sight. Stop, keep the control, and test something bigger or on a busier page.
• Something broke. A guardrail tripped, the split is off (a sample ratio mismatch), or a version stopped rendering. Fix it and restart; don't patch a running test and keep its data.
• The store changed under it. A sitewide sale, a new theme, a price change or a large ad campaign that started mid-test. The result may not hold once things are back to normal.

• Promotions. Shoppers who arrive for a sale behave differently. Avoid starting a test the week before a big promotion, and don't let one run across Black Friday unless the sale is what you're testing.
• Theme publishes and app installs. A theme update or a new app can change the pages being tested. Schedule them between tests.
• Email and ad bursts. A campaign day can bring a different crowd. Over whole weeks this evens out; over three days it doesn't.
• Revenue goals. Revenue per visitor varies far more than conversion, so it needs more visitors than the conversion plan says. Plan for longer when revenue or profit decides the test.
• Refunds. Orders can come back weeks later. For offer and threshold tests, compare refunds per version before rolling out.