How to A/B test on Shopify, step by step

A good hypothesis names one change, the page it runs on, why you expect it to help, and the one metric that will decide it. Write it before you build anything, so the result can't be reinterpreted afterwards.

A useful shape: "On product pages, showing delivery dates next to the add to cart button will raise orders per visitor, because shoppers can't currently see when an order arrives." The reason matters: if the test loses, it tells you which belief was wrong.

• Pick one primary metric. For most shop tests it's conversion rate (orders per visitor) or revenue per visitor. Everything else is a supporting or guardrail metric.
• Find your baseline. Shopify's behaviour reports show conversion rate over time, which Shopify defines as the percentage of online store visitors that make a purchase. Use the last four to eight weeks, outside a sale.
• Measure the page you're testing. If the test runs only on product pages, the baseline is the conversion rate of visitors who reach a product page, which is usually higher than the store-wide figure.

It depends on your baseline conversion rate and the smallest lift you want to detect: the smaller the lift, the more visitors you need. At 95% confidence and 80% power, a 3% baseline needs 24,193 visitors per version to detect a 15% lift.

If the plan comes out at months, don't start and hope. Test a bolder change, include more pages, or pick a metric closer to the change; the sample size and test duration guide shows how each choice moves the number, and the sample size calculator runs it for your store.

Build the variant so it differs from the control only in what the hypothesis describes. If you also change the image, the price and the button colour, a win can't tell you which change did it.

• Copy and layout changes (headlines, badges, delivery messages, the order of sections) can be made with a testing app's visual editor or a theme section, without changing the live theme.
• A whole new page layout is a template test: shoppers in the variant get a different template of the same theme.
• Price, discount and shipping tests must change what checkout charges as well as what the page shows; see the price testing guide and the free shipping threshold guide.
• Keep the control untouched. It's your measure of normal; don't tidy it up during the test.

Open every version on a phone and a desktop, on each page the test runs on, and place a test order in each. Most broken tests fail on something visible: a variant that doesn't load on mobile, a flash of the old page, or a button that no longer adds to cart.

• Check the page loads cleanly. No flash of the original before the variant appears, no layout jump.
• Check the cart and checkout. The price, discount or shipping the page promised is what checkout charges.
• Check tracking. A test order should show up once, credited to the version you saw.
• Leave your own visits out. Your team's visits and test orders shouldn't count in the results.
• If in doubt, run an [A/A test](/glossary/a-a-test) first: two identical versions should show no difference. If they do, the setup is wrong.

Start with a small share of traffic for a day, check that orders arrive and the split matches what you set, then widen it to the planned share. A split that is uneven by more than chance allows, a sample ratio mismatch, means something is dropping visitors from one version.

• Split at random, per visitor. The same shopper should see the same version on every visit.
• Set a [guardrail](/glossary/guardrail-metric) on what must not get worse (errors, add to cart, refunds) and stop if it does.
• Don't change the test while it runs. No new copy, no new traffic split, no sitewide sale halfway through.
• Avoid two tests on the same element at once; if they must overlap, keep their shoppers apart.

Run it until the planned sample size is reached, in whole weeks so every weekday and weekend is counted, and never less than 7 days. Stopping the first time the result looks good inflates false winners, because random swings look like wins early on.

• Decide the end before you start: the run length from your plan, rounded up to whole weeks.
• Look, but don't act on early numbers. Checking repeatedly and stopping at the first good-looking result is called peeking; the peeking simulation shows how much it inflates false winners.
• Watch for the first-week bump. Regular shoppers react to anything new; the novelty effect guide explains how to tell it from a lasting lift.
• Don't run across a sale you didn't plan for. Shoppers behave differently in a promotion, and the result may not hold afterwards.

There's more on this in how long a Shopify A/B test should run.

Look at the lift on the primary metric, its interval and the verdict together, then check the guardrails. A winner is a lift whose interval stays above zero on the primary metric, after the planned run, with no guardrail harmed.

The best estimate is a +16.6% lift, but the interval says anything from +6.2% to +28.1% fits the data. Plan with the low end, not the headline. The guide to reading an A/B test result covers each number, and what to do when the verdict is no meaningful difference.

Be sceptical of a surprising win on a small test. Kohavi, Deng and Vermeer show that a statistically significant result from an underpowered test is likely to exaggerate the size of the effect, and recommend replicating surprising results before you rely on them.

Ship the winner to everyone, keep watching the primary metric for a few weeks, and write down what you tested, what happened and why you think it happened. The written record is what stops your team re-running the same idea next year.

• Make it permanent. Move the winning change into your theme or settings, then end the test.
• Roll out in stages if the change is risky, such as a price or checkout change, so a problem shows up on a small share first.
• Measure the total, not just the test. If you run many tests in a season, a holdout group shows what they added up to.
• Record losers too. A test that didn't win still rules out an idea, which is worth knowing.

Benchmyrk runs every step above on Shopify through its theme app extension and web pixel, with the statistics checks built in. The table maps each step to what the app does.