What a threshold test answers
The question isn't whether free shipping converts better; it usually does. The question is whether it converts enough better, or grows baskets enough, to pay for the shipping you no longer charge, while you still pay the carrier for every parcel.
- Conversion: more visitors place an order when shipping is free above a reachable amount.
- Order value: shoppers just below the threshold add an item to reach it.
- Cost: every order at or above the threshold stops paying for shipping, including the ones that would have bought anyway.
That's why the test should be judged on profit per visitor, which captures all three, not on conversion rate alone.
The break-even maths
Profit per visitor = conversion rate × (order value × margin + shipping charged per order − shipping cost per order)Write down the control's profit per visitor from your last few weeks, then ask: with free shipping above the threshold, what conversion rate would give the same profit per visitor? If you can't realistically reach it, the threshold is too generous.
- Conversion rate
- 2.5%
- Average order value (merchandise)
- $62.00
- Merchandise margin
- 60%
- Shipping charged per order today
- $6.95
- Shipping cost per order
- $8.50
- Orders already at $75 or more
- 22%
- Contribution per order
- $35.65
- Profit per visitor (control)
- $0.89
Illustrative numbers, not from a real store.
| Scenario | Order value | Orders at $75+ | Contribution per order | Break-even conversion | Orders needed vs today |
|---|---|---|---|---|---|
| Baskets don't change | $62.00 | 22% | $34.12 | 2.61% | +4.5% |
| Some baskets grow to reach $75 | $66.00 | 35% | $35.62 | 2.50% | +0.1% |
| Many baskets grow to reach $75 | $70.00 | 50% | $36.98 | 2.41% | −3.6% |
Read it this way. If baskets don't change, free shipping from $75 needs about 4.5% more orders just to stand still. If enough shoppers add an item to reach it, the threshold can pay for itself with almost no extra orders, or even with fewer. Which of these happens is exactly what you can't know without a test.
Choosing thresholds to test
- Start from your order values. Export a few weeks of orders and look at how many fall just below each round number. A threshold works when a meaningful share of baskets sit a small step under it.
- Make the step one item. A threshold slightly above your typical basket plus the price of a popular add-on is reachable; one that needs a second full product usually isn't.
- Avoid giving it to everyone. If your best seller costs more than the threshold, nearly every order qualifies and you're testing plain free shipping, not a threshold.
- Test one or two thresholds against what you do today. Each extra arm needs more visitors (see below).
- Keep express at full price if you can, so the offer applies to standard shipping only.
Setting it up on Shopify
Shopify lets you offer free shipping to everyone with a free shipping discount, automatic or by code, with a minimum purchase amount. That's how you'll roll out the winner. To test it, you need to give the offer to one group of shoppers and not to another, at the same time, and make sure checkout charges what each group was shown.
| Option | How it works | Watch for |
|---|---|---|
| Shopify Rollouts experiment | Turns an existing free shipping discount on for the treatment group only (Grow plan or higher) | Experiment metrics are conversion-style rates; revenue attributed to the discount isn't reported |
| A testing app with a discount function | The app's discount gives each group its shipping offer at checkout; your rates stay as they are | Check how it combines with any free shipping discount you already run |
| A testing app that replaces your rates | The app calculates each group's rates at checkout | Intelligems' docs say its rate tests need Shopify Advanced or Plus |
| Your own Shopify Function | Each group's assignment travels as a cart attribute and a function returns its rate | Shoplift's guide calls this approach very technical, needing thorough QA |
- Pick the primary goal: gross profit per visitor. Add product costs in Shopify so profit can be calculated.
- Set the arms: your current shipping (control) and one or two thresholds.
- Tell each group its offer on the page: a banner or a progress bar that says exactly what checkout will give. A threshold nobody sees can't change behaviour.
- Preview each arm on a product page, the cart and checkout, and place a test order in each.
- Start with a small share of traffic, check orders come through with the right shipping, then widen it.
- Don't change shipping rates, run a sitewide sale or switch the offer while the test runs.
How long to run it
Profit per visitor varies a lot from visitor to visitor (most buy nothing, buyers spend different amounts), so a shipping test needs more visitors than a test judged on conversion rate. Decide before you start the smallest change worth detecting, work out the visitors it needs, and run whole weeks so weekdays and weekends are both in.
- A 5% change: 368,400 visitors per arm
- 246 days at 3,000 visitors a day; 74 days at 10,000 visitors a day; 25 days at 30,000 visitors a day
- A 10% change: 92,100 visitors per arm
- 62 days at 3,000 visitors a day; 19 days at 10,000 visitors a day; 7 days at 30,000 visitors a day
- A 20% change: 23,025 visitors per arm
- 16 days at 3,000 visitors a day; 7 days (the minimum) at 10,000 visitors a day; 7 days (the minimum) at 30,000 visitors a day
Two arms split 50/50, 95% confidence, 80% power, for the example store (2.5% conversion, $35.65 contribution per order, $25 spread between orders). Visitors per arm come from the same sample-size function Benchmyrk's statistics engine uses to plan revenue goals. Benchmyrk gives no verdict before 7 full days.
- With a few thousand visitors a day, only large effects are detectable in a reasonable time. That's worth knowing before you start, not after six weeks.
- Two thresholds against the control need more visitors per arm, because each is compared with the control and the comparisons are corrected.
- Don't stop the day the result looks good. Checking repeatedly and stopping at the first good-looking number inflates false winners; see peeking.
- Avoid running across a major sale (Black Friday, a launch): shoppers behave differently, and the winner may not hold afterwards.
What to look at in the results
| Measure | What it tells you |
|---|---|
| Gross profit per visitor | The decision: does the offer pay for the shipping it gives away? |
| Revenue per visitor | The same question before product and shipping costs |
| Conversion rate | Whether more visitors buy |
| Average order value | Whether baskets grow toward the threshold |
| Share of orders at or above the threshold | How many orders get free shipping in each arm |
| Shipping charged per order | What shoppers actually paid for shipping |
| Refunds | Whether add-on items bought to reach the threshold come back |
Common mistakes
- Calling the winner on conversion rate. Free shipping can win on conversion and still lose money.
- Forgetting the carrier bill. Free shipping to the shopper isn't free to you; use your real cost per parcel, by zone if it varies a lot.
- Showing one thing and charging another. If the banner says free shipping and checkout charges $6.95, you're testing a broken promise, and you'll get complaints.
- Overlapping offers. An existing free shipping discount, a loyalty perk or a code in an email can give some control shoppers free shipping too. Decide how they combine before you start.
- Testing during a promotion. A sale changes basket sizes and makes the result hard to reuse.
- Ending early, or never. Stop when the planned sample is reached, not when the chart looks good, and not months later when the season has changed.
- Ignoring returns. Items added only to reach the threshold can come back. Compare refunds per arm before you roll out.
After the test
If a threshold wins on profit per visitor, roll it out with Shopify's own free shipping discount (automatic, with a minimum purchase amount) or your shipping rates, and keep the banner that told shoppers about it. Re-test when your prices, product mix or carrier costs change, because the break-even moves with them.