In short
If your store gets fewer than about 10,500 visitors a week on the pages you'd test, a standard A/B test can't reliably detect even a 20% lift in a 2% conversion rate within four weeks. That threshold comes from the same sample-size formula Benchmyrk's planner uses (95% confidence, 80% power), and four weeks is the point where the planner flags a test as low traffic. Most new and small Shopify stores sit below it. This part of the series is about what to do instead.
Who this is for: owners of small or new Shopify stores, usually running the store alone or with one or two people, getting ready for their first or second Black Friday.
The short answer
- Fix the obvious problems before you test anything. Speed, the mobile checkout, delivery dates and the returns policy are things you can check and repair this week. None of them needs a test to tell you they matter.
- Run at most one test before the sale, on your busiest page. At small-store traffic, a test only detects large changes. Pick one big, clear idea and start it by Sunday 25 October.
- Start from a template, not a blank page. A reviewed template (a delivery promise, trust badges, a free-shipping banner) gets a clean test live in minutes.
- Freeze your store on Monday 16 November. No new apps, theme publishes or discount experiments after that.
- Measure the sale against last year, carefully. A before-and-after comparison in a sale week mixes your changes with the season. Treat it as a rough guide, not a result.
The 2026 timeline
| Date | What to do |
|---|---|
| By Wed 21 October | Finish the fix list below. Place a test order on your phone. |
| By Sun 25 October | Start your one test, if you run one. It then has three full weeks. |
| 2 to 15 November | Read the result (or accept it's inconclusive), set up your sale discounts with start and end dates, check every page on a phone. |
| Mon 16 November | Freeze: no new apps, theme publishes or test launches. |
| Thu 26 to Mon 30 November | Thanksgiving, Black Friday (27th) and Cyber Monday (30th). Watch orders and messages daily. |
| 1 to 11 December | Write down what happened. Part 10 of this series is a one-hour review. |
These dates follow the calendar in our general Black Friday testing guide, which explains the reasoning for stores of every size.
Why testing is hard at small-store traffic
An A/B test splits your visitors between two versions and asks whether the difference in orders is bigger than chance would produce. With few visitors, chance produces big differences. Two identical pages can easily show 20 orders against 28 in a week. So the test has to wait until the gap is too large for chance to explain, and at low traffic that means either a long wait or a very large effect.
A store gets 2,000 visitors a week to its product pages and 2% of them order (about 40 orders a week). It starts a two-version test on Monday 19 October and has to stop by Sunday 15 November: 28 days, 8,000 visitors, 4,000 per version.
At 95% confidence and 80% power, the smallest lift that test can reliably detect is about 49%, from 2.0% to roughly 3.0%. Started on Sunday 25 October instead (three weeks, 3,000 visitors per version), the smallest reliable lift is about 57%.
Computed with sampleSizePerVariant from Benchmyrk's statistics engine, the formula the planner and results use. The store is illustrative.
Real changes to a product page rarely lift orders by half. That doesn't mean the change didn't work. It means a test of this size can't tell a real 10% improvement from noise. If the test ends inconclusive, that is the honest answer, not a failure. Our guide to sample size and test duration shows the four inputs (baseline rate, smallest change worth detecting, confidence and power) and how they turn into visitors and days. You can put your own numbers into the sample-size calculator.
The fix list: what to do instead of testing
These are not ideas to test. They are problems to remove. If a shopper can't see when an order will arrive, or the checkout is broken on a phone, you don't need a statistic to know it costs orders.
1. Speed on a phone
Open your home page, a collection page and your best-selling product page on your own phone, on mobile data rather than Wi-Fi. Count how long until you can tap Add to cart.
- Remove apps you installed to try out and no longer use. Some apps keep loading their scripts on every page after you stop using their features.
- Compress large images, especially the first image on the home page and product pages.
- Check that popups don't cover the Add to cart button on small screens.
2. The mobile checkout
Place a real order on your phone with a discount code, then refund it. Check every step: the cart, the discount field, shipping options, payment, the Thank you page and the confirmation email. Note anything slow, confusing or broken and fix it.
3. Delivery dates
Baymard Institute's checkout survey lists "delivery was too slow" as a reason for 20% of abandoned purchases, and "extra costs too high (shipping, tax, fees)" as the top reason at 40% (shoppers who were not just browsing; page updated 22 September 2025). Shoppers buying gifts want to know the order will arrive in time.
- State your last order date for delivery before the holidays, by shipping method, on product pages and in the cart.
- Make the shipping cost visible before checkout. A shopper who first sees the shipping fee at the payment step is the shopper Baymard's 40% describes.
4. The returns policy
Baymard lists an unsatisfactory returns policy as a reason for 13% of abandoned purchases (same survey). Write your policy in plain words, say whether holiday purchases get a longer return window, and link it from product pages and the cart.
5. Back up your theme
Before you change anything for the sale, duplicate your live theme in the Shopify admin (Online Store, Themes, the theme's actions menu, Duplicate). Shopify keeps up to 20 themes, and the copy is your way back if a change breaks something during the sale.
The one test worth running
If your store gets at least a few thousand visitors a week, one test before the sale can still be worth it, as long as you choose it well.
- Choose your busiest page. Usually your best-selling product page, or the collection page your emails and ads point to. A test on a page with 200 visitors a week will not finish before the new year.
- Choose one big, clear change. Not a button color. A delivery promise under Add to cart, a clear statement of free returns, or badges next to the price that answer the shopper's main worry.
- Choose orders as the goal, and keep the test to two versions. Each extra version splits your visitors further.
- Start it by Sunday 25 October and decide its end date before you launch.
- Accept the answer. If it's inconclusive on 15 November, keep the version you prefer for reasons you can state (it's clearer, it's true, it answers a question shoppers ask you), and move on.
Benchmyrk's built-in templates include "Delivery promise under Add to cart", "Trust badges next to the price" and "Free-shipping threshold banner". Each creates a draft from New experiment, Start from a template, and you fill in the facts about your store (your delivery promise, your threshold). Templates never state a policy for you. See how to use test templates.
Mistakes to avoid
- Stopping a test on day three because it looks like a winner. Early leads at small traffic are mostly noise. Benchmyrk gives no verdict before 7 full days, and our peeking simulation shows why checking and stopping at the first good reading produces false winners.
- Running several tests at once on a small store. Each one gets a fraction of your traffic and none finishes.
- Testing discount depth. Shoppers share deals. Decide the discount with the margin maths in part 4 of this series instead.
- Installing new apps the week before Black Friday. Every new script is a new way for a page to break when traffic peaks.
- Treating a Black Friday result as permanent. Sale-week shoppers are not your December shoppers. A change that wins with deal hunters is a hypothesis for January.
Checklist
- [ ] Theme duplicated as a backup
- [ ] Unused apps removed
- [ ] Test order placed on a phone, with a discount code, and refunded
- [ ] Holiday delivery cut-off dates on product pages and in the cart
- [ ] Shipping cost visible before checkout
- [ ] Returns policy in plain words, linked from product pages
- [ ] At most one test, started by 25 October, with an end date
- [ ] Sale discounts created with start and end dates
- [ ] Freeze date (16 November) in your calendar
Running Black Friday for a larger brand? See the BFCM testing plan for a $1M to $10M Shopify brand.
Sources
- Baymard Institute, Cart abandonment rate statistics (updated 22 September 2025), read 11 October 2026
- Shopify Help Center, Duplicating themes, read 11 October 2026
- Shopify Help Center, Amount off products and orders (active dates), read 11 October 2026
- Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments, Cambridge University Press, 2020, read 11 October 2026