Can a small store A/B test over Black Friday?

• A four-day test at small-store traffic can only detect very large changes. At 1,000 visitors a day it can't reliably detect anything smaller than about a 48% lift.
• Before-and-after comparisons mislead in a sale week. Everything changes at once: traffic sources, discounts, shoppers' intent. You can't separate your change from the season.
• Measure the funnel instead, step by step, against last year or a normal week, and write down what else was different.
• If you want to check your setup, run an A/A test before the sale, not during it.
• Results from the sale describe sale shoppers. Anything you learn is a hypothesis for January.

The wider calendar, with the reasons behind each date, is in our general Black Friday testing guide.

The table assumes a store converting 4% of visitors during the sale and a two-version test split 50/50. It shows the smallest lift a test can reliably detect at 95% confidence and 80% power.

A store with 1,000 visitors a day during the sale can only trust a four-day test that moves orders by about half. Real changes rarely do that. And a test that ends inconclusive after four days hasn't told you the change doesn't work; it has told you the test was too small to say.

There's a second problem: Benchmyrk gives no verdict before 7 full days, because shoppers behave differently on different days of the week, and the Black Friday weekend is the most unusual stretch of the year. A four-day test can't finish a full week of normal shopping.

The tempting alternative is to change something on Thursday and compare the weekend with the week before. In a sale week that comparison mixes your change with:

• Different shoppers. Deal hunters from newsletters, ads and social posts, not your usual visitors.
• Different offers. Your discount, free shipping or gift changed at the same time.
• Different intent. Shoppers who've been waiting all month to buy.
• Different days. A Friday to Monday against a Monday to Thursday.

Any of these can move orders more than your change did. A before-and-after comparison can't separate them, and with small numbers the gap can be pure chance.

If identical pages can differ by a third in a normal week, a before-and-after gap in the sale week tells you even less.

1. The funnel, step by step

Write down a normal week before the sale and compare it with the sale weekend:

• Sessions, by source (email, social, search, ads)
• Added to cart per 100 sessions
• Checkouts started per 100 sessions
• Orders per checkout started, on phones and on computers

You're looking for breakages, not winners. If checkout completion on phones halves on Friday, something broke. Check it straight away.

2. Orders against last year, with context

Compare sale orders, revenue and average order value with last year's sale, and write down next to it what changed: your discount, your email list size, your ad spend, new products. That note is what makes the comparison useful in January. Without it, "orders up 30%" can't be explained.

3. An A/A test before the sale

An A/A test splits visitors between two identical versions. It checks that the split is even, that orders are counted once, and that your own visits are excluded. If an A/A test shows a sample ratio mismatch (the split is uneven by more than chance allows) or a "winner", there's a problem with your setup, not your page. Run it before the freeze, and read sample ratio mismatch if it flags one.

Some stores will run a test over the sale regardless. If you do:

• Decide the end date and the goal before you start, and don't stop early because it looks good on Saturday.
• Use one big change, two versions, and orders as the goal.
• Don't count the result as permanent. Re-test the winner in January with normal shoppers.

Checking a test every few hours and stopping at the first good reading raises the chance of a false winner well above 5%. Our peeking simulation shows by how much.

• Calling a four-day result. At small-store traffic it can't detect realistic changes.
• Crediting a change for the sale's orders. The sale itself, not your new banner, is the main reason orders went up.
• Comparing with last year without notes. Write down what changed, or the numbers can't be explained later.
• Starting an A/A test during the sale. Its job is to check your setup before you rely on it.
• Treating a sale-week result as your new default. Sale shoppers aren't your January shoppers.

• [ ] A normal week's funnel written down before 15 November
• [ ] Last year's sale numbers and what changed since
• [ ] No new test started after 16 November
• [ ] Daily funnel check during the sale, phones and computers
• [ ] Any test running over the sale has a written end date and goal
• [ ] Notes kept for the December review

For a larger store's view of the same question, see measuring tests under BFCM traffic spikes. Plan a test for a normal month with the sample-size calculator.