In short
Short answer
- Over Black Friday and Cyber Monday, a client's downside is much bigger than the upside of any test you could start that week. Most new tests should wait.
- Don't launch new scripts, new themes, price depth tests or checkout blocks that haven't run before the freeze.
- Tests that started before the freeze, passed their checks and have guardrails can usually keep running. Decide per test, with the client, in writing.
- A test that runs only over the sale can detect only large changes, and its result describes sale shoppers.
- Write down what you decided not to test and why. That record protects the client relationship as much as any result.
Who this is for: agency leads deciding, client by client, what stays on and what stays off from November 16 to 30.
Timeline for 2026
| Date | Step |
|---|---|
| Thursday, November 5 | Risk matrix filled in for every client |
| By Sunday, November 15 | Decisions per test signed off: keep, pause or end |
| Monday, November 16 | Freeze: no new tests, scripts, themes or checkout changes |
| November 26 to 30 | Sale: guardrails watched daily; pause only, never launch |
| From December 1 | Restart paused tests in a normal period |
Why the bar is higher in sale week
Three things change at once. Traffic and revenue are concentrated in a few days, so a broken arm costs more per hour. The mix of shoppers changes: a sale draws deal-seekers and people who don't usually buy from the store, so a result describes them rather than the client's regular customers. And the client's team is stretched, so a problem takes longer to notice and fix.
The upside shrinks too. A test that runs only over the sale has a few days of data, so it can only detect large changes.
(an illustrative store converting at 4% during the sale, split 50/50, fixed-horizon test at 95% confidence and 80% power)
| Daily visitors | Smallest detectable lift in 4 days (Black Friday to Cyber Monday) | In 7 days |
|---|---|---|
| 2,000 | 33.1% | 24.6% |
| 10,000 | 14.2% | 10.7% |
| 50,000 | 6.3% | 4.7% |
For a client with 2,000 visitors a day, a sale-only test can detect a change only if it is about a third, which few real changes produce. Even at 50,000 a day, it needs a change of over 6%. And whatever it finds describes BFCM shoppers, not the rest of the year.
Our guide to testing before Black Friday uses the same numbers; the brand track covers testing during Black Friday traffic in more depth.
The risk matrix
Fill one in per client. "Off" means don't launch it in the freeze or the sale. "Keep" means a test that started before the freeze can keep running if it passes the conditions in the last column.
| Change | Small store, thin team | Growing brand, in-house developer | Large Plus client, formal change control | Conditions to keep running |
|---|---|---|---|---|
| New tracking or third-party scripts | Off | Off | Off | None: freeze them |
| Theme changes or a new theme | Off | Off | Off | Keep a duplicate of the live theme as a backup |
| Price depth or new price tests | Off | Off | Off | End by November 15 |
| Offer and shipping tests started before the freeze | Off | Keep, with care | Keep, with care | Offer matches the sale plan; no clash with sale discounts |
| Checkout blocks not seen by real shoppers before the freeze | Off | Off | Off | None |
| Checkout tests running since before the freeze | n/a | Keep, with care | Keep | Guardrails on checkout completion and payment errors |
| Copy and layout tests on product pages | Keep, with care | Keep | Keep | Guardrails on; client agreed in writing |
| App impact tests (holding an app out) | Off | Off | Keep, with care | The app isn't part of the sale mechanics |
| Personalization started before the freeze | Keep, with care | Keep | Keep | Holdout kept; offer matches the sale |
Notes on the hardest calls
Discounts. Shopify's rules on how discounts combine decide what a shopper actually pays when a test offer meets a sale discount (Shopify Help Center, read October 11, 2026). An offer test that was fine in October can overlap a sitewide sale code in November. Check every running offer against the client's sale plan, or pause it. Count the sale's automatic discounts too: Shopify allows up to 25 active automatic discounts per store, app discounts included (Shopify Help Center, read October 11, 2026). Benchmyrk's price, offer and shipping tests use up to four of them. If a client wants a new free-shipping threshold for the sale, it needs to be tested before the freeze: see shipping threshold A/B tests for agency clients.
Themes. Shopify lets you duplicate a theme to keep a backup copy (Shopify Help Center, read October 11, 2026). Do it before the freeze for every client, so a bad theme change can be reverted in minutes.
App impact tests. Holding an app out for some shoppers is a good way to measure what it's worth, but not in sale week if the app runs the sale: a countdown, a bundle builder or a gift-with-purchase app.
Tests that run at the same time. Several tests on one store are usually fine when they're on different pages. Tests that could interact, such as two offers or a price test and an offer test on the same product, should be mutually exclusive or not run together.
Liability and documentation
Agencies are judged on what goes wrong, not on what they declined to do. A short record for each client protects both sides. It isn't legal advice; check what your own contract says.
For each client, record:
- Which tests stay on through the sale, and why.
- Which tests were paused or ended, and when.
- Which tests the client asked for and you advised against, with the reason.
- Who can pause a test during the sale, and how to reach them.
- The client's written agreement.
Attach it to the code-freeze agreement from part 3.
Mistakes to avoid
- "It's only a small script." A script added in sale week has never run under sale traffic, on the client's busiest pages, with the team least able to fix it.
- Testing discount depth in the sale. The answer arrives too late to use and describes deal-seekers.
- Keeping a test the client's team doesn't know about. Everyone on call should know what is running.
- Treating a sale-week result as permanent. Re-test in January before making it the default.
- No written record. A decision nobody wrote down becomes a disagreement later.
Checklist per client
- [ ] Risk matrix filled in and agreed in writing
- [ ] Every running test marked keep, pause or end
- [ ] No new scripts, themes, price tests or checkout blocks after November 16
- [ ] Running offers checked against the sale's discounts
- [ ] Theme duplicated as a backup
- [ ] Guardrails on for every test that keeps running
- [ ] On-call contact and pause authority named