In short
Short answer
- In November, clients check results more often and want answers sooner. Agree before the first report what will and won't be called, and when.
- Don't call a winner from a daily dashboard. A classic p-value checked every day finds false winners far more often than 5% of the time.
- Explain sequential checking in client language: "we can look every day, but the bar for calling it is higher until the planned sample arrives."
- If traffic doesn't split the way it was set, say so plainly and don't report the lift until the cause is found.
- Send short daily guardrail notes in sale week instead of results updates.
Who this is for: account leads and strategists who send test updates to clients in November.
Timeline for 2026
| Date | What the client hears from you |
|---|---|
| Monday, November 2 | The reporting agreement: cadence, what gets called, who decides |
| Weekly until November 15 | Status per test: running, verdict so far, planned end |
| By Sunday, November 15 | Final readouts for tests that finished; decisions recorded |
| November 16 to 25 | Freeze week: nothing new launches, reports cover what's running |
| November 26 to 30 | Daily guardrail notes, no results calls |
| From December 1 | Post-BFCM report, once refunds have had time to settle |
Step 1: Agree the reporting rules up front
Write a one-page reporting agreement and send it with the first November update. It should say:
- Cadence. A weekly status per test, plus daily guardrail notes in sale week.
- What gets called. Only the verdict on the primary goal agreed before launch.
- When. No verdict before 7 full days, and none on a test still short of its planned sample unless the sequential check clears it.
- Who decides. The client decides whether to ship; you recommend.
- What a result means. A result from November describes November shoppers.
Clients don't object to rules. They object to rules that appear after a result they liked.
Step 2: Explain why you won't call it early
The client sees a dashboard with a green number on day 5 and asks why you're waiting. The answer is that a classic p-value assumes you look once, at the end. Looking every day and stopping at the first reading below 0.05 raises the false-winner rate well above 5%. Our peeking simulation shows how much.
(an illustrative client test, not a real store)
A product page test splits 1,500 visitors a day into each arm. The control converts at 3.0% and the variant at 3.6%. The planned sample for a 10% change at this baseline is 53,211 visitors per arm.
| Day | Visitors per arm | Classic p-value | Always-valid p-value |
|---|---|---|---|
| 5 | 7,500 | 0.040 | 0.484 |
| 7 | 10,500 | 0.015 | 0.269 |
On day 5 the classic p-value is under 0.05, and a dashboard built on it would say "significant". The always-valid p-value, which stays correct however often you look, is nowhere near the line. On day 7 the gap is the same. The honest update is: "The variant is ahead, and the test is about a fifth of the way to its planned sample. It hasn't cleared the bar for an early call."
The always-valid approach comes from sequential testing research (Johari, Pekelis and Walsh, read October 12, 2026). In client language: "We can check every day without fooling ourselves, because the bar for calling a result early is higher. Once the planned sample is reached, the normal bar applies."
Two words to avoid in updates: "trending" and "directional". They invite the client to ship a result that isn't one.
Step 3: Explain a split problem without losing trust
A sample ratio mismatch (SRM) means the share of visitors in each arm differs from the split you set by more than chance allows. It usually means something in the setup is broken: a redirect, caching, a consent setting, bots or a script error in one arm. When it happens, the lift can't be trusted, however good it looks.
What to tell the client:
- "Traffic isn't splitting the way it was set, so we're not reporting the result until we know why."
- What you're checking, and when you'll update them.
- If the cause is found and fixed, whether the test restarts. Usually it should.
Clients rarely mind a problem you found. They mind one they found. Our guide to sample ratio mismatch lists the usual causes.
Step 4: Daily guardrail notes in sale week
From Thanksgiving to Cyber Monday, stop sending results and start sending guardrail notes. Each one takes two minutes to read:
- Tests live: which, and on which pages.
- Guardrails: checkout completion, payment errors and page errors per arm, and whether any moved.
- Actions: anything paused, and why.
- Next note: when.
A guardrail is a stop signal, not a second chance to find a winner. If the primary goal is flat and a guardrail looks better, the note says so as a fact, not as a win.
Step 5: Weekly status per test
| Field | What to write |
|---|---|
| Test and hypothesis | One line |
| Primary goal | As agreed before launch |
| Progress | Visitors per arm against the planned sample |
| Verdict so far | The tool's verdict, word for word |
| Lift | With its 95% interval, or "not yet meaningful" |
| Planned end | Date, and whether it ends before the freeze |
| Decision needed | None, or what and by when |
Use the verdict words your testing tool shows rather than your own. "Still collecting evidence" is a status the client can read without you.
Mistakes to avoid
- Reporting a classic p-value from a daily dashboard. It overstates certainty.
- Rules after the fact. Agree cadence and calls before the first update.
- Reporting a lift through a split problem. Fix the cause first.
- Segment wins in a status update. "It won on mobile" is a hypothesis for January, not a result.
- Treating sale-week numbers as permanent. Re-test in a normal period before making a change permanent.
Reporting for one brand? The brand track covers testing through Black Friday traffic. The general dates are in the guide to testing before Black Friday.