The novelty effect: why a new design's early lift fades

Shoppers who come back to your store know what it looks like. When something changes, a new menu, a bigger badge, a different product page layout, some of them click on it because it is different. That extra attention shows up as a lift in the first days of a test. As the same shoppers come back and the change stops being new, it fades, and what is left is the change's real effect.

The reverse is sometimes called a primacy effect or change aversion: regulars who knew where everything was take longer to find it, so the new version looks worse at first and recovers later. Either way the early days of a test say less about the long run than the later days.

Pages that the same shoppers see often are most exposed: the header and menu, the homepage, the cart. A product page that mostly gets new visitors from ads is less exposed, because most of its visitors never saw the old version.

Take a new product page layout. In its first week the variant converts at 3.60% against 3.00% for control. In its second week, with the same number of visitors, the variant converts at 3.09% against 3.00%.

Stopped after week 1 on the classic p-value (0.018), this test would have been reported as a +20% winner. Week 2 on its own shows +3.0%, with an interval that easily includes zero. Both weeks together land in between, and the engine still says "Still collecting evidence": the always-valid p-value, which allows for looking at results as they come in, is 0.468, and the planned sample of 53,211 visitors per version has not been reached.

Two weeks of data can't prove this is novelty. A real effect of a few percent with a noisy first week would look similar. What the example shows is why a first-week lift is not something to ship on.

• Watch the lift over time, not just the total. A cumulative lift that keeps drifting toward zero, or daily lifts that start high and settle lower, are the signs to look for.
• Compare the first week with the rest. Set the results to the first seven days, then to the days after. A big gap with the later days near zero is a warning.
• Split by new and returning visitors. Novelty lives in returning visitors. If the lift comes almost entirely from returning visitors and fades there, while new visitors show a steadier result, novelty is a likely explanation. Segment breakdowns are exploratory; treat this as a clue, not a result.
• Look at what moved. A jump in clicks on the changed element with no matching rise in orders often means curiosity, not intent.

• Regression to the mean. Early estimates are noisy, and the tests people notice early are the ones that drew a high number. Many early lifts shrink for no reason except more data.
• A change in who visits. A campaign starts, an email goes out, a sale begins. If the new traffic responds differently, the lift moves. In November this is the usual suspect.
• A broken split. If one version gets noticeably more visitors than planned, the comparison is off. Check for a sample ratio mismatch before reading anything else.
• Something changed on the page. A theme update, a new app or a price change part way through means the later days measure a different test.

• Run whole weeks, at least two. Weekdays and weekends differ, and the second week is where novelty starts to wear off.
• Decide the length before you start and don't stop on an early lift. If you check often, read the always-valid numbers, which are built for it. See the peeking simulation for what happens otherwise.
• Stores with many repeat buyers need longer. The more of your traffic is returning visitors, the bigger the novelty share of an early lift can be.
• Before Black Friday, launch new layouts early enough to see past their first week before your freeze. A layout launched the week before the sale is measured almost entirely on novelty and sale traffic.
• After you ship, keep a holdout for changes that matter, so the lift goes on being measured once nothing about the change is new.