Exploratory analysis
Looking at test results in ways that weren't planned, such as by segment, to find ideas rather than to confirm a result.
Exploratory analysis slices a finished or running test after the fact: by device, country, traffic source or any other cut. Each slice is another comparison, so some will look significant by chance.
Treat what it finds as a hypothesis for the next test, not as a result.
The overall test shows no difference, but mobile visitors from Instagram seem to love the variant. The team runs a follow-up test targeted at that audience.
Related terms
- SegmentA subset of a test's visitors, such as mobile visitors or new visitors, whose results can be looked at separately.
- Multiple comparisonsThe problem that the more comparisons you make in one test, the more likely one looks significant by chance.
- False positiveCalling a winner that isn't real: the test finds a difference although the variant made none.
Where it comes up
- How to read an A/B test result · Guide