A/A test
A test where both arms show the same page, used to check that the split, tracking and statistics behave before trusting real tests.
An A/A test splits traffic between two identical versions. Since nothing differs, any "winner" it finds is a false one, and the traffic split should match the plan.
Teams run one when they set up a new tool or suspect a tracking problem. A single A/A test can still show a false winner by chance (about 5% of the time at 95% confidence), so read it as a check on the setup, not a pass or fail.
A store installs a new testing tool and splits product-page traffic 50/50 with no changes. After two weeks the arms have 10,020 and 9,980 visitors and conversion rates within the margin of error: the setup looks sound.
Related terms
- Sample ratio mismatch (SRM)When the number of visitors in each variant differs from the planned split by more than chance can explain: a sign the test is broken.
- False positiveCalling a winner that isn't real: the test finds a difference although the variant made none.
- PeekingChecking a test's results while it runs and stopping as soon as they look significant, which inflates false winners with classic p-values.