Sample ratio mismatch (SRM)
When the number of visitors in each variant differs from the planned split by more than chance can explain: a sign the test is broken.
Also called SRM
If a 50/50 test ends with clearly unequal arms, something is adding or losing visitors in one of them: a variant that breaks on some browsers, a redirect, bot filtering or a mid-test change. The visitors lost are rarely random, so the comparison is biased.
A chi-square test of the observed counts against the plan detects it. A flagged test shouldn't be trusted until the cause is found.
A 50/50 test shows 10,000 visitors in the control and 9,400 in the variant. A gap that size is very unlikely by chance; the variant turns out to fail on an older Safari version.
Related terms
- Traffic allocationHow much of the eligible traffic enters a test, and how it is divided between the variants.
- BucketingHow a testing tool assigns each visitor to a variant and keeps them there on later visits.
- A/A testA test where both arms show the same page, used to check that the split, tracking and statistics behave before trusting real tests.
- P-valueThe probability of seeing a difference at least as large as the observed one if the variant actually made no difference.