You launch a test with a 50/50 traffic split. After a week, you check the data:
| Variant | Visitors | Ratio |
|---|---|---|
| Control (A) | 100,000 | 50.2% |
| Test (B) | 99,000 | 49.8% |
It looks close enough, right? Just a 0.4% difference.
Wrong. With 200,000 users, missing 1,000 users in the Test group is a statistical impossibility if the randomization was truly 50/50.
This is Sample Ratio Mismatch (SRM). It is the "Check Engine" light of A/B testing. If you have SRM, you cannot trust the result (Lift or P-Value), because your groups are no longer comparable.
How do we know if the difference is "real" or just random fluctuation? We use a Chi-Square test.
We compare the Observed count (what we got) with the Expected count (perfect 50/50 split). If the p-value of this SRM check is < 0.001, we have a problem.
(Note: Most advanced A/B platforms like Eppo or StatSig check this automatically. If you use Google Analytics or Adobe, you often have to check it manually using a calculator).
SRM usually means there is a systematic bias removing users from one variation. The 1,000 missing users in Variant B didn't just vanish; they likely experienced a bug and failed to track.
If the bug only affects "Chrome Users on Android," then Variant B is now "Variant A minus Android Users." You are comparing apples to oranges.
Now that we have confirmed our sample is valid (no SRM), we can finally look at the results. But wait—did you look at the results yesterday? And the day before?
If you did, you committed the sin of Peeking. In the next module, we explain why this habit inflates your False Positive rate.