Safe Testing for Large-Scale Experimentation Platforms: Mixture SPRT

Table of Links

1. Introduction

2. Hypothesis testing

   2.1 Introduction

   2.2 Bayesian statistics

   2.3 Test martingales

   2.4 p-values

   2.5 Optional Stopping and Peeking

   2.6 Combining p-values and Optional Continuation

   2.7 A/B testing

3. Safe Tests

   3.1 Introduction

   3.2 Classical t-test

   3.3 Safe t-test

   3.4 χ2 -test

   3.5 Safe Proportion Test

4. Safe Testing Simulations

   4.1 Introduction and 4.2 Python Implementation

   4.3 Comparing the t-test with the Safe t-test

   4.4 Comparing the χ2 -test with the safe proportion test

5. Mixture sequential probability ratio test

   5.1 Sequential Testing

   5.2 Mixture SPRT

   5.3 mSPRT and the safe t-test

6. Online Controlled Experiments

   6.1 Safe t-test on OCE datasets

7. Vinted A/B tests and 7.1 Safe t-test for Vinted A/B tests

   7.2 Safe proportion test for sample ratio mismatch

8. Conclusion and References

5.2 Mixture SPRT

Developing an A/B test for sequential testing involved expanding the SPRT to function with two-sample data. This was accomplished by Johari et al. [Joh+17] who pioneered a method of A/B testing known as the mixture Sequential Probability Ratio test (mSPRT). This test has been adopted in large technology companies such as Uber and Netflix [SA23]. As with the safe t-test, the mSPRT performs optimally with granular, sequential data. The mSPRT is essentially similar to the SPRT, with a prior belief that the true parameter lies close to θ0. Let’s examine the mathematical details of this test in more depth.

We will keep the mSPRT statistic in its martingale form in order to compare the performance with the safe t-test.