Safe Testing for Large-Scale Experimentation Platforms: Optional Stopping and Peeking

Table of Links

1. Introduction

2. Hypothesis testing

   2.1 Introduction

   2.2 Bayesian statistics

   2.3 Test martingales

   2.4 p-values

   2.5 Optional Stopping and Peeking

   2.6 Combining p-values and Optional Continuation

   2.7 A/B testing

3. Safe Tests

   3.1 Introduction

   3.2 Classical t-test

   3.3 Safe t-test

   3.4 χ2 -test

   3.5 Safe Proportion Test

4. Safe Testing Simulations

   4.1 Introduction and 4.2 Python Implementation

   4.3 Comparing the t-test with the Safe t-test

   4.4 Comparing the χ2 -test with the safe proportion test

5. Mixture sequential probability ratio test

   5.1 Sequential Testing

   5.2 Mixture SPRT

   5.3 mSPRT and the safe t-test

6. Online Controlled Experiments

   6.1 Safe t-test on OCE datasets

7. Vinted A/B tests and 7.1 Safe t-test for Vinted A/B tests

   7.2 Safe proportion test for sample ratio mismatch

8. Conclusion and References

2.5 Optional Stopping and Peeking

As an experimenter conducts an A/B test, modern data infrastructure allows them to view the results in real-time. There are good reasons for them to do this. First of all, experiments are expensive to run. If an experiment’s target metric is showing negative results, there may be pressure to stop the experiment as it costs the company money. A second reason to consider stopping an experiment has to do with secondary and guardrail metrics, which provide additional information about possible causal factors of the hypothesis or unintended impacts of the test. If these metrics are showing negative results, this may suggest that the experimental feature has an unintended negative consequences for the users. A further reason for monitoring results is to check the effect size for the feature. The effect size determines the sample size, and hence the length of time that the test must run. If the effect is large, the experimenter may suggest to stop the experiment since the necessary information has been collected.

Examining the results of the test before it’s complete is known as peeking, and it has unintended consequences for the results of the test. With standard A/B testing, peeking leads to an inflated false positive rate for each metric being monitored. Figure 1 shows how the false positive probability increases with successive peeks. The data are derived from the same distribution and tested with a two-sided, two-sample t-test. If the data are observed at the end of the test, there should be a false positive rate of α. However, since each peek gives a new opportunity for a false positive, the probability of a false positive becomes more and more likely throughout the test.

The false positive probability increases throughout the test because the test is not safe under optional stopping. In other words, continuously monitoring the results in Figure 1 to decide when to stop the experiment can impact the outcome of the test. This is a problem for which the ideal solution is one which allows the experimenter to monitor their results while keeping false positives below α. As we will see, safe testing is the solution that allows for this continuous monitoring and anytime-valid inference of test results.