Safe Testing for Large-Scale Experimentation Platforms: χ2 -test

Table of Links

1. Introduction

2. Hypothesis testing

   2.1 Introduction

   2.2 Bayesian statistics

   2.3 Test martingales

   2.4 p-values

   2.5 Optional Stopping and Peeking

   2.6 Combining p-values and Optional Continuation

   2.7 A/B testing

3. Safe Tests

   3.1 Introduction

   3.2 Classical t-test

   3.3 Safe t-test

   3.4 χ2 -test

   3.5 Safe Proportion Test

4. Safe Testing Simulations

   4.1 Introduction and 4.2 Python Implementation

   4.3 Comparing the t-test with the Safe t-test

   4.4 Comparing the χ2 -test with the safe proportion test

5. Mixture sequential probability ratio test

   5.1 Sequential Testing

   5.2 Mixture SPRT

   5.3 mSPRT and the safe t-test

6. Online Controlled Experiments

   6.1 Safe t-test on OCE datasets

7. Vinted A/B tests and 7.1 Safe t-test for Vinted A/B tests

   7.2 Safe proportion test for sample ratio mismatch

8. Conclusion and References

3.4 χ2-test

The χ2 test is a classical statistical test that is used to assess the distribution of contingency table cells. A contingency table contains the frequencies of the multinomial data, allowing one to assess the similarities of the two distributions’ parameters. In the case of binomial data, the contingency table is 2x2, which will be the focus of this section.

The χ2 statistic in converted to a p-value using the χ2 distribution with (r − 1)(c − 1) degrees of freedom, where r and c are the number of rows and columns in the table. As with the classical t-test, the χ2 is not safe under optional stopping, and thus peeking can inflate their false positive rate [Xu+22]. For this reason, safe alternatives that allow anytime-valid inference have been developed, which we will explore now.