Safe Testing for Large-Scale Experimentation Platforms: Classical t-test

Table of Links

1. Introduction

2. Hypothesis testing

   2.1 Introduction

   2.2 Bayesian statistics

   2.3 Test martingales

   2.4 p-values

   2.5 Optional Stopping and Peeking

   2.6 Combining p-values and Optional Continuation

   2.7 A/B testing

3. Safe Tests

   3.1 Introduction

   3.2 Classical t-test

   3.3 Safe t-test

   3.4 χ2 -test

   3.5 Safe Proportion Test

4. Safe Testing Simulations

   4.1 Introduction and 4.2 Python Implementation

   4.3 Comparing the t-test with the Safe t-test

   4.4 Comparing the χ2 -test with the safe proportion test

5. Mixture sequential probability ratio test

   5.1 Sequential Testing

   5.2 Mixture SPRT

   5.3 mSPRT and the safe t-test

6. Online Controlled Experiments

   6.1 Safe t-test on OCE datasets

7. Vinted A/B tests and 7.1 Safe t-test for Vinted A/B tests

   7.2 Safe proportion test for sample ratio mismatch

8. Conclusion and References

3.2 Classical t-test

The t-statistic in converted to a p-value using the t-distribution with ν = n+m−2 degrees of freedom,

which is then used to make a decision about the hypothesis.

The sample size for the t-test is determined by α, β, and the effect size δ. Before the data are collected, the effect size is unknown and must be estimated. After the test, the effect size can be calculated with Cohen’s d, which represents the overall difference between the groups

where sp is the pooled standard deviation

With the formulation for the classical t-test established, let’s examine the safe t-test