Table of Links Introduction Hypothesis testing 2.1 Introduction 2.2 Bayesian statistics 2.3 Test martingales 2.4 p-values 2.5 Optional Stopping and Peeking 2.6 Combining p-values and Optional Continuation 2.7 A/B testing Safe Tests 3.1 Introduction 3.2 Classical t-test 3.3 Safe t-test 3.4 χ2 -test 3.5 Safe Proportion Test Safe Testing Simulations 4.1 Introduction and 4.2 Python Implementation 4.3 Comparing the t-test with the Safe t-test 4.4 Comparing the χ2 -test with the safe proportion test Mixture sequential probability ratio test 5.1 Sequential Testing 5.2 Mixture SPRT 5.3 mSPRT and the safe t-test Online Controlled Experiments 6.1 Safe t-test on OCE datasets Vinted A/B tests and 7.1 Safe t-test for Vinted A/B tests 7.2 Safe proportion test for sample ratio mismatch Conclusion and References Introduction Introduction Introduction Hypothesis testing 2.1 Introduction 2.2 Bayesian statistics 2.3 Test martingales 2.4 p-values 2.5 Optional Stopping and Peeking 2.6 Combining p-values and Optional Continuation 2.7 A/B testing Hypothesis testing 2.1 Introduction 2.1 Introduction 2.2 Bayesian statistics 2.2 Bayesian statistics 2.3 Test martingales 2.3 Test martingales 2.4 p-values 2.4 p-values 2.5 Optional Stopping and Peeking 2.5 Optional Stopping and Peeking 2.6 Combining p-values and Optional Continuation 2.6 Combining p-values and Optional Continuation 2.7 A/B testing 2.7 A/B testing Safe Tests 3.1 Introduction 3.2 Classical t-test 3.3 Safe t-test 3.4 χ2 -test 3.5 Safe Proportion Test Safe Tests 3.1 Introduction 3.1 Introduction 3.2 Classical t-test 3.2 Classical t-test 3.3 Safe t-test 3.3 Safe t-test 3.4 χ2 -test 3.4 χ2 -test 3.5 Safe Proportion Test 3.5 Safe Proportion Test Safe Testing Simulations 4.1 Introduction and 4.2 Python Implementation 4.3 Comparing the t-test with the Safe t-test 4.4 Comparing the χ2 -test with the safe proportion test Safe Testing Simulations 4.1 Introduction and 4.2 Python Implementation 4.1 Introduction and 4.2 Python Implementation 4.3 Comparing the t-test with the Safe t-test 4.3 Comparing the t-test with the Safe t-test 4.4 Comparing the χ2 -test with the safe proportion test 4.4 Comparing the χ2 -test with the safe proportion test Mixture sequential probability ratio test 5.1 Sequential Testing 5.2 Mixture SPRT 5.3 mSPRT and the safe t-test Mixture sequential probability ratio test 5.1 Sequential Testing 5.1 Sequential Testing 5.2 Mixture SPRT 5.2 Mixture SPRT 5.3 mSPRT and the safe t-test 5.3 mSPRT and the safe t-test Online Controlled Experiments 6.1 Safe t-test on OCE datasets Online Controlled Experiments Online Controlled Experiments 6.1 Safe t-test on OCE datasets 6.1 Safe t-test on OCE datasets Vinted A/B tests and 7.1 Safe t-test for Vinted A/B tests 7.2 Safe proportion test for sample ratio mismatch Vinted A/B tests and 7.1 Safe t-test for Vinted A/B tests Vinted A/B tests and 7.1 Safe t-test for Vinted A/B tests 7.2 Safe proportion test for sample ratio mismatch 7.2 Safe proportion test for sample ratio mismatch Conclusion and References Conclusion and References Conclusion and References 3 Safe Tests 3.1 Introduction Safe testing [GHK23] is a novel method of hypothesis testing developed to address many issues with modern statistical inference. The safe in safe testing refers to the fact that the false positive rate does not increase above α in the optional continuation setting. As we will see, many safe tests also allow for optional stopping [GHK23], specifically the ones we will apply to the safe t-test and the safe proportion test. Figure 2 shows how the false positive rate of the safe t-test changes over an experiment. safe Safe testing is based on E-variables or (E-test statistics), which are non-negative random variables which satisfy E-variables Under the null hypothesis, many E-variables behave as test martingales [GHK23], which are closely related to Bayes factors. In addition to this intuitive interpretation, E-variables provide many mathematical benefits as well. Earlier we highlighted p-values, optional stopping, and optional continuation as a few problems with classical statistical testing. We proceed now by discussing these issues in the context of E-variables. For situations in which effect sizes are unknown or for tests with nuisance parameters, GROW may be indeterminable. However, the optimal growth can be determined relative to the unknown parameter. An E-variable with this property is known as relative GROW. These concepts will be applied in the derivations of the safe t-statistic and the safe proportion test statistic. relative While there exist E-variables that are not safe under optional stopping [GHK23], A/B testing uses fairly common statistical tests for which optional stopping E-variables are available. The first such test we’ll explore is the t-test, beginning with the theory behind the classical t-test. 3.2 Classical t-test The t-statistic in converted to a p-value using the t-distribution with ν = n+m−2 degrees of freedom, which is then used to make a decision about the hypothesis. The sample size for the t-test is determined by α, β, and the effect size δ. Before the data are collected, the effect size is unknown and must be estimated. After the test, the effect size can be calculated with Cohen’s d, which represents the overall difference between the groups where sp is the pooled standard deviation 3.3 Safe t-tes The one-sided safe t-test statistic has been shown to be GROW and the two-sided test statistic to be relative GROW [Pér+22]. Next, we discuss the χ2 test and its safe alternative. 3.4 χ2-test The χ2 test is a classical statistical test that is used to assess the distribution of contingency table cells. A contingency table contains the frequencies of the multinomial data, allowing one to assess the similarities of the two distributions’ parameters. In the case of binomial data, the contingency table is 2x2, which will be the focus of this section. The χ2 statistic in converted to a p-value using the χ2 distribution with (r − 1)(c − 1) degrees of freedom, where r and c are the number of rows and columns in the table. As with the classical t-test, the χ2 is not safe under optional stopping, and thus peeking can inflate their false positive rate [Xu+22]. For this reason, safe alternatives that allow anytime-valid inference have been developed, which we will explore now. 3.5 Safe Proportion Test Next, consider the quantity and n1 = na1 + nb1. Under H1 the joint distribution is Combining 6, 7, and 8 and simplifying (see [TLG22] for details) gives the final expression for the relative GROW E-variable of batch size na + nb: In the next section, we compare the safe t-test and the safe proportion test to their classical alternatives. Author: (1) Daniel Beasley Author: Author: (1) Daniel Beasley This paper is available on arxiv under ATTRIBUTION-NONCOMMERCIAL-SHAREALIKE 4.0 INTERNATIONAL license. This paper is available on arxiv under ATTRIBUTION-NONCOMMERCIAL-SHAREALIKE 4.0 INTERNATIONAL license. available on arxiv available on arxiv
