The Conflict Between Speed and Statistical Rigor In the fast-paced world of B2C experimentation, speed is currency. Product managers and executives constantly monitor dashboards, hoping to see a significant uplift so they can roll out a winning feature and move to the next hypothesis. However, traditional A/B testing (Fixed Horizon) operates under a strict constraint: you must determine the sample size in advance and wait until it is fully collected. Checking results intermediate - peeking - and stopping if the p-value is less than 0.05 is a violation of statistical validity. If you peek at the data daily during a month-long experiment, your False Positive Rate (Type I Error) inflates from the standard 5% to upwards of 30%. This means nearly a third of your "successful" experiments might actually be neutral changes, driven purely by random noise. This article explores Sequential Testing, specifically the mSPRT (Mixture Sequential Probability Ratio Test) method used by companies like Uber and Netflix. We will examine how this approach allows for continuous monitoring and early stopping while maintaining strict error control, and demonstrate how to implement it using the open-source Python library trustworthy-experiments-core. Sequential Testing mSPRT (Mixture Sequential Probability Ratio Test) trustworthy-experiments-core trustworthy-experiments-core From Fixed Horizons to Anytime-Valid Inference To solve the peeking problem, we need a statistical framework that adjusts the significance threshold based on the amount of evidence gathered. The Limits of Group Sequential Historically, analysts used Group Sequential designs (like O'Brien-Fleming or Pocock boundaries). These methods require you to pre-define the number of interim checks ("we will check results exactly 4 times"). While effective, they lack flexibility. If a test shows a massive regression on day 2 between scheduled checks, the framework technically does not support stopping. Group Sequential The mSPRT Approach Modern experimentation platforms increasingly rely on Confidence Sequences derived from mSPRT. This method provides Anytime-Valid Inference, meaning you can check the results as often as you like (even after every single user) without inflating the error rate. Confidence Sequences Instead of a fixed p-value, mSPRT uses the Likelihood Ratio. It compares how likely the data is under the Alternative Hypothesis (H1) versus the Null Hypothesis (H0). Since we do not know the exact effect size beforehand, we mix the likelihoods over a probability distribution of possible effects. Likelihood Ratio The resulting test statistic behaves as a martingale under the Null Hypothesis. In simple terms, this means the statistic does not systematically drift upward when there is no real effect. This property allows us to construct a confidence interval that typically starts wide and narrows over time. If this interval excludes zero at any point, we can stop the test early with mathematical confidence. martingale Tuning Sensitivity: The Tau Parameter A critical component of mSPRT is the parameter Tau. It controls the shape of the mixing distribution and essentially acts as a tuning knob for the test's sensitivity. Tau Small Tau: The test becomes more sensitive to small effects but requires more data to converge;Large Tau: The test detects large effects very quickly but may miss subtle uplifts. Small Tau: The test becomes more sensitive to small effects but requires more data to converge; Small Tau: Large Tau: The test detects large effects very quickly but may miss subtle uplifts. Large Tau: To choose the optimal Tau, you can use a heuristic based on your Minimum Detectable Effect (MDE) and the standard deviation of your metric (sigma): sigma tau approx MDE / sigma tau approx MDE / sigma This ensures the test is most efficient for the effect size you actually care about. Python Implementation Implementing mSPRT from scratch requires complex integration of variance and probability mixing. My library encapsulates this logic, allowing for a streamlined workflow. Below is an example of how to configure and run a sequential test using the library. 1. Configuration We define the test parameters. Note that we select CONFIDENCE_SEQUENCE mode for mSPRT and set alpha to 0.05. CONFIDENCE_SEQUENCE alpha from tecore.sequential.schema import SequentialConfig, SequentialMode, EffectDirection

# Configure for mSPRT (Anytime-Valid)
config = SequentialConfig(
    mode=SequentialMode.CONFIDENCE_SEQUENCE,
    alpha=0.05,
    two_sided=True,
    effect_direction=EffectDirection.TWO_SIDED,
    cs_tau=0.5  # Tuned based on MDE/sigma
) from tecore.sequential.schema import SequentialConfig, SequentialMode, EffectDirection

# Configure for mSPRT (Anytime-Valid)
config = SequentialConfig(
    mode=SequentialMode.CONFIDENCE_SEQUENCE,
    alpha=0.05,
    two_sided=True,
    effect_direction=EffectDirection.TWO_SIDED,
    cs_tau=0.5  # Tuned based on MDE/sigma
) 2. Data Preparation and Execution The library expects a look_table: a DataFrame where each row represents a cumulative snapshot of the experiment (e.g., daily aggregates). look_table from tecore.sequential.confidence_sequences import run_confidence_sequence
from tecore.sequential.preprocess import build_look_table_mean
from tecore.sequential.schema import SequentialSpec, LookSchedule

spec = SequentialSpec(
    group_col="group",
    y_col="revenue",
    control_label="A",
    test_label="B",
    timestamp_col="date"
)

look_table, warnings = build_look_table_mean(
    df=experiment_data,
    spec=spec,
    schedule=LookSchedule(n_looks=30), # Daily checks for a month
    cfg=config
)

# Run the sequential test
result = run_confidence_sequence(look_table, config)

if result.stopped:
    print(f"Test stopped early at look {result.stop_look}!")
    print(f"Decision: {result.decision}")
else:
    print("Continue testing.") from tecore.sequential.confidence_sequences import run_confidence_sequence
from tecore.sequential.preprocess import build_look_table_mean
from tecore.sequential.schema import SequentialSpec, LookSchedule

spec = SequentialSpec(
    group_col="group",
    y_col="revenue",
    control_label="A",
    test_label="B",
    timestamp_col="date"
)

look_table, warnings = build_look_table_mean(
    df=experiment_data,
    spec=spec,
    schedule=LookSchedule(n_looks=30), # Daily checks for a month
    cfg=config
)

# Run the sequential test
result = run_confidence_sequence(look_table, config)

if result.stopped:
    print(f"Test stopped early at look {result.stop_look}!")
    print(f"Decision: {result.decision}")
else:
    print("Continue testing.") The result object contains the decision flag, the stopping point, and the full trajectory of the confidence sequence, which can be visualized to show stakeholders exactly when the boundary was crossed. result Ratio Metrics and Heavy Tails While the theory is elegant, applying it to real B2C data introduces complications. Standard mSPRT assumes normal distributions or simple sums, but key business metrics like Conversion Rate or ARPU (Average Revenue Per User) are often Ratio Metrics. Ratio Metrics Linearization for Ratio Metrics A ratio metric (e.g., Sum(Revenue) / Count(Users)) has a dependency between the numerator and denominator. Using standard formulas directly can lead to unstable variance estimates early in the test. Sum(Revenue) / Count(Users) To address this, the library applies Linearization (also known as the Delta Method). This technique transforms the ratio into a linearized metric: Linearization Linearized_Y = Numerator - theta * Denominator Linearized_Y = Numerator - theta * Denominator Where theta is the baseline ratio estimated from the control group. This transformation decouples the variance, allowing us to apply standard mSPRT logic to complex ratios without losing statistical rigor. theta Handling Heavy Tails In e-commerce data often follows heavy-tailed distributions (a few whales making massive purchases). A single outlier can spike the variance, causing the confidence interval to expand unexpectedly or, conversely, triggering a false positive. To mitigate this robust sequential testing requires Guardrails: Guardrails Minimum Sample Size: Enforcing a burn-in period (1000 observations) before the stopping rule is activated;Winsorization: Capping extreme values at a high percentile (99.9%) to prevent single observations from dominating the variance calculation. Minimum Sample Size: Enforcing a burn-in period (1000 observations) before the stopping rule is activated; Minimum Sample Size: Winsorization: Capping extreme values at a high percentile (99.9%) to prevent single observations from dominating the variance calculation. Winsorization: Conclusion Sequential testing transforms A/B testing from a passive wait-and-see approach into an active monitoring tool. By adopting mSPRT, product teams can stop poorly performing tests immediately, limiting the blast radius of bad features. Conversely, highly successful features can be rolled out 30-50% faster than with fixed-horizon tests, freeing up traffic for new hypotheses. However, the implementation details matter. Simply applying formulas from a textbook without accounting for ratio metrics or outliers can lead to incorrect decisions. The trustworthy-experiments-core library aims to bridge this gap, providing a production-ready toolkit for rigorous sequential analysis. trustworthy-experiments-core By shifting to valid sequential methods, we align statistical incentives with business goals: making faster decisions without sacrificing the truth.

This story contains new, firsthand information uncovered by the writer.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Stop A/B Tests Early: The Complete Guide to Sequential Testing (mSPRT) in Python

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Enhancing Experiment Sensitivity in B2C: A Robust Framework for Heavy-Tailed Metrics

Enhancing Experiment Sensitivity in B2C: A Robust Framework for Heavy-Tailed Metrics

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps