This story draft by @escholar has not been reviewed by an editor, YET.
Authors:
(1) Muhammad Zia Hydari, Katz Graduate School of Business, University of Pittsburgh and Corresponding author;
(2) Idris Adjerid, Pamplin College of Business;
(3) AAaron D. Striegel, Department of Computer Science and Engineering, University of Notre Dame.
2. Background and 2.1. Leaderboards
3. Effect of Leaderboards on Healthful Physical Activity and 3.1. Competition
3.3. Moderating Effects of Prior Activity Levels and Leaderboard Size
4. Data and Model
5. Estimation and Robustness of the Main Effects of Leaderboards
5.2. Robustness Check for Leaderboard Initiation
5.4. Fitbit Attrition, Leaderboard De-Adoption, and Additional Robustness Checks
6. Heterogeneous Effect of Leaderboards
6.1. Heterogeneity by Prior Activity Levels
6.2. Interaction of Leaderboard Size, Rank, and Prior Activity Levels
6.3. Summary of Findings from Heterogeneous Effect Analysis
7. Conclusions and Discussion, Endnotes, and References
In our main analysis, we estimate variants of Specification (1), which is a DID specification with flexible user specific time trends. Table 2, columns 1 and 2, presents the estimation results for Specification (1). The first column presents results for a specification that includes individual-specific linear time trends only, whereas the second column additionally includes individual specific quadratic time trends. In both columns, we find a significant (p< 0.05) and meaningful leaderboard effect of 338–370 steps daily. Column 2 is our preferred model as it includes more flexible time trends. This model suggest that the students who adopted leaderboards have a daily increase of 370 steps, equivalent to a 3.5% increase in physical activity on the average daily step count of 10,268. These initial results suggest some support for a main effect of leaderboard adoption on physical activity. In the Online Appendix, we extend this analysis to add time-varying survey variables as controls in Specification (1). The estimated effects with these additional controls have higher magnitudes, which increases the plausibility of the main results. However, a number of concerns commonly arise with analyses using observational data. In the remainder of this section, we discuss the robustness of our main results.
Identification of the treatment effect with a DID design crucially depends on the common trends assumption. As mentioned earlier, one way to weaken this assumption is to control for individual-specific linear and quadratic time trends, which we have incorporated in our model estimation. In this section, we will further probe the plausibility of assuming that no unobserved time varying covariates may be confounding our analysis (i.e., the common trends assumption).
5.1.2. Pretreatment Period Placebo Treatments. Given that we have multiple pretreatment periods for most users in our sample, we can probe the plausibility of the common trends assumption by creating placebo treatments in
the pretreatment data alone, that is, by dropping the posttreatment data and using only the pretreatment data for this analysis. A failure to reject the null effect for the placebo treatment would provide support for the common trends assumption.16 In our study, users opt into the treatment in different periods. Moreover, our primary concern is the presence of some unobserved time varying factor (e.g., spurts in motivation) that affected the adopters in the periods closely preceding the treatment. Hence, we implemented our placebo treatment in the preceding month prior to the actual treatment and estimated the model in Equation (1) on the altered data. Table 2, column 4, presents the estimated effect of the placebo treatment. This estimated effect is small in magnitude, opposite in sign, and statistically insignificant. However, if we include four weeks of actual treatment period in our sample, the estimated effect is ≈ 420 steps (p=0.12) as presented in Table 2, column 5. Thus, for comparable time periods, the placebo effect is null, whereas the actual treatment effect is comparable to our estimated main effect. This null effect in the pretreatment period enhances the plausibility of our common trends assumption.
5.1.2.1. Leads-Lags Model. The placebo treatment effect can be further broken into weekly placebo effects in the pretreatment period and the actual effect in the posttreatment period using the full data set and a lead-slags specification:
This paper is available on arxiv under CC BY 4.0 DEED license.