paint-brush

This story draft by @escholar has not been reviewed by an editor, YET.

Estimation and Robustness of the Main Effects of Leaderboards

EScholar: Electronic Academic Papers for Scholars HackerNoon profile picture

Authors:

(1) Muhammad Zia Hydari, Katz Graduate School of Business, University of Pittsburgh and Corresponding author;

(2) Idris Adjerid, Pamplin College of Business;

(3) AAaron D. Striegel, Department of Computer Science and Engineering, University of Notre Dame.

Table of Links

Abstract and 1 Introduction

2. Background and 2.1. Leaderboards

3. Effect of Leaderboards on Healthful Physical Activity and 3.1. Competition

3.2. Social Influence

3.3. Moderating Effects of Prior Activity Levels and Leaderboard Size

4. Data and Model

4.1. Data

4.2. Model

5. Estimation and Robustness of the Main Effects of Leaderboards

5.2. Robustness Check for Leaderboard Initiation

5.3. Fitbit Compliance

5.4. Fitbit Attrition, Leaderboard De-Adoption, and Additional Robustness Checks

6. Heterogeneous Effect of Leaderboards

6.1. Heterogeneity by Prior Activity Levels

6.2. Interaction of Leaderboard Size, Rank, and Prior Activity Levels

6.3. Summary of Findings from Heterogeneous Effect Analysis

7. Conclusions and Discussion, Endnotes, and References

5. Estimation and Robustness of the Main Effects of Leaderboards

In our main analysis, we estimate variants of Specification (1), which is a DID specification with flexible user specific time trends. Table 2, columns 1 and 2, presents the estimation results for Specification (1). The first column presents results for a specification that includes individual-specific linear time trends only, whereas the second column additionally includes individual specific quadratic time trends. In both columns, we find a significant (p< 0.05) and meaningful leaderboard effect of 338–370 steps daily. Column 2 is our preferred model as it includes more flexible time trends. This model suggest that the students who adopted leaderboards have a daily increase of 370 steps, equivalent to a 3.5% increase in physical activity on the average daily step count of 10,268. These initial results suggest some support for a main effect of leaderboard adoption on physical activity. In the Online Appendix, we extend this analysis to add time-varying survey variables as controls in Specification (1). The estimated effects with these additional controls have higher magnitudes, which increases the plausibility of the main results. However, a number of concerns commonly arise with analyses using observational data. In the remainder of this section, we discuss the robustness of our main results.

5.1. Probing the Common Trends Assumption

Identification of the treatment effect with a DID design crucially depends on the common trends assumption. As mentioned earlier, one way to weaken this assumption is to control for individual-specific linear and quadratic time trends, which we have incorporated in our model estimation. In this section, we will further probe the plausibility of assuming that no unobserved time varying covariates may be confounding our analysis (i.e., the common trends assumption).



5.1.2. Pretreatment Period Placebo Treatments. Given that we have multiple pretreatment periods for most users in our sample, we can probe the plausibility of the common trends assumption by creating placebo treatments in


Table 2. Fitness Activity and Leaderboard Participation


the pretreatment data alone, that is, by dropping the posttreatment data and using only the pretreatment data for this analysis. A failure to reject the null effect for the placebo treatment would provide support for the common trends assumption.16 In our study, users opt into the treatment in different periods. Moreover, our primary concern is the presence of some unobserved time varying factor (e.g., spurts in motivation) that affected the adopters in the periods closely preceding the treatment. Hence, we implemented our placebo treatment in the preceding month prior to the actual treatment and estimated the model in Equation (1) on the altered data. Table 2, column 4, presents the estimated effect of the placebo treatment. This estimated effect is small in magnitude, opposite in sign, and statistically insignificant. However, if we include four weeks of actual treatment period in our sample, the estimated effect is ≈ 420 steps (p=0.12) as presented in Table 2, column 5. Thus, for comparable time periods, the placebo effect is null, whereas the actual treatment effect is comparable to our estimated main effect. This null effect in the pretreatment period enhances the plausibility of our common trends assumption.


5.1.2.1. Leads-Lags Model. The placebo treatment effect can be further broken into weekly placebo effects in the pretreatment period and the actual effect in the posttreatment period using the full data set and a lead-slags specification:



This paper is available on arxiv under CC BY 4.0 DEED license.


L O A D I N G
. . . comments & more!

About Author

EScholar: Electronic Academic Papers for Scholars HackerNoon profile picture
EScholar: Electronic Academic Papers for Scholars@escholar
We publish the best academic work (that's too often lost to peer reviews & the TA's desk) to the global tech community

Topics

Around The Web...