Conclusions and Discussion, Endnotes, and References

Table of Links

Abstract and 1 Introduction

2. Background and 2.1. Leaderboards

3. Effect of Leaderboards on Healthful Physical Activity and 3.1. Competition

3.2. Social Influence

3.3. Moderating Effects of Prior Activity Levels and Leaderboard Size

4. Data and Model

4.1. Data

4.2. Model

5. Estimation and Robustness of the Main Effects of Leaderboards

5.2. Robustness Check for Leaderboard Initiation

5.3. Fitbit Compliance

5.4. Fitbit Attrition, Leaderboard De-Adoption, and Additional Robustness Checks

6. Heterogeneous Effect of Leaderboards

6.1. Heterogeneity by Prior Activity Levels

6.2. Interaction of Leaderboard Size, Rank, and Prior Activity Levels

6.3. Summary of Findings from Heterogeneous Effect Analysis

7. Conclusions and Discussion, Endnotes, and References

7. Conclusions and Discussion

The rapid and increasingly broad adoption of health wearables coupled with the gamification services built on top of them provides a potentially powerful vehicle for improving health behaviors at scale. Our results lend credence to this potential value, particularly when considered over time. In our data, the average user participated in a leaderboard for 237 days (conditional on participating in a leaderboard). Thus, sedentary users who participated in a leaderboard for at least 237 days took more than 300,000 additional steps (with a conservative estimate of 1,300 additional daily steps). To put this in context, the aggregated benefit of leaderboards (for these participants) amounts to 150 miles of distance (at 2,000 steps a mile). Importantly, the benefit is not homogeneous and there is a decrease in daily steps of nearly equal measure for those who, prior to adoption, were highly active. However, this health harm to the highly active subgroup may not be symmetrical to the gain for those who were previously sedentary, as this group remains very active in absolute terms. We also identify additional heterogeneity in benefit based on the number of other active participants and leaderboard rank.

This research has some important limitations. First, we use secondary data in which individuals organically choose to adopt leaderboards, rather than being randomly assigned to the treatment. Although we put in significant effort to address potential confounding factors for our analysis, only a large-scale randomized control trial can provide a theoretical guarantee that the treatment is unconfounded. Additionally, we were missing variables in our data set which can be particularly informative of leaderboard mechanisms and the heterogeneous value they generate. Specifically, we do not have deterministic measures of whether the focal user initiated the leaderboard or whether it was initiated by another user. A more comprehensive measure of leaderboard initiation could have provided important insights into how different types of users are initiating leaderboards, whether initiating a leaderboard impacts competitive dynamics, and whether individuals are selecting into leaderboards of value to them or if others are prompting them to do so. We hope to address these questions in subsequent data collection efforts. We also do not observe granular data on Fitbit app and device usage. However, this lack of usage data could attenuate our results (i.e., lack of use could drive our treatment effects closer to zero) as we are currently assuming all adopters to be using leaderboards in all periods after adoption. This makes our results more conservative.

Another potential limitation is that the student population in the sample may not be representative of the general population (e.g., they may be healthier, more physically active, or have more free time). Although broad generalization of results to the average population may be somewhat uncertain, it is useful to note that the largest leaderboard benefits accrue to the least active participants in our sample. These individuals are more comparable to the average population. More so, a younger population may be more amenable to gamification approaches and be more likely to be motivated by such interventions. Finally, we observe in our data the impact on physical activity and do not observe other downstream health outcomes (e.g., weight loss). However, given the documented relationship between physical exercise and other health outcomes and the magnitude of our effects, it is likely that individuals are, on average, healthier after leaderboard adoption.

These limitations notwithstanding, our results have significant implications for research and practice. Mitchell et al. (2013) argue for the potential of wearable technologies and mobile health more generally to be “leveraged to more promptly assess and reward behaviors on a population scale, further reducing the need for prohibitively costly incentives” (p. 666); however, there has been limited research validating this conjecture. We help substantiate this notion by demonstrating the significant potential for gamification-based interventions to meaningfully impact physical activity levels. Moreover, we find that the effects of leaderboards persist over time.

Our results also have implications for the general literature on motivating changes in health behavior. Specifically, gamification interventions coupled with health wearables provide notable advantages over other approaches studied in the literature. First, and unlike most other behavioral interventions, the widespread adoption of health wearables allows for much larger scale interventions. Second, because they are based in digital platforms, the design of these interventions can be tailored at the individual or group level to maximize benefit to the population. This customization is particularly valuable given the heterogeneity in benefit between subpopulations in our study and recognition by scholars of the limits of “one size fits all” approaches typically taken by most of the extant literature (Rogers et al. 2014, Rogers and Feller 2016).

Our research also highlights potential areas for future research. The variation in leaderboard effects points to the complex interplay between competition and social influence that underlies leaderboard effects and impacts motivation. Given that gamification approaches are highly varied, even for the same class of interventions (leaderboards can vary in terms of who can participate, who on the leaderboard is made salient to users, etc.), more work needs to be done to rigorously evaluate the potential benefits of these varied gamification approaches, the key mechanisms which drive their effects, and how they may have differential impacts across health contexts and individuals. Finally, there is a need to explore whether digital gamification interventions can be used in conjunction with classic interventions (decision aids, economic incentives, wellness programs, etc.) to unlock further health benefits.

Our results also have important implications for firms and policy makers. From the perspective of firms that sell health wearables and their related platforms, a key element of their value proposition is that their products positively impact health and well-being. Our results help substantiate the notion that physical activity can be positively impacted for large swaths of their adopters but also that some of their adopters (perhaps their earliest and most enthusiastic ones) can be harmed by some of the interventions they offer. Our results also suggests that these interventions need to be designed with careful consideration for how mechanisms intended to be motivational may not be so for all users. More so, this highlights the value of customizing gamification interventions to individuals and the value of nudging individuals toward a set of features that are likely to benefit them most. Thus, these firms may need to better incorporate insights from behavioral research in the design of gamification interventions available alongside health wearables.

Furthermore, policy makers, employers, and insurers are experimenting with health wearables and gamification as they have significant incentives to encourage more active lifestyles, which may improve health, improve worker performance, and lower healthcare costs. Scholars similarly suggest that gamification coupled with health wearables can be a cost-effective way to encourage increases in physical activity and to do so at scale (Mitchell et al. 2013). However, a salient concern with these approaches is that less healthy individuals, who are often of primary interest to policy makers and employers, can be demotivated when competing with their more active counterparts. Our results highlight that this concern may be unfounded or at least less salient, as sedentary individuals may benefit substantially from these approaches. Although the harm to highly active individuals is not ideal, some of these harms can be alleviated by tailoring leaderboards for these groups, and in net, these approaches are still likely to be valuable to employers and policy makers. A lingering challenge with reaping this value may be encouraging less healthy individuals to take up these interventions (e.g., join leaderboards), and employers and policy makers may need to invest in incentives to increase uptake for this subset of individuals to maximize potential value.

Endnotes

1 The World Health Organization (WHO) Constitution defines health as “a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity.”

2 See www.who.int/news-room/fact-sheets/detail/physical-activity.

3 See www.cdc.gov/chronicdisease/resources/publications/factsh eets/physical-activity.htm.

4 See www.lexico.com/en/definition/activity_tracker.

5 See www.wired.com/story/science-says-fitness-trackers-dont-workwear-one-anyway and www.fastcompany.com/3031324/why-yourcompany-should-think-twice-about-gamification.

6 Descriptive results substantiate this conjecture and suggest that highly active individuals are much more likely to dominate smaller leaderboards compared with larger ones, that is, the smaller leaderboards do not seem to challenge these highly active individuals as they easily dominate them even with their reduced activity levels.

7 See www.merriam-webster.com/dictionary/leaderboard.

8 Although we chose a particular form factor and a particular vendor, which is arguably the market leader at the time of the study, the similarity of leaderboards across platforms means our results may be relevant to other platforms as well.

9 Fitbit previously offered 30-day fixed-time leaderboards that, to our knowledge, have been discontinued.

10 Mechanisms of individual accountability and changes to exercise reference points are distinct from competition mechanisms. For instance, I may have a friend or family member on my leaderboard who is not credibly competing with me (e.g., because their performance exceeds my own by a huge margin) but this individual can still reach out and hold me accountable for my exercise goals or impact my perception of what is achievable for me.

11 Approximately 600 students were recruited but we are left with 516 students after excluding early dropouts and always adopters of leaderboards.

12 Due to the coarseness of the survey data relative to the Fitbit data and some nonresponse in the sample, we use these controls for robustness checks.

13 The adoption of Fitbit is not a central concern in our study as every user in our sample is a Fitbit user.

14 Another method to increase the plausibility of the common trends assumptions is to use propensity score to achieve covariate balance across the treated and untreated individuals. We perform this propensity score–based adjustment as a robustness check in a later section.

15 In Online Appendix C, we discuss GBM, IPTW, and IPTW use with DID. Furthermore, Table C.3 shows pre- and post-IPTW covariate balance, and Table C.4 and Figure C.1 explain the influence of covariates on leaderboard adoption.

16 Please see Abadie and Cattaneo (2018) for a recent discussion on such examinations of the common trend assumption and Abadie and Dermisi (2008) for an example in a two-period setting.

18 Please see apnews.com/article/2700956044de4517a471a47c3243078b.

19 Because the average number of participants on leaderboards was relatively small (2.3), a binary indicator for being first was sufficient to capture the impact of rank. Results are consistent when we use a continuous measure of the prior week’s rank.

20 We focused on active users because the Fitbit leaderboard hides inactive users (i.e., with zero steps) from the view of the focal user and does not use them in the ranking presented to individuals.

21 The labels “sedentary” and “highly active” are specific to our population and may not reflect average steps for sedentary or highly active individuals in the general population.

22 Please see Online Appendix Table H.11 for similar analysis using full data and interaction terms.

References

Abadie A, Cattaneo MD (2018) Econometric methods for program evaluation. Annu. Rev. Econom. 10:465–503.

Abadie A, Dermisi S (2008) Is terrorism eroding agglomeration economies in central business districts? J. Urban Econom. 64(2): 451–463.

Bai Y, Hibbing P, Mantis C, Welk GJ (2018) Comparative evaluation of heart rate-based monitors: Apple Watch vs Fitbit Charge HR. J. Sports Sci. 36(15):1734–1741.

Charness G, Gneezy U (2009) Incentives to exercise. Econometrica 77(3):909–931.

Deci EL, Betley G, Kahle J, Abrams L, Porac J (1981) When trying to win: Competition and intrinsic motivation. Personality Soc. Psych. Bull. 7(1):79–83.

Deterding S, Dixon D, Khaled R, Nacke L (2011) From game design elements to gamefulness. Proc. 15th Internat. Academic MindTrek Conf.: Envisioning Future Media Environ. (ACM, New York), 9–15.

Festinger L (1954) A theory of social comparison processes. Human Relations 7(2):117–140.

Finkelstein EA, Haaland BA, Bilger M, Sahasranaman A, Sloan RA, Nang EEK, Evenson KR (2016) Effectiveness of activity trackers with and without incentives to increase physical activity. Lancet Diabetes Endocrinology 4(12):983–995.

Garcia SM, Tor A (2009) The N-effect: More competitors, less competition. Psych. Sci. 20(7):871–877.

Garcia SM, Tor A, Schiff TM (2013) The psychology of competition: A social comparison perspective. Perspective Psych. Sci. 8(6):634–650.

Hamari J, Koivisto J (2013). Social motivations to use gamification. Proc. 21st Eur. Conf. on Inform. Systems, 5–8.

Hamari J, Huotari K, Tolvanen J (2014a) Gamification and economics. Walz SP, Deterding S, eds. The Gameful World (MIT Press, Cambridge, MA), 139–162.

Hamari J, Koivisto J, Sarsa H (2014b) Does gamification work? Proc. 47th Hawaii Internat. Conf. on System Sci. (IEEE, New York), 3025–3034.

Handel B, Kolstad J (2017) Wearable technologies and health behaviors: New data and new methods to understand population health. Amer. Econom. Rev. 107(5):481–485.

Ho Y-JI, Liu S, Wang L (2022) Fun shopping: A randomized field experiment on gamification. Inform. Systems Res., ePub ahead of print August 1, https://doi.org/10.1287/isre.2022.1147.

Jakicic JM, Davis KK, Rogers RJ, King WC, Marcus MD, Helsel D, Rickman AD, Wahed AS, Belle SH (2016) Effect of wearable technology combined with a lifestyle intervention on long-term weight loss. JAMA 316(11):1161–1171.

James TL, Wallace L, Deane JK (2019) Using organismic integration theory to explore the associations between users’ exercise motivations and fitness technology feature set use. Management Inform. Systems Quart. 43(1):287–312.

Johnson D, Deterding S, Kuhn K-A, Staneva A, Stoyanov S, Hides L (2016) Gamification for health and wellbeing: A systematic review of the literature. Internet Interventions 6:89–106.

Lewis ZH, Lyons EJ, Jarvis JM, Baillargeon J (2015) Using an electronic activity monitor system as an intervention modality: A systematic review. BMC Public Health 15(1):1–15.

Liu C-W, Gao G, Agarwal R (2019a) Reciprocity or Self-Interest? Leveraging Digital Social Connections for Healthy Behavior (Social Science Research Network).

Liu C-W, Gao G, Agarwal R (2019b) Unraveling the “social” in social norms: The conditioning effect of user connectivity. Inform. Systems Res. 30(4):1272–1295.

Liu D, Santhanam R, Webster J (2017) Toward meaningful engagement: A framework for design and research of gamified information systems. Management Inform. Systems Quart. 41(4):1011–1034.

Lupton D (2016) The Quantified Self (John Wiley & Sons, Hoboken, NJ).

Mitchell MS, Goodman JM, Alter DA, John LK, Oh PI, Pakosh MT, Faulkner GE (2013) Financial incentives for exercise adherence in adults: Systematic review and meta-analysis. Amer. J. Preventive Medicine 45(5):658–667.

Pamuru V, Khern-am nuai W, Kannan K (2021) The impact of an augmented-reality game on local businesses: A study of Poke´mon Go on restaurants. Inform. Systems Res. 32(3):950–966.

Patel MS, Asch DA, Volpp KG (2015) Wearable devices as facilitators, not drivers, of health behavior change. JAMA 313(5): 459–460.

Patel MS, Volpp KG, Rosin R, Bellamy SL, Small DS, Fletcher MA, Osman-Koss R (2016) A randomized trial of social comparison feedback and financial incentives to increase physical activity. Amer. J. Health Promotion 30(6):416–424.

Penedo FJ, Dahn JR (2005) Exercise and well-being: A review of mental and physical health benefits associated with physical activity. Curr. Opin. Psychiatry 18(2):189–193.

Piwek L, Ellis DA, Andrews S, Joinson A (2016) The rise of consumer health wearables: Promises and barriers. PLoS Medicine 13(2):e1001953.

Rogers T, Feller A (2016) Discouraged by peer excellence: Exposure to exemplary peer performance causes quitting. Psych. Sci. 27(3): 365–374.

Rogers T, Milkman KL, Volpp KG (2014) Commitment devices: Using initiatives to change behavior. JAMA 311(20):2065–2066.

Santhanam R, Liu D, Shen W-CM (2016) Gamification of technology-mediated training: Not all competitions are the same. Inform. Systems Res. 27(2):453–465.

Schultz PW, Nolan JM, Cialdini RB, Goldstein NJ, Griskevicius V (2007) The constructive, destructive, and reconstructive power of social norms. Psych. Sci. 18(5):429–434.

Shameli A, Althoff T, Saberi A, Leskovec J (2017) How gamification affects physical activity: Large-scale analysis of walking challenges in a mobile application. Proc. 26th Internat. Conf. on World Wide Web Companion, 455–463.

Steinhage AL, Cable D, Wardley DP (2015) Winning through cheating or creativity. Academy of Management Proc. vol. 2015.

Sullivan AN, Lachman ME (2017) Behavior change with fitness technology in sedentary adults. Frontiers Public Health 4:289.

Sun T, Gao G, Jin GZ (2019) Mobile messaging for offline group formation in prosocial activities: A large field experiment. Management Sci. 65(6):2717–2736.

Swan M (2013) The quantified self. Big Data 1(2):85–99.

Treiblmaier H, Putz L-M, Lowry PB (2018) Research commentary: Setting a definition, context, and theory-based research agenda for the gamification of non-gaming applications. AIS Trans. Human-Comput. Interactions 10(3):129–163.

Uetake K, Yang N (2019) Inspiration from the “biggest loser”: Social interactions in a weight loss program. Marketing Sci. 39(3):487–499.

Wahl Y, Du¨king P, Droszez A, Wahl P, Mester J (2017) Criterionvalidity of commercially available physical activity tracker to estimate step count, covered distance and energy expenditure during sports conditions. Frontiers Physiology 8:725.

Warburton DE, Nicol CW, Bredin SS (2006) Health benefits of physical activity: The evidence. Canadian Med. Assoc. J. 174(6):801–809.

Wu Y, Kankanhalli A, Huang KW (2015) Gamification in fitness apps: How do leaderboards influence exercise? ICIS Proc.

Xu Y (2017) Generalized synthetic control method: Causal inference with interactive fixed effects models. Political Anal. 25(1):57–76.