Causal

causal

Author:
(1) Anuar Assamidanov, Department of Economics, Claremont Graduate University, 150 E 10th St, Claremont, CA 91711. (Email: anuar.assamidanov@cgu.edu). Table of Links Abstract Background Data Methodology Results Discussion and Conclusion, and References A Appendix Tables and Figures 3 Methodology The identification strategy used in this study is Difference in Differences with Dummy Outcomes (Price and Wolfers, 2010), which focuses on coach and artist gender interaction. Specifically, we estimate the following linear probability model: The regression model in the difference-in-differences framework does not capture potential heterogeneous effects across covariates. There could be a positive effect on some groups in the covariates but a negative or no effect on other groups. For example, the coaches in the initial stage of the blind audition could elicit own-gender bias, but in the later stage, opposite-gender bias. The standard regression model will provide a weighted average of effect size, even if an effect varies across a population represented by different groups. One plausible solution is to estimate heterogeneous treatment effects. Instead of just estimating one effect, I estimate a distribution of effect size for a given unit with a given set of attributes and what their impact might be. To analyze this heterogeneous effect of own-gender bias, I incorporated the first difference approach into the causal forest approach. I will discuss the performance of the proposed causal forest approach and how to apply it to my empirical example. The causal forest approach is consistent and asymptotically Gaussian and provides an estimator for their asymptotic variance that allows us to construct valid confidence intervals (Athey and Wager, 2019). Causal forest utilizes a forest-based approach to calculate a similarity weight and then applies the local generalized method of moments to estimate based on a weighted set of neighbors. The authors also show that using the centered instead of the original outcome and treatment allows a forest to focus its splits on the features that affect the treatment effect instead of the primary outcome. Intuitively, a causal forest consists of regression trees (Athey and Imbens, 2016; Breiman et al., 1984), which estimate conditional average treatment effects (CATE) at the leaves of the trees instead of predicting the outcome variables. I utilize a two-step approach by incorporating the first-difference approach into the causal forest approach, the similar two-step approach in the standard differencein-difference method. In the first step, I estimate the first difference, which is the change in the selection of individual contestants from a male coach to a female coach. This step teases out the systematic differences between coaches in female and male coach groups. In the second step, I use the outcome variables obtained from the first step and apply the causal forest approach for heterogeneous treatment-effect analysis. This step teases out the effect of contestant gender trends that exist in both women and men. My methods rely heavily on the causal forest approach, a machine learning method that can estimate heterogeneous causal effect functions under the above assumptions. The main strength associated with the causal forest is that it provides a strategy for learning patterns of heterogeneity from the data. This method requires little researcher discretion compared to traditional subgroup analyses. For instance, one does not have to decide which cutoffs to use to analyze heterogeneity in effects along continuous covariates, nor do we require to define the functional condition of the heterogeneity; the causal forest algorithm is designed to automate this search and find the most appropriate model that best characterizes the heterogeneous effect in the data (Bonander and Svensson, 2021). This paper is available on arxiv under CC 4.0 license. Author: (1) Anuar Assamidanov, Department of Economics, Claremont Graduate University, 150 E 10th St, Claremont, CA 91711. (Email: anuar.assamidanov@cgu.edu). Author: Author: (1) Anuar Assamidanov, Department of Economics, Claremont Graduate University, 150 E 10th St, Claremont, CA 91711. (Email: anuar.assamidanov@cgu.edu). Table of Links Abstract Abstract Background Background Data Data Methodology Methodology Results Results Discussion and Conclusion, and References Discussion and Conclusion, and References A Appendix Tables and Figures A Appendix Tables and Figures 3 Methodology The identification strategy used in this study is Difference in Differences with Dummy Outcomes (Price and Wolfers, 2010), which focuses on coach and artist gender interaction. Specifically, we estimate the following linear probability model: The regression model in the difference-in-differences framework does not capture potential heterogeneous effects across covariates. There could be a positive effect on some groups in the covariates but a negative or no effect on other groups. For example, the coaches in the initial stage of the blind audition could elicit own-gender bias, but in the later stage, opposite-gender bias. The standard regression model will provide a weighted average of effect size, even if an effect varies across a population represented by different groups. One plausible solution is to estimate heterogeneous treatment effects. Instead of just estimating one effect, I estimate a distribution of effect size for a given unit with a given set of attributes and what their impact might be. To analyze this heterogeneous effect of own-gender bias, I incorporated the first difference approach into the causal forest approach. I will discuss the performance of the proposed causal forest approach and how to apply it to my empirical example. The causal forest approach is consistent and asymptotically Gaussian and provides an estimator for their asymptotic variance that allows us to construct valid confidence intervals (Athey and Wager, 2019). Causal forest utilizes a forest-based approach to calculate a similarity weight and then applies the local generalized method of moments to estimate based on a weighted set of neighbors. The authors also show that using the centered instead of the original outcome and treatment allows a forest to focus its splits on the features that affect the treatment effect instead of the primary outcome. Intuitively, a causal forest consists of regression trees (Athey and Imbens, 2016; Breiman et al., 1984), which estimate conditional average treatment effects (CATE) at the leaves of the trees instead of predicting the outcome variables. I utilize a two-step approach by incorporating the first-difference approach into the causal forest approach, the similar two-step approach in the standard differencein-difference method. In the first step, I estimate the first difference, which is the change in the selection of individual contestants from a male coach to a female coach. This step teases out the systematic differences between coaches in female and male coach groups. In the second step, I use the outcome variables obtained from the first step and apply the causal forest approach for heterogeneous treatment-effect analysis. This step teases out the effect of contestant gender trends that exist in both women and men. My methods rely heavily on the causal forest approach, a machine learning method that can estimate heterogeneous causal effect functions under the above assumptions. The main strength associated with the causal forest is that it provides a strategy for learning patterns of heterogeneity from the data. This method requires little researcher discretion compared to traditional subgroup analyses. For instance, one does not have to decide which cutoffs to use to analyze heterogeneity in effects along continuous covariates, nor do we require to define the functional condition of the heterogeneity; the causal forest algorithm is designed to automate this search and find the most appropriate model that best characterizes the heterogeneous effect in the data (Bonander and Svensson, 2021). This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Using Causal Forests to Unveil Gender Bias in Hiring

Causal Analysis – Experimentation (AB Testing) and Statistical Techniques

Welcome to a two-part exploration of Causal Impact analysis. In this concise overview, we aim to equip you with theoretical foundations and practical insights. of the current article provides a quick dive into the theoretical aspects of Causal Impact analysis. Discover how this method operates, its suitability, and essential limitations. The first part delves into the practical application of Causal Impact analysis. We guide you through a specific dataset, demonstrating how to implement the library and interpret results. This hands-on approach empowers you to not only use the Causal Impact library effectively but also draw meaningful conclusions from your analyses. The second part Introduction In the current work of data analysts, evaluating the effectiveness after the introduction of a particular feature is one of the most important tasks. While randomized experiments like AB tests are the traditional gold standard for effect estimation, real-world challenges sometimes make their implementation impractical. The following common scenarios illustrate these challenges. Companies may lack the inherent infrastructure or suitability for experiments The inability to experiment due to geographical limitations. For example, you have launched a broad brand marketing company Practical constraints such as cost, resource limitations, or the need to analyze past actions In these situations, need to find alternative methods to estimate the feature’s effects and understand the impact of changes except for AB tests. This article explores the CausalImpact library, an open-source tool developed by Google. What is CausalImpact? At its heart, causal impact is about figuring out the impact of an action by guessing how a certain measure would have changed if that action (like testing a feature) hadn't happened. We predict this hypothetical change, compare it to what actually occurred, and the gap between them tells us how much the action influenced the outcome. Imagine launching a wide advertising campaign in the UK to promote a new app feature. The goal is to increase installations by reaching a larger audience through bloggers. However, placing part of this audience in a control group, where they don't see the new feature, might create a negative impression. To address this, a decision is made to roll out the feature for the entire Region B, while Region A serves as the control group without the brand campaign. Here's how CausalImpact comes into play: Control and Test Groups: Region A (Control Group): No brand campaign. Region B (Test Group): Brand campaign introduced to the audience. Targeted Metric: The metric of interest is the number of installations. Date of Intervention: The date when the campaign is rolled out to all users in Region B. With CausalImpact, a model is built based on installs from Region A. This model then predicts the expected values for the same time period in Region B, assuming no brand campaign occurred. These predicted values serve as a baseline, representing what could be expected in Region B without the advertising campaign. The important step involves comparing the actual results in Region B against these expected results. This comparison reveals the impact of the advertising campaign on installations. CausalImpact essentially allows us to quantify and understand how much the intervention influenced the outcome, providing valuable insights for decision-makers and data scientists. Limitations of Using Causal Impact Analysis Causal Impact analysis has its limitations that practitioners need to be mindful of: Data Quality Matters: Causal Impact's effectiveness relies on the quality of the data it receives. Inaccurate or incomplete data can compromise the reliability of the analysis. Correlation in Control Group: Control group must exhibit reasonable correlation with test groups. It's crucial to ensure that the chosen control group is comparable to the test group to enhance the accuracy of predictions. Macro Event Impacts: Both test and control groups should be affected by macro events in a similar manner. Any significant disparities in external factors can introduce bias into the analysis, affecting the accuracy of causal effect estimation. Post-Analysis Considerations: What happens in the control groups during the post-analysis period is vital. Interrupting regular marketing activities or making changes in the control groups during this period can lead to misleading results. It's essential to maintain consistency in control group conditions. External Influences: External factors beyond the scope of the analysis can also impact the results. Understanding and accounting for these external influences is necessary for a more accurate interpretation of causal effects. Being aware of these limitations and carefully addressing them in the analysis process is crucial to obtaining meaningful insights and making informed decisions based on Causal Impact analysis. In the , I will give a practical example of using the library and tell you how to interpret the results next article

Causal Impact Analysis as an Alternative to A/B Testing

Hey everyone! Welcome to the second part of our exploration into Causal Impact analysis. In the first part, we provided a quick dive into the theoretical aspects of Causal Impact analysis. Now, let's roll up our sleeves and delve into the practical application of Causal Impact analysis. We will guide you through a specific dataset, demonstrating how to implement the library and interpret results. This hands-on approach empowers you not only to use the Causal Impact library effectively but also to draw meaningful conclusions from your analyses. How to Use Causal Impact Imagine launching a wide advertising campaign in the UK to promote a new app feature, aiming to increase installations by reaching a larger audience through bloggers. However, placing part of this audience in a control group, where they don't see the new feature, might create a negative impression. To address this, a decision is made to roll out the feature for the entire Region B, while Region A serves as the control group without the brand campaign. Our Goal Compare the two groups, and find out whether the brand campaign has influenced the growth of installations in the region or not. Dataset Description Installs dynamic by dates in Region A: control group without a brand campaign. Installs dynamic by dates in Region B: test group with the brand campaign. Assumption: The regions used have a similar audience structure, and the control group must exhibit a reasonable correlation with test groups. Solving the Problem Step 1: Packages First, install the CausalImpact package with and execute it. pip install from causal_impact import CausalImpact Step 2: Dataset Preparation Load the dataset you want to analyze. Below is an example dataset with installs: plaintextCopy code| date       | y_installs (Test Group) | x_installs (Control Group) |
|------------|--------------------------|----------------------------|
| 2023-01-01 | ...                      | ...                        |
| 2023-01-02 | ...                      | ...                        |
| ...        | ...                      | ...                        | Step 3: Calculations and Graph date_infer = '2023-01-18'  # Date of feature rollout
df.columns = ['y', 'x']
ci = CausalImpact(df, date_infer, n_seasons=7)
result = ci.run(max_iter=1000, return_df=True)
ci.plot() Step 4: Reading Graph Results Causal Impact produces several outputs, but two are especially useful: the graphs above and a summary of impact. Plots: Observation vs Predictions: Blue line: Synthetic baseline created by the control group. Black line: Actual performance of the test group. Red dotted line: Estimated performance of the test group without intervention. Grey outline: Bounds of a 95% confidence interval. Vertical line: Represents the intervention date. Difference and Cumulative Impact: Plots the (cumulative/not) difference between the observed outcome and the predicted outcome. Statistical significance is reached whenever the shaded area goes above or below the 0 lines within the 95% confidence interval. Relative Uplift Calculation: index_infer = np.where(np.array(df.index)==date_infer)[0][0]
post_infer_result = result[result.index>=index_infer]
rel_diff = post_infer_result.pred_diff/ post_infer_result.pred
print ('Relative uplift is {}%'.format(np.round(rel_diff.mean()*100, 1))) This calculation helps represent the relative uplift, providing valuable insights into the effectiveness of the advertising campaign. Conclusion By comparing the test and control groups, we gained insights into whether the brand campaign influenced the growth of installations in the region. The visualizations, especially the Observation vs Predictions and Difference and Cumulative Impact plots offer an understanding of the intervention's effects. Armed with this hands-on experience, you are now equipped to leverage Causal Impact analysis in real-world scenarios, making informed decisions based on statistical significance and relative uplift. Cheers to unraveling insights and driving data-driven strategies!

Unveiling Causal Impact: From Theory to Practice

Authors: (1) Davide Viviano, Department of Economics, Harvard University; (2) Lihua Lei, Graduate School of Business, Stanford University; (3) Guido Imbens, Graduate School of Business and Department of Economics, Stanford University; (4) Brian Karrer, FAIR, Meta; (5) Okke Schrijvers, Meta Central Applied Science; (6) Liang Shi, Meta Central Applied Science. Table of Links Abstract & Introduction Setup (When) should you cluster? Choosing the cluster design Empirical illustration and numerical studies Recommendations for practice References A) Notation B) Endogenous peer effects C) Proofs C Proofs Throughout the proofs, expectations are conditional on the adjacency matrix A. C.1 Proof of Lemma 3.1 We have C.2 Proof of Lemma 3.2 C.3 Proof of Lemma 3.3 We consider the case where two units are in the same or different clusters separately. We will refer to µi(Di , D−i) as µi(D) for notational convenience. Following the same steps as for the case where i, j are in different clusters, accounting for Equation (27), the proof completes. C.4 Proof of Lemma 3.4 other units are not zero for individuals in the sets Bi , Gi defined in Lemma 3.2. where the first inequality is due to Cauchy-Schwarz inequality and last equality follows from Assumption 5. The proof completes after collecting the terms. C.5 Proof of Theorem 3.5 C.6 Proof of Theorem 3.6 C.7 Proof of Theorem 4.1 The bias follows directly from Lemma 3.1. We now discuss the variance component. Under Lemmas 3.2, 3.3, and following Equations (28), (29), we can write C.8 Proof of Theorem 4.2 C.9 Proof of Theorem 4.3 This paper is under CC 1.0 license. available on arxiv

Proofs and Insights into Causal Clustering Experiments under Network Interference 

Authors: (1) Davide Viviano, Department of Economics, Harvard University; (2) Lihua Lei, Graduate School of Business, Stanford University; (3) Guido Imbens, Graduate School of Business and Department of Economics, Stanford University; (4) Brian Karrer, FAIR, Meta; (5) Okke Schrijvers, Meta Central Applied Science; (6) Liang Shi, Meta Central Applied Science. Table of Links Abstract & Introduction Setup (When) should you cluster? Choosing the cluster design Empirical illustration and numerical studies Recommendations for practice References A) Notation B) Endogenous peer effects C) Proofs B Endogenous peer effects This paper is under CC 1.0 license. available on arxiv

Causal Clusterings in the Presence of Endogenous Peer Effects 

Authors: (1) Davide Viviano, Department of Economics, Harvard University; (2) Lihua Lei, Graduate School of Business, Stanford University; (3) Guido Imbens, Graduate School of Business and Department of Economics, Stanford University; (4) Brian Karrer, FAIR, Meta; (5) Okke Schrijvers, Meta Central Applied Science; (6) Liang Shi, Meta Central Applied Science. Table of Links Abstract & Introduction Setup (When) should you cluster? Choosing the cluster design Empirical illustration and numerical studies Recommendations for practice References A) Notation B) Endogenous peer effects C) Proofs References Alberto Abadie, Susan Athey, Guido W Imbens, and Jeffrey Wooldridge. When should you adjust standard errors for clustering? Technical report, National Bureau of Economic Research, 2017. Vivi Alatas, Abhijit Banerjee, Rema Hanna, Benjamin A Olken, and Julia Tobias. Targeting the poor: evidence from a field experiment in indonesia. American Economic Review, 102 (4):1206–1240, 2012. Peter M Aronow and Cyrus Samii. Estimating average causal effects under general interference, with application to a social network experiment. The Annals of Applied Statistics, 11(4):1912–1947, 2017. Susan Athey, Dean Eckles, and Guido W Imbens. Exact p-values for network interference. Journal of the American Statistical Association, 113(521):230–240, 2018. Sarah Baird, J Aislinn Bohren, Craig McIntosh, and Berk Ozler. Optimal design of experi- ¨ ments in the presence of interference. Review of Economics and Statistics, 100(5):844–860, 2018. G Basse and A Feller. Analyzing multilevel experiments in the presence of peer effects. arXiv preprint arXiv, 1608, 2016. Guillaume W Basse and Edoardo M Airoldi. Limitations of design-based causal inference and a/b testing under arbitrary and network interference. Sociological Methodology, 48(1): 136–151, 2018a. Guillaume W Basse and Edoardo M Airoldi. Model-assisted design of experiments in the presence of network-correlated outcomes. Biometrika, 105(4):849–858, 2018b. Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008. Paul S Bradley, Kristin P Bennett, and Ayhan Demiriz. Constrained k-means clustering. Microsoft Research, Redmond, 20(0):0, 2000. Yann Bramoull´e, Habiba Djebbari, and Bernard Fortin. Identification of peer effects through social networks. Journal of econometrics, 150(1):41–55, 2009. Emily Breza, Arun G Chandrasekhar, Tyler H McCormick, and Mengjie Pan. Using aggregated relational data to feasibly identify network structure without network data. American Economic Review, 110(8):2454–2484, 2020. Jing Cai, Alain De Janvry, and Elisabeth Sadoulet. Social networks and the decision to insure. American Economic Journal: Applied Economics, 7(2):81–108, 2015. Aureo De Paula, Seth Richards-Shubik, and Elie Tamer. Identifying preferences in networks ´ with bounded degree. Econometrica, 86(1):263–288, 2018. Dean Eckles, Brian Karrer, and Johan Ugander. Design and analysis of experiments in networks: Reducing bias from interference. Journal of Causal Inference, 5(1), 2017. Dennis Egger, Johannes Haushofer, Edward Miguel, Paul Niehaus, and Michael Walker. General equilibrium effects of cash transfers: experimental evidence from kenya. Econometrica, 90(6):2603–2643, 2022. Stefan Faridani and Paul Niehaus. Rate-optimal linear estimation of average global effects. arXiv preprint arXiv:2209.14181, 2022. Paul Goldsmith-Pinkham and Guido W Imbens. Social networks and the identification of peer effects. Journal of Business & Economic Statistics, 31(3):253–264, 2013. Keisuke Hirano and Jack R Porter. Asymptotics for statistical treatment rules. Econometrica, 77(5):1683–1701, 2009. David Holtz, Ruben Lobel, Inessa Liskovich, and Sinan Aral. Reducing interference bias in online marketplace pricing experiments. arXiv preprint arXiv:2004.12489, 2020. Daniel Hong, Hyunwoo Lee, and Alex Wei. Optimal solutions and ranks in the max-cut sdp. arXiv preprint arXiv:2109.02238, 2021. Michael G Hudgens and M Elizabeth Halloran. Toward causal inference with interference. Journal of the American Statistical Association, 103(482):832–842, 2008. Ravi Jagadeesan, Natesh S Pillai, and Alexander Volfovsky. Designs for estimating the treatment effect in networks with interference. 2020. Igor Kabiljo, Brian Karrer, Mayank Pundir, Sergey Pupyrev, Alon Shalita, Alessandro Presta, and Yaroslav Akhremtsev. Social hash partitioner: a scalable distributed hypergraph partitioner. arXiv preprint arXiv:1707.06665, 2017. Hyunseung Kang and Guido Imbens. Peer encouragement designs in causal inference with partial interference and identification of local average network effects. arXiv preprint arXiv:1609.04464, 2016. Brian Karrer, Liang Shi, Monica Bhole, Matt Goldman, Tyrone Palmer, Charlie Gelman, Mikael Konutgan, and Feng Sun. Network experimentation at scale. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3106– 3116, 2021. Toru Kitagawa and Guanyi Wang. Who should get vaccinated? individualized allocation of vaccines over sir network. Journal of Econometrics, 2021. Lihua Lei. Unified ℓ2→∞ eigenspace perturbation theory for symmetric random matrices. arXiv preprint arXiv:1909.04798, 2019. Lihua Lei, Xiaodong Li, and Xingmei Lou. Consistency of spectral clustering on hierarchical stochastic block models. arXiv preprint arXiv:2004.14531, 2020. Michael P Leung. Treatment and spillover effects under network interference. Review of Economics and Statistics, 102(2):368–380, 2020. Michael P Leung. Causal inference under approximate neighborhood interference. Econometrica, 90(1):267–293, 2022. Michael P Leung. Network cluster-robust inference. Econometrica, 91(2):641–667, 2023. Shuangning Li and Stefan Wager. Random graph asymptotics for treatment effect estimation under network interference. The Annals of Statistics, 50(4):2334–2358, 2022. Tianxi Li, Lihua Lei, Sharmodeep Bhattacharyya, Koen Van den Berge, Purnamrita Sarkar, Peter J Bickel, and Elizaveta Levina. Hierarchical community detection by recursive partitioning. Journal of the American Statistical Association, 117(538):951–968, 2022. Shuyang Ling and Thomas Strohmer. Certifying global optimality of graph cuts via semidefinite relaxation: A performance guarantee for spectral clustering. Foundations of Computational Mathematics, 20(3):367–421, 2020. Elena Manresa. Estimating the structure of social interactions using panel data. Unpublished Manuscript. CEMFI, Madrid, 2013. Charles F Manski. Identification of endogenous social effects: The reflection problem. The review of economic studies, 60(3):531–542, 1993. Charles F Manski. Identification of treatment response with social interactions. The Econometrics Journal, 16(1):S1–S23, 2013. Karthik Muralidharan and Paul Niehaus. Experimentation at scale. Journal of Economic Perspectives, 31(4):103–24, 2017. Mark EJ Newman. Community detection and graph partitioning. Europhysics Letters, 103 (2):28003, 2013a. Mark EJ Newman. Spectral methods for community detection and graph partitioning. Physical Review E, 88(4):042822, 2013b. Elizabeth L Ogburn, Oleg Sofrygin, Ivan Diaz, and Mark J van der Laan. Causal inference for social network data. arXiv preprint arXiv:1705.08527, 2017. Jean Pouget-Abadie. Dealing with Interference on Experimentation Platforms. PhD thesis, 2018. Fredrik S¨avje, Peter Aronow, and Michael Hudgens. Average treatment effects in the presence of unknown interference. Annals of statistics, 49(2):673, 2021. Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888–905, 2000. Betsy Sinclair, Margaret McConnell, and Donald P Green. Detecting spillover effects: Design and analysis of multilevel experiments. American Journal of Political Science, 56(4):1055– 1069, 2012. Sean J Taylor and Dean Eckles. Randomized experiments to detect and estimate social influence in networks. In Complex Spreading Phenomena in Social Systems, pages 289– 322. Springer, 2018. Johan Ugander, Brian Karrer, Lars Backstrom, and Jon Kleinberg. Graph cluster randomization: Network exposure to multiple universes. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 329–337. ACM, 2013. Davide Viviano. Experimental design under network interference. arXiv preprint arXiv:2003.08421, 2020a. Davide Viviano. Policy choice in experiments with unknown interference. arXiv preprint arXiv:2011.08174, 2020b. Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17:395–416, 2007. Jeffrey M Wooldridge. Cluster-sample methods in applied econometrics. American Economic Review, 93(2):133–138, 2003. This paper is under CC 1.0 license. available on arxiv

Causal Clustering: Design of Cluster Experiments Under Network Interference: References

Authors: (1) Davide Viviano, Department of Economics, Harvard University; (2) Lihua Lei, Graduate School of Business, Stanford University; (3) Guido Imbens, Graduate School of Business and Department of Economics, Stanford University; (4) Brian Karrer, FAIR, Meta; (5) Okke Schrijvers, Meta Central Applied Science; (6) Liang Shi, Meta Central Applied Science. Table of Links Abstract & Introduction Setup (When) should you cluster? Choosing the cluster design Empirical illustration and numerical studies Recommendations for practice References A) Notation B) Endogenous peer effects C) Proofs 6 Recommendations for practice Our algorithm depends on the choice of the network, and spillover effects. We conclude with a summary of our method and explicit recommendations for practice. The network choice must depend on researchers’ prior knowledge of which dimension spillovers propagate over. When researchers only have some prior distribution over which network matters the most, our framework directly extends to these settings, where the objective must average over the distribution of the network. In this case researchers can compute the expected bias and expected variance over multiple networks by first computing the bias and variance for each network separately as we discuss in Section 3, and then taking a weighted average with some pre-specified weights (e.g., equal weights). Choosing the adjacency matrix Given the range of values of ξn, and the choice of the network, Theorem 3.6 provides a rule of thumb to choose between a cluster or Bernoulli design. Table 1 suggests a rule of thumb ϕ¯ n √ Kn > 2.3 for binary outcomes when ψ¯ = 4. Figure 2 provides a wider and specific range of values of the spillover effects that motivate running a cluster experiment, based on the size of the neighbors assigned to different clusters, and the number of clusters. Choosing between a Cluster or Bernoulli design: an explicit rule of thumb Finally, once researchers decide to run a cluster experiment, Algorithm 1 provides an explicit algorithm to estimate the optimal clusters via semi-definite programming, choosing both the clustering and number of clusters. Choosing the optimal clusters This paper is under CC 1.0 license. available on arxiv

Navigating Network Selection and Design with Causal Clustering

Authors: (1) Davide Viviano, Department of Economics, Harvard University; (2) Lihua Lei, Graduate School of Business, Stanford University; (3) Guido Imbens, Graduate School of Business and Department of Economics, Stanford University; (4) Brian Karrer, FAIR, Meta; (5) Okke Schrijvers, Meta Central Applied Science; (6) Liang Shi, Meta Central Applied Science. Table of Links Abstract & Introduction Setup (When) should you cluster? Choosing the cluster design Empirical illustration and numerical studies Recommendations for practice References A) Notation B) Endogenous peer effects C) Proofs 2 Setup The overall effect defines the effect if all units had received the treatment compared to none of the individuals receiving the treatment. Table 1 summarizes the notation. 2.1 Spillover effects Next, we impose restrictions on the spillover effects. (First-order local interference). For i ∈ {1, · · · , n}, Assumption 1 Assumption 1 states that spillovers occur between neighbors and allows for arbitrary dependence of potential outcomes with neighbors’ assignments. One-degree neighborhood dependence follows similarly to Leung [2020], Li and Wager [2022], and it is consistent with models often used in applications, e.g., Cai et al. [2015], Sinclair et al. [2012], Muralidharan and Niehaus [2017]. Athey et al. [2018] provide a framework for testing Assumption 1. The reader may refer to Remark 2 for higher-order interference. Higher-order interference can be accomodated, although, in practice, higher order effects can be small and difficult to detect. We will refer to τn,µ as the overall effect in Equation (1) to make the dependence of τn on (µ1, · · · , µn) explicit. We do not assume that we know or can estimate consistently µi . (Because the functions µi and their classes M0,i,M1,i are indexed by i, such functions cannot be consistently estimated.) Instead, we allow for arbitrary classes M1,i,M0,i of potential outcome functions, as long as such classes satisfy the conditions below. These classes have to be both sufficiently large in order to accomodate rich structures in the data, as well as satisfy some restrictions to be able to estimate features of causal effects. (Sufficiently large function class). For all i ∈ {1, · · · , n}, d ∈ {0, 1}, Assumption 2 Assumption 2 directly holds if Md,i contains all functions, and, requires that the function class under consideration is sufficiently large otherwise. Specifically, (i) the function class Md,i contains each function and its opposite. This restriction does not impose restrictions on exposure mappings and allows for arbitrary sign flips.[4] Condition (ii) states that there exists one function in the admissible function class that is constant and positive. Condition (ii) is a regularity condition that guarantees non-degenerate solutions in worst-case scenarios studied in the following section. Condition (iii) considers a product space of potential outcome functions. These conditions are not restrictive, and they hold if M0,i,M1,i can contain arbitrary functions. Below we impose the main restrictions on the function class. (Restricted class of potential outcome models). For all i ∈ {1, · · · , n}, d ∈ {0, 1} Assumption 3 Assumption 3 imposes two restrictions. Condition (i) states that potential outcomes are uniformly bounded.[5] Condition (ii) is our main restriction on the exposure mapping. Condition (ii) is attained if M1,i,M0,i are Lipschitz function classes in the share of treated neighbors. Condition (ii) states that potential outcomes vary in the share of neighbors’ treatments by at most αiϕ¯ n. Here ϕ¯ n captures the magnitude of (largest) spillovers, and αi captures individual-level heterogeneity. The component ϕ¯ n depends on n and will play an important role in our asymptotic analysis as n → ∞. We focus on settings where ϕ¯ n is small (i.e., ϕ¯ n = o(1) as n → ∞), in the spirit of a local asymptotic framework [e.g. Hirano and Porter, 2009], but its convergence rate can be arbitrarily slow. These scenarios formalize the idea that spillover effects (and possibly but not necessarily also overall treatment effects τn) are local to zero. For example, in an information campaign, we might expect spillover (and direct) effects to be small but non-negligible for inference, as often occurs in online experiments [Karrer et al., 2021]. We show how different magnitudes of spillover effects justify different designs. Assumption 4 formalizes the local asymptotic framework considered here: spillovers converge to zero, but the rate can be arbitrary slow up to a term that depends on the squared maximum degree.[6] Assumption 4 holds for arbitrary rates of convergence of the spillover effects ϕ¯ n for networks with bounded degree [e.g. De Paula et al., 2018, where Nn,max is bounded], and requires faster rates of ϕ¯ n (smaller spillover effects), for dense networks. We conclude this discussion with examples and remarks (Local asymptotics and direct effect). Our local asymptotic framework also allows direct treatment effects (and global effects) to be local to zero. Specifically, it is possible that τn = o(1) at arbitrary rate, e.g., at the same rate as ϕ¯ n. Therefore, our local asymptotic assumption does not require that spillover effects are local to the global effect (since τn can also converge to zero). Instead, our local asymptotics formalizes settings where the noise-to-signal ratio decreases slower than n −1/2 . Remark 1 (Higher order interference and endogenous peer effects). Our setting generalizes to higher-order interference in two scenarios. Remark 2 First, suppose that friends up to degree d < ∞ generate spillovers in magnitude similar to first-degree friends. Our results extend after we define the set of friends as the set of friends up to degree d, and ϕ¯ n the largest effect that such friends generate. Sparsity restrictions on the largest degree in Assumption 4 are with respect to the number of friends up to degree d. In the second scenario, suppose that the assumption of first-order effects approximates higher-order effects up to a term of smaller order than first-order effects. Specifically, suppose that Yi(d) = µi(di , dNi )+O(hn), for some hn → 0. Our results hold if hn = o(1/n), capturing the idea that first-order effects ϕ¯ n are larger than second-order effects. In Appendix B we provide an example and sufficient conditions for the approximation error due to higher order interference being of order o(1/n) in the presence of endogenous peer effects [Bramoull´e et al., 2009, Manski, 1993], where the individual outcome depends on other units’ outcomes. In practice, higher-order effects can be small, leading to under-powered studies, especially when individuals have many friends and first order effects capture most of the spillovers. This motivates our focus on first order effects. 2.2 Experimental design and estimation Next, we turn to the class of designs and estimators considered here. Define a clustering of size Kn as a set of sets of indicators satisfying Assumption 5 states the clustering is constructed using information from the adjacency matrix only (i.e., it is independent of potential outcomes), clusters are proportional in size[7] , and individuals in a given cluster are all assigned either treatment or control with equal probability. Assumption 5 restricts the class of designs to cluster designs, motivated by our focus on overall treatment effects, and empirical practice.[8] Motivated by standard practice both in industrial applications and in field experiments with clusters [Baird et al., 2018], we consider estimators obtained by simple difference in means between treated and control clusters. Because P(Di = 1) = 1/2, we construct a (biased) estimator of treatment effects as The estimator ˆτ cl n (CC,n) is a simple difference in means between treated and control units, that normalizes by the probability of treatment. Therefore, ˆτ cl n (CC,n) depends on the clustering CC,n only because the distribution of the treatments depends on the clusters under Assumption 5. Studying the estimator in Equation (3) is a natural starting point for the analysis of cluster experiments. Variants of difference in means estimators (possibly also with regression adjustments discussed in Remark 3) are often used or studied in practice [Holtz et al., 2020, Karrer et al., 2021, S¨avje et al., 2021]. One could also normalize each sum in Equation (3) by the number of treated and control units (instead of using knowledge about the treatment probability). This would improve the stability of the estimators, but complicate the analysis of the estimators’ properties when the number of treated units is stochastic (e.g., when clusters have different size).[9] (Covariate adjustment). Our framework directly generalizes to settings that use covariate adjustment for baseline outcomes. Denote ¯µi an arbitrary predictor for µi(0) that only uses information from some arbitrary baseline observable characteristics (i.e., does not depend on the treatments or end-line outcomes in the experiment). The estimator with such an adjustment takes the form Remark 3 Our analysis continues to hold after defining the outcome of interest Yi − µ¯i . (Alternative estimators). Alternative estimators studied in the literature are inverse probability weights estimators [e.g., Aronow and Samii, 2017, Ugander et al., 2013]. Unless researchers impose additional restrictions on the exposure mapping, in the network context, these estimators can be subject to instability of the propensity score because they reweight by the inverse probability that all friends of a given individual are either under treatment or control. A different alternative are model-based estimators, which however can be subject to model misspecification. Remark 4 [4] Our results also hold if instead of imposing (i) we assume that the difference between µi(1, d) − µi(1), µi(0, d) − µ(0) can be symmetric for all d ∈ {0, 1} n. [5] This restriction is common in the literature, e.g., Kitagawa and Wang [2021], and can be relaxed by assuming random sub-gaussian potential outcomes. [6] The restriction on the maximum degree squared can be sharpened by imposing restrictions on the average second-order degree, omitted for expositional convenience. [7] This restriction is sufficient but not necessary for our analysis, and it is imposed for expositional convenience, as the rates of convergence only depend on n and Kn (population size and number of clusters) instead of also the dimension of the largest and smallest cluster. It is possible to relax such an assumption by assuming that PKn k=1 γ 2 k /Kn = O(1). [8] Saturation designs [e.g. Baird et al., 2018] would also be interesting to study if researchers are interested in estimands other than the overall treatment effect, such as the effect of treating a certain percentage of individuals in a given cluster. We leave saturation designs to future research. [9] With a stochastic number of treated units, the estimators with and without normalization are equivalent up-to an error of order O(1/ √ Kn). This paper is under CC 1.0 license. available on arxiv

Causal Clustering: Assessing Spillover Effects in Experimental Designs

Authors: (1) Davide Viviano, Department of Economics, Harvard University; (2) Lihua Lei, Graduate School of Business, Stanford University; (3) Guido Imbens, Graduate School of Business and Department of Economics, Stanford University; (4) Brian Karrer, FAIR, Meta; (5) Okke Schrijvers, Meta Central Applied Science; (6) Liang Shi, Meta Central Applied Science. Table of Links Abstract & Introduction Setup (When) should you cluster? Choosing the cluster design Empirical illustration and numerical studies Recommendations for practice References A) Notation B) Endogenous peer effects C) Proofs Abstract This paper studies the design of cluster experiments to estimate the global treatment effect in the presence of spillovers on a single network. We provide an econometric framework to choose the clustering that minimizes the worst-case mean-squared error of the estimated global treatment effect. We show that the optimal clustering can be approximated as the solution of a novel penalized min-cut optimization problem computed via off-the-shelf semi-definite programming algorithms. Our analysis also characterizes easy-to-check conditions to choose between a cluster or individual-level randomization. We illustrate the method’s properties using unique network data from the universe of Facebook’s users and existing network data from a field experiment. Experimental Design, Spillover Effects, Causal Inference, Cluster Designs. JEL Codes: C10, C14, C31, C54 Keywords: 1 Introduction Consider a (large) population of n individuals connected under a single observed network. Researchers are interested in conducting an experiment to estimate the global average treatment effect, i.e., the difference between the average effect of treating all versus none of the individuals in the population. Treating an individual may generate spillovers to her friends in the network. To capture such effects, researchers conduct a cluster experiment. Individuals are first partitioned into clusters. Within a cluster, either all units are assigned to the treatment or all units are assigned to the control group. Finally, researchers estimate treatment effects by taking a difference between the average outcomes of treated and control units (possibly adjusting for baseline covariates). The cluster design does not require modeling the dependence of individual outcomes on neighbors’ assignments, but it requires a choice of clusters and some assumptions on the extend of the spillovers along the network. For example, cluster experiments on online platforms require choosing a partition of the social network, and field experiments require choosing the unit of randomization, such as villages or regions. This raises the question of how many and which clusters to use in experiments. Typical approaches in economic research assume prior knowledge of many independent clusters. There are many settings where this information is not available, and instead units in the population have different degrees of connections.[1] This paper provides an econometric framework to choose when and how to design the clusters in cluster experiments. Different from existing clustering algorithms geared towards community detection, we motivate the choice of the clusters based on the task of estimating global treatment effects. The choice of clustering must balance two competing objectives: the larger the clusters (and the smaller the number of clusters), the smaller the bias of the estimated global effect, but the larger its variance. We introduce an algorithmic procedure – entitled Causal Clustering – to choose the clustering that minimizes a weighted combination of the worst-case bias and variance as a function of the network and clusters. The worst-case approach encodes uncertainty over the dependence of individual outcomes on neighbors’ assignments. We study (i) whether to run a cluster-level instead of individual-level randomization; (ii) how to cluster individuals (and how many clusters to use). We focus on a class of models where spillover effects are small relative to the outcomes’ variance but possibly non-negligible for inference. This is formalized in a novel framework of local asymptotics where individual outcomes depend arbitrarily on neighbors’ treatments, and, as n grows, spillovers from neighbors (and possibly also direct effects) converge to zero, but at an arbitrary slow rate (e.g., slower than n −1/2 ). This framework encodes the researchers’ uncertainty on the presence (and magnitude) of spillover effects by modeling first-order neighbors’ effect local to zero, with the convergence rate capturing the expected magnitude of spillovers.[2] The local asymptotic framework we study is consistent with settings with small (but non-negligible) treatment and spillover effects, typical, for instance, in online experiments [e.g., Karrer et al., 2021]. We characterize optimal clustering as a function of the expected magnitude of the largest spillover effects that the experiment can generate. The largest size of the spillover effects is a key input in our algorithms, whose characterization, in practice, is necessary for the design of the experiment but can be challenging. This parameter can be informed by previous experiments, in the same spirit of minimum detectable effects used in power analysis [e.g. Baird et al., 2018], or using some particular modeling assumptions. We provide guidance to practitioners in Section 6. Our analysis proceeds as follows. First, we provide a formal characterization of the worstcase bias and variance. We show that the worst-case bias is closely related to a particular notion of between-clusters connectedness, defined as the per-individual average number of friends in other clusters. The worst-case variance can potentially be an arbitrary function of within-clusters covariances and between-clusters covariances: individuals in the same cluster have identical assignments and in different clusters may share common neighbors. We show that the variance only depends on the average squared cluster size, up to an asymptotically negligible error. This result formalizes the intuition that a larger number of clusters, with a small variance in cluster size, decreases the variance of the estimator. We draw the implications of these results for choosing between a cluster experiment (for a given clustering) or assigning treatments independently between individuals (i.e., Bernoulli design or completely randomized design). Suppose the magnitude of the spillover effects is smaller than the square root of the number of clusters. In that case, the variance component dominates the bias, and a Bernoulli design is preferred (where a Bernoulli design is a special case of a cluster design with clusters containing a single unit). Vice-versa, a cluster design is preferred if the bias dominates the variance. Intuitively, because our objective trades off the bias and variance of the estimator, whenever the number of clusters is small, it is best to run a Bernoulli design for any value of spillover effects local to zero. On the other hand, if the number of clusters is sufficiently large, and the cluster design appropriately controls the bias of the estimator, a cluster design is preferred. We provide practitioners with a simple decision rule between cluster and Bernoulli designs that only depends on the number of clusters and the expected spillover effects’ magnitude. [1] For example, when using villages as clusters, individuals may interact in the same and nearby villages. See Egger et al. [2022] for an example in cash-transfer programs. This paper is under CC 1.0 license. available on arxiv

Stories, News & Mentions about CAUSAL

Causal’s Stories on HackerNoon