20,853 research outputs found
False Discovery Rate Controlled Heterogeneous Treatment Effect Detection for Online Controlled Experiments
Online controlled experiments (a.k.a. A/B testing) have been used as the
mantra for data-driven decision making on feature changing and product shipping
in many Internet companies. However, it is still a great challenge to
systematically measure how every code or feature change impacts millions of
users with great heterogeneity (e.g. countries, ages, devices). The most
commonly used A/B testing framework in many companies is based on Average
Treatment Effect (ATE), which cannot detect the heterogeneity of treatment
effect on users with different characteristics. In this paper, we propose
statistical methods that can systematically and accurately identify
Heterogeneous Treatment Effect (HTE) of any user cohort of interest (e.g.
mobile device type, country), and determine which factors (e.g. age, gender) of
users contribute to the heterogeneity of the treatment effect in an A/B test.
By applying these methods on both simulation data and real-world
experimentation data, we show how they work robustly with controlled low False
Discover Rate (FDR), and at the same time, provides us with useful insights
about the heterogeneity of identified user groups. We have deployed a toolkit
based on these methods, and have used it to measure the Heterogeneous Treatment
Effect of many A/B tests at Snap
Searching for the Causal Structure of a Vector Autoregression
Vector autoregressions (VARs) are economically interpretable only when identified by being transformed into a structural form (the SVAR) in which the contemporaneous variables stand in a well-defined causal order. These identifying transformations are not unique. It is widely believed that practitioners must choose among them using a priori theory or other criteria not rooted in the data under analysis. We show how to apply graph-theoretic methods of searching for causal structure based on relations of conditional independence to select among the possible causal orders β or at least to reduce the admissible causal orders to a narrow equivalence class. The graph-theoretic approaches were developed by computer scientists and philosophers (Pearl, Glymour, Spirtes among others) and applied to cross-sectional data. We provide an accessible introduction to this work. Then building on the work of Swanson and Granger (1997), we show how to apply it to searching for the causal order of an SVAR. We present simulation results to show how the efficacy of the search method algorithm varies with signal strength for realistic sample lengths. Our findings suggest that graph-theoretic methods may prove to be a useful tool in the analysis of SVARs.search, causality, structural vector autoregression, graph theory, common cause, causal Markov condition, Wold causal order, identification; PC algorithm
- β¦