Decomposition and Interpretation of Treatment Effects in Settings with Delayed Outcomes

This paper studies settings where the analyst is interested in identifying and estimating the average causal effect of a binary treatment on an outcome. We consider a setup in which the outcome realization does not get immediately realized after the treatment assignment, a feature that is ubiquitous in empirical settings. The period between the treatment and the realization of the outcome allows other observed actions to occur and affect the outcome. In this context, we study several regression-based estimands routinely used in empirical work to capture the average treatment effect and shed light on interpreting them in terms of ceteris paribus effects, indirect causal effects, and selection terms. We obtain three main and related takeaways. First, the three most popular estimands do not generally satisfy what we call strong sign preservation, in the sense that these estimands may be negative even when the treatment positively affects the outcome conditional on any possible combination of other actions. Second, the most popular regression that includes the other actions as controls satisfies strong sign preservation if and only if these actions are mutually exclusive binary variables. Finally, we show that a linear regression that fully stratifies the other actions leads to estimands that satisfy strong sign preservation

Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely highpowered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.UCR::Vicerrectoría de Investigación::Unidades de Investigación::Ciencias Sociales::Instituto de Investigaciones Psicológicas (IIP

A Service of zbw Approximate permutation tests and induced order statistics in the regression discontinuity design Approximate Permutation Tests and Induced Order Statistics in the Regression Discontinuity Design *

Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Abstract This paper proposes an asymptotically valid permutation test for a testable implication of the identification assumption in the regression discontinuity design (RDD). Here, by testable implication, we mean the requirement that the distribution of observed baseline covariates should not change discontinuously at the threshold of the so-called running variable. This contrasts to the common practice of testing the weaker implication of continuity of the means of the covariates at the threshold. When testing our null hypothesis using observations that are &quot;close&quot; to the threshold, the standard requirement for the finite sample validity of a permutation does not necessarily hold. We therefore propose an asymptotic framework where there is a fixed number of closest observations to the threshold with the sample size going to infinity, and propose a permutation test based on the so-called induced order statistics that controls the limiting rejection probability under the null hypothesis. In a simulation study, we find that the new test controls size remarkably well in most designs. Finally, we use our test to evaluate the validity of the design in Lee (2008), a well-known application of the RDD to study incumbency advantage. Terms of use: Documents i

EL inference for partially identified models: Large deviations optimality and bootstrap validity

This paper addresses the issue of optimal inference for parameters that are partially identified in models with moment inequalities. There currently exists a variety of inferential methods for use in this setting. However, the question of choosing optimally among contending procedures is unresolved. In this paper, I first consider a canonical large deviations criterion for optimality and show that inference based on the empirical likelihood ratio statistic is optimal. Second, I introduce a new empirical likelihood bootstrap that provides a valid resampling method for moment inequality models and overcomes the implementation challenges that arise as a result of non-pivotal limit distributions. Lastly, I analyze the finite sample properties of the proposed framework using Monte Carlo simulations. The simulation results are encouraging.Empirical likelihood Partial identification Large deviations Empirical likelihood bootstrap Asymptotic optimality

Simultaneous selection and weighting of moments in GMM using a trapezoidal kernel

This paper proposes a novel procedure to estimate linear models when the number of instruments is large. At the heart of such models is the need to balance the trade off between attaining asymptotic efficiency, which requires more instruments, and minimizing bias, which is adversely affected by the addition of instruments. Two questions are of central concern: (1) What is the optimal number of instruments to use? (2) Should the instruments receive different weights? This paper contains the following contributions toward resolving these issues. First, I propose a kernel weighted generalized method of moments (GMM) estimator that uses a trapezoidal kernel. This kernel turns out to be attractive to select and weight the number of moments. Second, I derive the higher order mean squared error of the kernel weighted GMM estimator and show that the trapezoidal kernel generates a lower asymptotic variance than regular kernels. Finally, Monte Carlo simulations show that in finite samples the kernel weighted GMM estimator performs on par with other estimators that choose optimal instruments and improves upon a GMM estimator that uses all instruments.Optimal instruments Flat-top kernels Empirical likelihood bootstrap GMM

EL Inference for Partially Identified Models: Large Deviations Optimality and Bootstrap Validity

This paper addresses the issue of optimal inference for parameters that are partially identified in models with moment inequalities. There currently exists a variety of inferential methods for use in this setting. However, the question of choosing optimally among contending procedures is unresolved. In this paper, I first consider a canonical large deviations criterion for optimality and show that inference based on the empirical likelihood ratio statistic is optimal. This finding is a direct analog to that in Kitamura (2001) for moment equality models. Second, I introduce a new empirical likelihood bootstrap that provides a valid resampling method for moment inequality models and overcomes the implementation challenges that arise as a result of non-pivotal limit distributions. Lastly, I analyze the finite sample properties of the proposed framework using Monte Carlo simulations. The simulation results are encouraging