Mostly Harmless Simulations? Using Monte Carlo Studies for Estimator Selection

We consider two recent suggestions for how to perform an empirically motivated Monte Carlo study to help select a treatment effect estimator under unconfoundedness. We show theoretically that neither is likely to be informative except under restrictive conditions that are unlikely to be satisfied in many contexts. To test empirical relevance, we also apply the approaches to a real-world setting where estimator performance is known. Both approaches are worse than random at selecting estimators which minimise absolute bias. They are better when selecting estimators that minimise mean squared error. However, using a simple bootstrap is at least as good and often better. For now researchers would be best advised to use a range of estimators and compare estimates for robustness

When Should We (Not) Interpret Linear IV Estimands as LATE?

In this paper I revisit the interpretation of the linear instrumental variables (IV) estimand as a weighted average of conditional local average treatment effects (LATEs). I focus on a practically relevant situation in which additional covariates are required for identification while the reduced-form and first-stage regressions implicitly restrict the effects of the instrument to be homogeneous, and are thus possibly misspecified. I show that the weights on some conditional LATEs are negative and the IV estimand is no longer interpretable as a causal effect under a weaker version of monotonicity, i.e. when there are compliers but no defiers at some covariate values and defiers but no compliers elsewhere. The problem of negative weights disappears in the overidentified specification of Angrist and Imbens (1995) and in an alternative method, termed "reordered IV," that I also develop. Even if all weights are positive, the IV estimand in the just identified specification is not interpretable as the unconditional LATE parameter unless the groups with different values of the instrument are roughly equal sized. I illustrate my findings in an application to causal effects of college education using the college proximity instrument. The benchmark estimates suggest that college attendance yields earnings gains of about 60 log points, which is well outside the range of estimates in the recent literature. I demonstrate that this result is driven by the existence of defiers and the presence of negative weights. Corrected estimates indicate that attending college causes earnings to be roughly 20% higher

Mostly harmless simulations? Using Monte Carlo studies for estimator selection

We consider two recent suggestions for how to perform an empirically motivated Monte Carlo study to help select a treatment effect estimator under unconfoundedness. We show theoretically that neither is likely to be informative except under restrictive conditions that are unlikely to be satisfied in many contexts. To test empirical relevance, we also apply the approaches to a real-world setting where estimator performance is known. Both approaches are worse than random at selecting estimators which minimise absolute bias. They are better when selecting estimators that minimise mean squared error. However, using a simple bootstrap is at least as good and often better. For now researchers would be best advised to use a range of estimators and compare estimates for robustness

New Evidence on Linear Regression and Treatment Effect Heterogeneity

In this paper I provide new evidence on the implications of treatment effect heterogeneity for least squares estimation when the effects are inappropriately assumed to be homogenous. I prove that under a set of benchmark assumptions linear regression provides a consistent estimator of the population average treatment effect on the treated times the population proportion of the nontreated individuals plus the population average treatment effect on the nontreated times the population proportion of the treated individuals. Consequently, in many empirical applications the linear regression estimates might not be close to any of the standard average treatment effects of interest

New Evidence on Linear Regression and Treatment Effect Heterogeneity

It is standard practice in applied work to rely on linear least squares regression to estimate the effect of a binary variable ("treatment") on some outcome of interest. In this paper I study the interpretation of the regression estimand when treatment effects are in fact heterogeneous. I show that the coefficient on treatment is identical to the outcome of the following three-step procedure: first, calculate the linear projection of treatment on the vector of other covariates ("propensity score"); second, calculate average partial effects for both groups of interest from a regression of outcome on treatment, the propensity score, and their interaction; third, calculate a weighted average of these two effects, with weights being inversely related to the unconditional probability that a unit belongs to a given group. Each of these steps is potentially problematic, but this last property – the reliance on implicit weights which are inversely related to the proportion of each group – can have particularly devastating consequences for applied work. To illustrate the severity of this issue, I perform Monte Carlo simulations as well as replicate two prominent applied papers: Berger, Easterly, Nunn and Satyanath (2013) on the effects of successful CIA interventions during the Cold War on imports from the US; and Martinez-Bravo (2014) on the effects of appointed officials on village-level electoral results in Indonesia. In both cases some of the conclusions change dramatically after allowing for heterogeneity in effects

Doubly Robust Estimation of Local Average Treatment Effects Using Inverse Probability Weighted Regression Adjustment

We revisit the problem of estimating the local average treatment effect (LATE) and the local average treatment effect on the treated (LATT) when control variables are available, either to render the instrumental variable (IV) suitably exogenous or to improve precision. Unlike previous approaches, our doubly robust (DR) estimation procedures use quasi-likelihood methods weighted by the inverse of the IV propensity score - so-called inverse probability weighted regression adjustment (IPWRA) estimators. By properly choosing models for the propensity score and outcome models, fitted values are ensured to be in the logical range determined by the response variable, producing DR estimators of LATE and LATT with appealing small sample properties. Inference is relatively straightforward both analytically and using the nonparametric bootstrap. Our DR LATE and DR LATT estimators work well in simulations. We also propose a DR version of the Hausman test that compares different estimates of the average treatment effect on the treated (ATT) under one-sided noncompliance

New Evidence on Linear Regression and Treatment Effect Heterogeneity

It is standard practice in applied work to rely on linear least squares regression to estimate the effect of a binary variable ("treatment") on some outcome of interest. In this paper I study the interpretation of the regression estimand when treatment effects are in fact heterogeneous. I show that the coefficient on treatment is identical to the outcome of the following three-step procedure: first, calculate the linear projection of treatment on the vector of other covariates ("propensity score"); second, calculate average partial effects for both groups of interest from a regression of outcome on treatment, the propensity score, and their interaction; third, calculate a weighted average of these two effects, with weights being inversely related to the unconditional probability that a unit belongs to a given group. Each of these steps is potentially problematic, but this last property – the reliance on implicit weights which are inversely related to the proportion of each group – can have particularly devastating consequences for applied work. To illustrate the severity of this issue, I perform Monte Carlo simulations as well as replicate two prominent applied papers: Berger, Easterly, Nunn and Satyanath (2013) on the effects of successful CIA interventions during the Cold War on imports from the US; and Martinez-Bravo (2014) on the effects of appointed officials on village-level electoral results in Indonesia. In both cases some of the conclusions change dramatically after allowing for heterogeneity in effects

Joachima Lelewela wizja dziejowej roli Słowian

Opposition to the idea of the historical process of enlightenment and the romantic vision of the development of specific national communities is a simplification. Apology multiplicity of values implemented by the nations in the historical process is present in the phraseology of the era. Attempts reference to its historical reality showed that only one nation (this, of course, to which the author of expression) carries universal values. Other values should follow the lead of this guide humanity. So too is the vision LelewelaPrzeciwstawienie uniformizujących proces dziejowy idei oświecenia oraz romantycznej wizji specyficznego rozwoju społeczności narodowych, multilinearnej koncepcji historii stanowi uproszczenie. Apologia wielości wartości realizowanych przez narody w procesie dziejowym, faktycznie powszechnie obecna we frazeologii epoki, nie przekładała się tam przeważnie na historyczny konkret. Próby odniesienia jej do rzeczywistości dziejowej wskazywały, że tylko jeden naród (ten oczywiście, do którego należy autor wypowiedzi) realizuje wartości uniwersalne, zaś pozostał inny iść śladem tego przewodnika ludzkości. Taka też jest wizja Lelewela, choć jego poglądy na ogół wpisywano w przywołany schemat, uznając słynną koncepcję gminowładczą za przykład konkretyzacji idei niepowtarzalnego ducha narod