How Much Can We Generalize from Impact Evaluations?

Impact evaluations aim to predict the future, but they are rooted in particular contexts and to what extent they generalize is an open and important question. The author exploit a new data set of results on a wide variety of interventions and find more heterogeneity than in other literatures. This has implications for how evidence is generated and used to inform policy. This paper uses a database of impact evaluation results collected by AidGrade, a U.S. non-profit research institute founded by the author in 2012. AidGrade focuses on gathering the results of impact evaluations and analyzing the data, including through meta-analysis. Its data on impact evaluation results were collected in the course of its meta-analyses from 2012-2014. The research also found evidence of systematic variation in effect sizes that is surprisingly robust across different interventions and outcomes. Smaller studies tended to have larger effect sizes, which we might expect if the smaller studies are better-targeted, are selected to be evaluated when there is a higher a priori expectation they will have a large effect size, or if there is a preference to report larger effect sizes, which smaller studies would obtain more often by chance. Government-implemented programs also had smaller effect sizes than academic/NGO-implemented programs, even after controlling for sample size. This is unfortunate given we often do smaller impact evaluations with NGOs in the hopes of finding a strong positive effect that can scale through government implementation and points to the importance of research on scaling up interventions

Forecasting the Results of Experiments: Piloting an Elicitation Strategy

Forecasts of experimental results can clarify the interpretation of research results, mitigate publication bias, and improve experimental designs. We collect forecasts of the results of three Registered Reports preliminarily accepted to the Journal of Development Economics, randomly varying four features: (1) small versus large reference values; (2) whether predictions are in raw units or standard deviations; (3) text-entry versus slider responses; and (4) small versus large slider bounds. Forecasts are generally robust to elicitation features, though wider slider bounds are associated with higher forecasts throughout the forecast distribution. We make preliminary recommendations on how many forecasts should be gathered

Essays in Development Economics and Trade

Economic development has the potential to improve lives. Three issues that directly affect economic development are conflict, trade, and innovation, the subjects of the three chapters in this dissertation.Conflict causes enormous suffering, but the study of peacekeeping is plagued by endogeneity issues. The first chapter in this dissertation uses an instrumental variables approach to estimate the effectiveness of U.N. peacekeepers at ending episodes of conflict, maintaining the peace once peace has been obtained, and preventing another episode from ever re-occurring. I find that the likelihood of being sent U.N. peacekeepers varies with temporary membership in the U.N. Security Council and exploit this variation in my estimation. This variation also suggests that the leaders of countries in conflict often do not want their country to receive peacekeepers. The results indicate that even though peacekeepers are often unwanted, they help to maintain the peace after an episode of conflict has ended and reduce the likelihood that the conflict resumes.After peace, trade is also considered crucial to development. In the standard trade literature, more productive firms should export over less productive firms, all else equal. This premise appears in Melitz (2003), Luttmer (2007), and Eaton et al. (2009), among others. However, we know that developing countries often suffer from market distortions (Hsieh and Klenow, 2009). The theory behind the second chapter of this dissertation combines Hsieh and Klenow-like distortions with a Melitz-like model that accords productivity a key role. Under this model, firm productivity matters less in the decision to export in sectors with greater distortions and firms facing greater distortions exhibit less of the productivity-based "churning" and re-allocation that Melitz predicts. The implication is that trade is less beneficial to productivity in developing countries. These predictions are tested using plant-level data from Colombia.Finally, new products have been shown to increase welfare in a few studies. One branch of the literature has focused on estimating the welfare effects of very narrow and specific new products; another has estimated elasticities of substitution across a large number of varieties and then imputed gains from the new varieties that appear in the data. However, one might suspect that the most important innovations occurred over a much longer period of time than has been studied to date. Thus, the final chapter of my dissertation focuses on a different question. It assigns an innovation date to each good and asks the question: how would welfare be affected if one were restricted to the set of goods available at an earlier time period? I modify the methodology in Feenstra (1994) and Broda and Weinstein (2006a) to answer this question. The estimates suggest that innovations are more important to welfare than previously thought. I also find that the price index that takes varieties into consideration favoured by the literature can yield deeply misleading results

How Often Should We Believe Positive Results? Assessing the Credibility of Research Findings in Development Economics

Under-powered studies combined with low prior beliefs about intervention effects increase the chances that a positive result is overstated. We collect prior beliefs about intervention impacts from 125 experts to estimate the false positive and false negative report probabilities (FPRP and FNRP) as well as Type S (sign) and Type M (magnitude) errors for studies in development economics. We find that the large majority of studies in our sample are generally credible. We discuss how more systematic collection and use of prior expectations could help improve the literature

The Impact of New Products on Ethical Beliefs

We test whether the availability of consumer goods affects ethical beliefs. Several new firms are developing "clean" animal products: lab-grown meat, egg, and dairy products that do not rely on traditional animal agriculture. Standard models of cognitive dissonance would predict that the mere availability of such a product would lead consumers to put more moral weight on the environment and farm animals. We do not initially observe this and in fact find that information about clean meat may even negatively affect beliefs. A second experiment in which we use priming to randomly manipulate how positively respondents view the product explains the surprising result: due to concerns about the "unnaturalness" of the product, many do not find it an acceptable substitute, however, those who perceive the product positively do change their ethical beliefs

Effective strategies for overcoming the naturalistic heuristic: Experimental evidence on consumer acceptance of “clean” meat

The naturalistic heuristic of “what is natural is good” poses a serious barrier to consumer adoption of genetically modified foods, childhood vaccinations, and related technologies. While existing evidence suggests that pro-acceptance messaging appeals based on debunking this heuristic are ineffective at increasing consumer acceptance, there is little evidence on whether this ineffectiveness extends to new products for which consumers have not yet formed crystallized opinions. In this study, we examine three messaging strategies — direct debunking, embracing unnaturalness, and descriptive norms — for overcoming consumer resistance in the context of a new food technology: “clean meat” — also known as “cultured” or “in vitro” meat. We compare the effects of these three pro-clean meat appeals against “anti-clean meat social information” from anonymous consumers. We find persistent negative effects of anti-clean meat social information over 10 weeks. In contrast, improvements in consumer acceptance following the direct debunking and descriptive norms appeals were short-lived. The only appeal to successfully offset the undermining effects of anti-clean meat social information over 10 weeks was the embrace unnaturalness appeal, suggesting that advocates wishing to enhance consumer acceptance of new food technologies should focus on how these technologies are similar to products that also seem “unnatural” but which are already widely adopted by consumers

Forecasting the Results of Experiments: Piloting an Elicitation Strategy

Forecasts of experimental results can clarify the interpretation of research results, mitigate publication bias, and improve experimental designs. We collect forecasts of the results of three Registered Reports preliminarily accepted to the Journal of Development Economics, randomly varying four features: (1) small versus large reference values; (2) whether predictions are in raw units or standard deviations; (3) text-entry versus slider responses; and (4) small versus large slider bounds. Forecasts are generally robust to elicitation features, though wider slider bounds are associated with higher forecasts throughout the forecast distribution. We make preliminary recommendations on how many forecasts should be gathered