In this paper we reply to Mitesh Kataria’s comment, which criticized the simulations of Maniadis, Tufano, and List (2014, Am. Econ. Rev.104(1), 277-290). We view these simulations as a means to illustrating the fact that we economists are unaware of the value of key variables that determine the credibility of our own empirical findings.  Such variables include priors (i.e., the pre-study probability that a tested phenomenon is true) and the statistical power of the empirical design.  Economists should not hesitate to use Bayesian tools and meta-analysis in order to quantify what we know about these variables

List, John A.

Maniadis, Zacharias

Tufano, Fabio

Nottingham eTheses

Maniadis, Zacharias and Tufano, Fabio and List, John A. (2014) One swallow doesn’t make a summer: reply to Kataria. Econ Journal Watch, 11 (1). pp. 11-16. ISSN 1933-527X Access from the University of Nottingham repository: http://eprints.nottingham.ac.uk/32950/8/ManiadisTufanoListJan2014%20%281%29.pdfCopyright and reuse: The Nottingham ePrints service makes this work by researchers of the University of Nottingham available open access under the following conditions.This article is made available under the Creative Commons Attribution licence and may be reused according to the conditions of the licence.  For more details see: http://creativecommons.org/licenses/by/2.5/A note on versions: The version presented here may differ from the published version or from the version of record. If you wish to cite this item you are advised to consult the publisher’s version. Please see the repository url above for details on accessing the published version and note that access may require a subscription.For more information, please contact eprints@nottingham.ac.ukOne Swallow DoesnÕt Make aSummer: Reply to KatariaZacharias Maniadis1, Fabio Tufano2, and John A. List3LINK TO ABSTRACTIn his comment, Mitesh Kataria (2014) makes three main points about aspecific part of our paper (Maniadis, Tufano, and List 2014), namely about Tables2 and 3. In our paper, we employ these tables in order to illustrate the idea thatvery inconclusive post-study probabilities that a tested phenomenon is true mayresult from novel, surprising findings. The main arguments in Kataria (2014) arethe following:First, if P(H0) is unknown, as is often the case with economicapplications, the post-study probability can lead to even worseinference than the Classical significance test, depending on the qualityof the prior. Second, the simulation in Maniadis et al. (2014) ignoresprevious assessments of P(H0) and instead utilizes a selective empiricalsetup that favors the use of post-study probabilities. … [Third,]contrary to what Maniadis et al. (2014) argue, their results do not allowfor drawing general recommendations about which approach is themost appropriate. (Kataria 2014, abs.)We believe that our work might have been misunderstood by Kataria.Moreover, it seems that some of his claims are not supported by relevant empiricalevidence.Discuss this article at Journaltalk:http://journaltalk.net/articles/5816ECON JOURNAL WATCH 11(1)January 2014: 11-161. University of Southampton, Southampton SO17 1BJ, UK.2. University of Nottingham, Nottingham NG7 2RD, UK.3. University of Chicago, Chicago, IL 60637.VOLUME 11, NUMBER 1, JANUARY 2014 11In Maniadis, Tufano, and List (2014), our basic aim is to draw on the generalproblem of the credibility crisis in disciplines other than economics (Ioannidis2005; Bettis 2012; Jennions and Moller 2002), and to convey the disquieting newsto economists by relying on insights and tools from the life sciences literature.While conveying the troubling news, we also emphasize the good news that usuallyit takes only a few independent replications to advance considerably the credibilityof empirical exercises. We wish to understand how confident one should be inthe published empirical findings in economics. Simply put, we are not discardingclassical significance testing, just arguing that we should be interpreting itaccurately. For an educated assessment of the empirical evidence we need to knownot just whether tests were significant but also the value of key variables such asresearch priors and statistical power. Admittedly, these variables are not easy toestimate, and in economics it is often, even typically, the case that there is notmuch relevant evidence. But this is exactly our point: We wished to show that if wewish to assess how confident we are in our findings, evidence is lacking in criticaldimensions. Given the recent evidence pointing to non-replicability in several lifesciences (Ioannidis 2012), such lack of evidence may cause serious questions to beraised about economics as well (see Ioannidis and Doucouliagos 2013; Alexander2013).Whereas Kataria claims that “for economic hypotheses, the unconditionalprobability P(H0) is hardly ever known” (Kataria 2014, 8), we suggest that theissue of such knowledge accumulation needs to be regarded as endogenous. If theinvestigator’s frame of analysis disregards the variable P(H0), there is no need toestimate it. Other disciplines have developed meta-analytic methods that can befruitfully employed in economics for estimating the relevant variables (Cooper,Hedges, and Valentine 2009). Replication has a key role in these methods.To encourage such a structured approach, we illustrated with Tables 2 and3, using Bayesian language, the fact that we should be cautious of new evidenceand—as we argue later in Maniadis, Tufano, and List (2014)—that we should alsoincrease our efforts to replicate original studies. We clearly note in the paper thatthe combinations of parameter values used in Tables 2 and 3 should be thought ofas applying to novel and surprising findings (Maniadis, Tufano, and List 2014, 278,286 n. 27). So these combinations were truly selected to illustrate what happens inthe case of such findings. Moreover, we acknowledged the difficulty of pinpointingthose combinations exactly (ibid., 286). Essentially, the degree to which ourdiscipline is characterized by such combinations of priors and power is an empiricalquestion. We hope that the message of the tables itself will encourage work on thisunderexplored question. Once more, we view as one of our key messages that welack sufficient evidence to evaluate the credibility of much work in our field. Wejoin others in prompting economists to grapple with such questions as: What isMANIADIS, TUFANO, AND LIST12 VOLUME 11, NUMBER 1, JANUARY 2014a reasonable estimate for the typical prior in each subfield of economic research?What is the typical power of a research study? How common is replication ineconomics and how common should it be?Given the scarcity of relevant empirical studies, we find the particularconfigurations suggested by Kataria (2014) somewhat unsupported by theevidence. In particular, there seems to be no empirical foundation for the claimsthat “effect sizes follow a standard normal distribution centered at zeroand…scientists only detect and consider effect sizes |d| > 0.2 as relevant” (Kataria2014, 6). Despite this, Kataria claims that “the neighborhood of P(H0) ≈ 0.16 … isappreciated to be a more realistic estimate” (ibid., 7). Estimating P(H0) is a difficultempirical question that would require much more research. With respect to power,Kataria mentions evidence from the related field of psychology, namely JosephRossi (1990), who estimated that the average power for medium effect sizes is equalto 0.57. However, it is not clear on which evidence the assumption of mediumeffect sizes is based. Furthermore, more recent evidence reveals that typical powerin psychology is about 0.35, even if we assume that the average effect size |d| isequal to 0.5 (Bakker, van Dijk, and Wicherts 2012).The spirit of our paper is to encourage work such as the very recent paper byLe Zhang and Andreas Ortman (2013). They retrospectively estimated the powerof several experimental designs reported in Christoph Engel’s meta-analysis ofdictator games (Engel 2011), and they found that the median level of power wasless than 0.25. It is important to note the critical role of meta-analysis for generatingthis piece of new evidence. The point is not to argue in the absence of evidence butto try to accumulate the necessary evidence. As economists, we hope that our fieldis very credible, but we need to provide empirical evidence using the relevant tools.At this point we need to acknowledge the important issue of “previousassessments of P(H0),” although Kataria mentioned it without justification. As wesaid in Maniadis, Tufano, and List (2014), we aimed to make a claim about novel,surprising results. We do believe that many types of economic research are moregrounded in theory than research in other social sciences, so for them “surprising”results may not be as important for publication. In fact, Brad DeLong and KevinLang (1992) found that P(H0) is very close to zero for a set of hypotheses publishedin top economic journals in the 1980s. If their interpretation—that the refereeprocess somehow manages to filter true associations—is correct, that would bereassuring for the credibility of the economics profession. As DeLong and Lang(1992) acknowledge, however, there are alternative interpretations for theirfindings, such as the existence of selection issues and data mining in the discipline,so their optimistic interpretation should be taken with caution. There is a need forfurther research on the matter, following the seminal analysis of DeLong and LangREPLY TO KATARIAVOLUME 11, NUMBER 1, JANUARY 2014 13(1992). We are particularly interested in the field of experimental economics, wherewe worry that “surprising” findings might be more frequently published.4From the previous arguments it should be clear that in Maniadis, Tufano,and List (2014) we did not put forward any general recommendation about whichinference approach, Classical or Bayesian, is the most appropriate. In fact, in thecontext of the current “publish or perish” culture (see, e.g., Fanelli 2010) andthe related structure and incentives of the economics knowledge system (Oswald2007; Glaeser 2008; Young, Ioannidis, and Al-Ubaydli 2008), we merely resortto Bayesian language to argue in favor of a much more careful interpretation ofClassical inference.Summing up, we believe that studying systematically the factors that affectthe credibility of empirical findings might have an important role to play ineconomics. Meta-analysis and Bayesian tools are of central importance forconceptualizing the problem and quantifying key variables, and should not beignored by economists. Our point was not to argue in favor of a specificconfiguration of parameter values, but to show that we cannot ignore factors suchas priors and power, because if we do, something can go very wrong with economicresearch.ReferencesAlexander, Ruth. 2013. Reinhart, Rogoff… and Herndon: The Student WhoCaught Out the Profs. BBCNews, April 20. LinkBakker, Marjan, Annette van Dijk, and Jelte M. Wicherts. 2012. The Rules ofthe Game Called Psychological Science. Perspectives on Psychological Science 7(6):543-554.Bettis, Richard A. 2012. The Search for Asterisks: Compromised Statistical Testsand Flawed Theories. Strategic Management Journal 33(1): 108-113.Cooper, Harris, Larry V. Hedges, and Jeffrey C. Valentine, eds. 2009. TheHandbook of Research Synthesis and Meta-Analysis, 2nd ed. New York: RussellSage Foundation.DeLong, J. Bradford, and Kevin Lang. 1992. Are All Economic HypothesesFalse? Journal of Political Economy 100(6): 1257-1272.Engel, Christoph. 2011. Dictator Games: A Meta Study. Experimental Economics14(4): 583-610.Fanelli, Daniele. 2010. Do Pressures to Publish Increase Scientists’ Bias? AnEmpirical Support from US States Data. PLoS ONE 5(4): e10271. Link4. We would welcome more empirical evidence on this and related issues.MANIADIS, TUFANO, AND LIST14 VOLUME 11, NUMBER 1, JANUARY 2014Glaeser, Edward L. 2008. Researcher Incentives and Empirical Methods. In TheFoundations of Positive and Normative Economics: A Handbook, eds. AndrewCaplin and Andrew Schotter, 300-319. New York: Oxford University Press.Ioannidis, John P. A. 2005. Why Most Published Research Findings Are False.PLoSMedicine 2(8): e124. LinkIoannidis, John P. A. 2012. Why Science Is Not Necessarily Self-Correcting.Perspectives on Psychological Science 7(6): 645-654.Ioannidis, John P. A., and Chris Doucouliagos. 2013. What’s to Know Aboutthe Credibility of Empirical Economics? Journal of Economic Surveys 27(5):997-1004.Jennions, Michael D., and Anders P. M¿ller. 2002. Relationships Fade withTime: A Meta-Analysis of Temporal Trends in Publication in Ecology andEvolution. Proceedings of the Royal Society B 269: 43-48.Kataria, Mitesh. 2014. One Swallow Doesn’t Make a Summer: A Comment onZacharias Maniadis, Fabio Tufano, and John List. Econ Journal Watch 11(1):4-10. LinkManiadis, Zacharias, Fabio Tufano, and John A. List. 2014. One SwallowDoesn’t Make a Summer: New Evidence on Anchoring Effects. AmericanEconomic Review 104(1): 277-290.Oswald, Andrew J. 2007. An Examination of the Reliability of PrestigiousScholarly Journals: Evidence and Implications for Decision-Makers.Economica 74(293): 21-31.Rossi, Joseph S. 1990. Statistical Power of Psychological Research: What HaveWe Gained in 20 Years? Journal of Consulting and Clinical Psychology 58(5):646-656.Young, Neal S., John P. A. Ioannidis, and Omar Al-Ubaydli. 2008. WhyCurrent Publication Practices May Distort Science. PLoS Medicine 5(10):e201. LinkZhang, Le, and Andreas Ortmann. 2013. Exploring the Meaning of Significancein Experimental Economics. University of New South Wales Australian School ofBusiness Research Paper 2013-32. LinkREPLY TO KATARIAVOLUME 11, NUMBER 1, JANUARY 2014 15Zacharias Maniadis is Assistant Professor of Economics atthe Economics Division, School of Social Sciences, Universityof Southampton. His research interests include micro-economics, experimental economics, behavioral economics,and political economy. His email address is Z.Maniadis@soton.ac.uk.Fabio Tufano is Assistant Professor of Economics at the Uni-versity of Nottingham. His research interests include psych-ology and economics, experimental economics, and gametheory. His email address is fabio.tufano@nottingham.ac.uk.John A. List is Homer J. Livingston Professor in Economicsand the College at the University of Chicago. His researchinterests include experimental economics, field experiments,education, youth violence, economics of charity, environmen-tal economics, experiments in firms, multi-unit auctions, andneuro-economics. His e-mail address is jlist@uchicago.edu.About the AuthorsGo to archive of Comments sectionGo to January 2014 issueDiscuss this article at Journaltalk:http://journaltalk.net/articles/5816MANIADIS, TUFANO, AND LIST16 VOLUME 11, NUMBER 1, JANUARY 2014

One swallow doesn’t make a summer: reply to Kataria

http://eprints.nottingham.ac.uk/32950/8/ManiadisTufanoListJan2014%20%281%29.pdf

One swallow doesn’t make a summer: reply to Kataria

Abstract

Similar works

Full text

Available Versions

Nottingham eTheses