Rohrer, J.

English

MPG.PuRe

The Loss-of-Confidence ProjectJulia M. RohrerDepartment of PsychologyLeipzig University@dingding_pengwww.the100.ci“A new scientific truth does not generally triumph by persuading its opponents and getting them to admit their errors, but rather by its opponents gradually dying out and giving way to a new generation that is raised on it”http://www.the100.ci/2017/11/23/stupid-solutions-to-real-problems-in-science/Does it need to be that way?“methodological terrorism”Encouraging researchers tobe open about their pastresearchThe Loss-of-Confidence Project» website inviting psychologists to describe their loss of confidence» rules» author has lost confidence in primary/central finding» because of theoretical or methodological problems» for which they take the primary responsibility» all submitters become co-authors of the resulting manuscriptThe Loss-of-Confidence Project» initial public reaction: very positive» but only a handful of submissions» preprint, then invite more submissionsExample: Statement by Tal YarkoniI now think most of the conclusions drawn in this article were absurd on their face. My understanding of statistics has improved a bit since writing the article, and it is now abundantly clear to me that (a) I p-hacked to a considerable degree and that (b) because of the “winner’s curse,” statistically significant effect sizes from underpowered studies cannot be taken at face value.13 Loss-of-Confidence Statements» from a broad variety of psychological fields» neuro, social, evolutionary, experimental, personality,…» broad variety of issues» p-hacking, model misspecification, invalid inference,…How often does somethinglike this happen?Loss-of-Confidence Survey» non-representative online survey» open to researchers from all fields»N = 31604080120160Yes No Not sureHave you ever lost confidence in one of your own published research articles?Why did researchers lose confidence?» mostly questionable research practices, but again broadvariety of reasons» e.g.» “I was a junior co-author who collected and cleaned data and I felt some of the results were HARKed or hacked, but did not do very much about it”» “I think there was a mistake in the analysis script that I didn't double-check because the results were in the direction of our hypotheses”» “poor understanding of causal inference”04080120Yes No Not sureWas the loss of confidence due to a mistake or shortcoming in judgment on the part of the researcher?04080Primary responsibility Some responsibility Little or noresponsibilityTo what extent do you take personal responsibility?04080120Yes NoIs your loss of confidence a matter of public record in some way?Why not public?» not sufficiently sure, not necessary, co-authors feelings, not surehow to communicate, worry about perception…» e.g.» “I haven't had time to try and redo the analyses that I know are wrong.”» “Would hurt my career plans”» “I need published papers to get my phd. Supervisor basically pressured to apply QRPs for publication, otherwise story wouldnt be sexy enough. Honestly, i stopped to care, all in all, i'll leave science anyway.”What can we learn from the survey?» substantial number of researchers have lost confidence in oneof their findings» few of them make their loss of confidence public» host of reason keeps researchers from correcting their own claims» concerns about own reputation and career» concerns about co-authors, doctoral students» information not important, nobody would care» lack of protocol for how to deal with situation, lack of venuePutting the Self in Self-Correction» prescriptive vs. descriptive norm» fixing the formalities:» lower threshold for correction, retraction (under different label?)» make self-correction discoverable» more dynamic models of publishingPutting the Self in Self-Correction» problem: perception that self-correcting is not worth thehassle/a sign of weakness/career killer» Researchers who actually retracted one of their own papersreport that concerns about suffering reputational damageturned out to be unfounded (Hosseini et al., 2018)» Need for a general shift towards more openness aboutthe whole research processThank you for yourattention!

The Loss of Confidence Project

http://hdl.handle.net/21.11116/0000-0009-62F7-2

The Loss of Confidence Project

Abstract

Similar works

Full text

Available Versions

MPG.PuRe