Slides from a presentation,  Is Your Research Reproducible? , offered at the University of Rhode Island Libraries on December 1 and December 5, 2017.
 Learn more about the reproducibility crisis, how tools like the Open Science Framework can help you contribute to solutions, and about how proper management of your code and data helps you and other researchers. 
Part of the University Libraries\u27 Search Savvy Seminar series.
Main file is a PDF file of speaker slides as presented. Supplementary file is a PDF file of speaker slides with notes, including extra slides that were not used for presentation

Dekker, Harrison

Izenstark, Amanda

Rathemacher, Andrée

English

DigitalCommons@URI

University of Rhode IslandDigitalCommons@URITechnical Services Faculty Presentations Technical Services2017Is Your Research Reproducible?Andrée RathemacherUniversity of Rhode Island, andree@uri.eduHarrison DekkerUniversity of Rhode Island, hdekker@uri.eduSee next page for additional authorsFollow this and additional works at: http://digitalcommons.uri.edu/lib_ts_presentationsPart of the Scholarly Communication CommonsThis Article is brought to you for free and open access by the Technical Services at DigitalCommons@URI. It has been accepted for inclusion inTechnical Services Faculty Presentations by an authorized administrator of DigitalCommons@URI. For more information, please contactdigitalcommons@etal.uri.edu.Recommended CitationRathemacher, Andrée; Dekker, Harrison; and Izenstark, Amanda, "Is Your Research Reproducible?" (2017). Technical Services FacultyPresentations. Paper 56.http://digitalcommons.uri.edu/lib_ts_presentations/56http://digitalcommons.uri.edu/lib_ts_presentations/56AuthorsAndrée Rathemacher, Harrison Dekker, and Amanda IzenstarkThis article is available at DigitalCommons@URI: http://digitalcommons.uri.edu/lib_ts_presentations/56Is Your Research Reproducible?Andrée RathemacherHarrison DekkerAmanda IzenstarkUniversity of Rhode Island LibrariesSearch Savvy SeminarDecember 1 & 5, 20171. The Reproducibility Crisis2. Reproducible Workflows3. Introduction to the Open Science FrameworkThe Reproducibility Crisis“It can be proven that most claimed research findings are false.”— John P. A. Ioannidis, 2005“Reproducibility crisis”(aka “replication crisis”)“A methodological crisis in science in which scientists have found that the results of many scientific experiments are difficult or impossible to replicate on subsequent investigation, either by independent researchers or by the original researchers themselves.”— WikipediaPsychology“EEG Experiment” from Dr. Hirt’s Psychology Lab, Indiana University91.5% of all published studies in psychology found positive results.Economics“Homeless man in Vancouver” by Jay Black is licensed under CC BY-SA 2.0.“...We assert that economics research is usually not replicable.”— Andrew C. Chang and Phillip Li, 2015Animal studies“Lobund Wistar-Rat” by Janet Stephens is in the public domain.“I think it may have confounded, to whatever degree, some very large subset of existing research.”— Jeffrey Mogil, 2014Biomedical research“The NIAMS Cartilage Biology and Orthopaedics Branch” by NIH Image Gallery is licensed under CC BY-NC 2.0.Why? “File-drawer problem”“Filing” by Jeff Youngstrom is licensed under CC BY-NC 2.0.Researchers do not bother to write up experiments with negative / null results or the results of replication studies.Instead of submitting them to journals, they file them away.Why? Publication biasCover of Science v. 332, no. 6034 by the American Association for the Advancement of Science. Image by Stephen R. White. “...the small proportion of results chosen for publication are unrepresentative of scientists’ repeated samplings of the real world.”— Neal S. Young, John P. A. Iaonnidis, and Omar Al-Ubaydli, 2008Why? Bad experimental design & analysis“The Relationship Between Sample Size and Power” by Online Statistics Education: A Multimedia Course of Study is in the public domain.“If you torture the data long enough, it will confess.”— Ronald Coase, recipient of the 1991 Nobel Prize in EconomicsWhy? Incentive structure“Prof. Meyerson in his funky Stanford gown” by Anna Majkowska is licensed under CC BY 2.0.“Today I wouldn’t get an academic job. It’s as simple as that. I don’t think I would be regarded as productive enough.”— Peter Higgs, 2013 (winner of the 2013 Nobel Prize in Physics)What about peer review?“Peer Review Monster” by Gideon Burton is licensed under CC BY-SA 2.0.“We need to get away from the notion, proven wrong on a daily basis, that peer review of any kind at any journal means that a work of science is correct.”— Michael Eisen, 2014Reproducible WorkflowsA Manifesto for Reproducible Science.Marcus R. Munafò, Brian A. Nosek, Dorothy V. M. Bishop et al. Nature Human Behaviour, Vol. 1, No. 1. (10 January 2017)Workflow templateData ProcessingRaw data files are prepared for analysis. Removal of invalid data, subsetting, recoding, and so on. Ideally all steps are written in code which in turn is documented and organized.Data AcquisitionTypical sources are experimental observation and existing data sources. Acquired files must be named, organized, structured.Data AnalysisStatistical test outputs, creation of tables and figures. Also possible to create the entire documents containing formatted text and embedded code. Overarching goal of automating most if not all tasks. Adapted from Kitzes, 2018First stepsThe first step to making science reproducible is to build good habits. Your most important collaborator is your future self. It’s important to make a workflow that you can use time and time again, and even pass on to others in such a way that you don’t have to be there to walk them through it.Aaron CulichCase study from Kitzes, 2018More informationCase Studies:Kitzes, J., Turek, D., & Deniz, F. (Eds.). (2018). The Practice of Reproducible Research: Case Studies and Lessons from the Data-Intensive Sciences. Oakland, CA: University of California Press.  (A free pre-print edition is available)Documentation standard:The DRESS ProtocolTeaching materials:Project TIERIntroduction toThe Open Science FrameworkWhy the Open Science Framework? Project of the Center for Open Science, a nonprofit based in Charlottesville, VAFunded by a variety of grants and sponsors, including DARPA, the NSF, NIH, and others.https://osf.io/What it does1. Connects various partsof your workflow, wherever they are○ Google Drive○ Dropbox○ Mendeley○ FigShare○ GitHub...2. Supports versioningWhat it does1. Centralizes access to your research information2. Provides granular sharing of elements with collaborators3. Provides access for others who can provide feedback at any stage of the research processAdditional Related Project - OSF PreprintsNot just for science - includes the Arts & Humanities, Business, Education, Law, and more.* Once your article is published, please post your final manuscript in the DigitalCommons@URI for increased visibility!Closing thoughts“As readers of scientific work, all we can do is be more skeptical of everything that is published.”— Christobal Young, Assistant Professor of Sociology, Stanford University, 2015“I want to adopt a stance of humility and assume that there are errors and that’s why I need to be cautious in my conclusions.”— Brian Nosek, Professor of Psychology, University of Virginia and co-founder and director of the Center for Open Science, 2016Closing thoughtsSharing research at various stages of the process for feedback and input from others can improve your visibility, your research, and your final product.https://simplystatistics.org/2017/11/21/rr-sress/From “A few things…”2. We can remember that replication is statistical, not deterministic3. We can remember that there is a difference between exploratory and confirmatory research6. We can be persistent and private as long as possible7. We can make the realization that data is valuable but in science you don’t own itThank you!Andrée Rathemacher andree@uri.eduProfessor, Head of AcquisitionsHarrison Dekker hdekker@uri.eduAssociate Professor, Data Services Librarian Amanda Izenstark amanda@uri.eduProfessor, Reference & Instructional Design Librarian

Is Your Research Reproducible?

https://digitalcommons.uri.edu/cgi/viewcontent.cgi?article=1055&amp;context=lib_ts_presentations

Is Your Research Reproducible?

Abstract

Similar works

Full text

Available Versions

DigitalCommons@URI