Ashton, Raquel

Burgman, Mark

Bush, Martin

Fidler, Fiona

Fraser, Hannah

Gould, Elliot

Gray, Charles

Groenewegen, Rebecca

Hamilton, Dan

Hanea, Anca

Head, Andrew

Hemming, Victoria

Hoekstra, Rink

Mandel, David

Marcoci, Alexandru

McBride, Marissa

Mody, Fallon

Nakagawa, Shinichi

Parker, Timothy

Pearson, Ross

Ross, Melissa

Rumpff, Libby

Sinnott, Richard

Smith, Eden

Thorn, Felix Singleton

van Ravenzwaaij, Don

Vercammen, Ans

Vesk, Peter

Wilkinson, David Peter

Willcox, Aaron

Wintle, Bonnie

English

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies

ARTS repository - University of Groningen

Predicting Reliability Through Structured Expert Elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)

https://pure.rug.nl/ws/files/553355203/Predicting_reliability_through_structured_expert_elicitation_with_repliCATS_PREPRINT.pdf

Predicting Reliability Through Structured Expert Elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)

Abstract

Similar works

Full text

Available Versions

ARTS repository - University of Groningen