Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol

In recent years there has been increased interest in replicating prior research. One of the biggest challenges to assessing replicability is the cost in resources and time that it takes to repeat studies. Thus there is an impetus to develop rapid elicitation protocols that can, in a practical manner, estimate the likelihood that research findings will successfully replicate. We employ a novel implementation of the IDEA (‘Investigate’, ‘Discuss’, ‘Estimate’ and ‘Aggregate) protocol, realised through the repliCATS platform. The repliCATS platform is designed to scalably elicit expert opinion about replicability of social and behavioural science research. The IDEA protocol provides a structured methodology for eliciting judgements and reasoning from groups. This paper describes the repliCATS platform as a multi-user cloud-based software platform featuring (1) a technical implementation of the IDEA protocol for eliciting expert opinion on research replicability, (2) capture of consent and demographic data, (3) on-line training on replication concepts, and (4) exporting of completed judgements. The platform has, to date, evaluated 3432 social and behavioural science research claims from 637 participants

Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies

Doctors down under: European medical migrants in Victoria (Australia), 1930-60

© 2019 Dr Fallon ModyThe middle of the twentieth-century saw an unprecedented mass relocation of medical practitioners – through forced migration, military service, and as economic migrants. Between 1930-60, over three thousand medical migrants – that is, overseas-trained medical graduates – are known to have arrived in Australia. Their arrival was transformative as they challenged longstanding Australian legislative structures, and came to occupy critical gaps in local medical manpower. However, medical migrants in Australia are understudied. My research begins to redress what historians have called the ‘conspicuous silence’ or ‘collective amnesia’ that characterises nation-centric medical histories, where medical migrants are largely invisible. Through a series of case studies, underpinned by a prosopographic database documenting over two hundred ‘European medical migrants’, I examine the resettlement and professional lives of two broad groups registered in the state of Victoria between 1930-60: British and Irish medical migrants (the privileged invisible) and continental European medical migrants (the marginalised ‘aliens’). Each case study can stand alone, and addresses an identified gap in the historiography. However, taken together, these case studies enable a more nuanced reflection of the differences and intersections between groups of medical migrants that historians have tacitly held as being too disparate to study collectively. Key outcomes of this research include the recovery and contextualisation of the ‘special types of labour’ medical migrants undertook; the impact of gender in the process; and the agency displayed by more marginalised groups of medical migrants

SIPS 2019 Workshop: Contemporary Philosophy of Scientific Practices for Practicing Scientists

WORKSHOP on Contemporary Philosophy of Scientific Practices for Practicing Scientists Location: Juliana in Engels Day/time: Sunday July 7, 15.45 - 17.15 Abstract: Many scientists re-examining the foundations of their practices are looking for collaborations with philosophers of science (see: SIPS Meeting of Minds workshop and hackathon) but it can be difficult to find contemporary starting points. Most attempts to integrate some philosophy of science fall back on Popper, Kuhn, Lakatos, and not much beyond. To overcome this, we are developing a set of contemporary philosophy of scientific practices resources, for scientists (not philosophers). In this workshop, we will introduce the philosophy of scientific practice and explain how it departs from the older traditions, and to facilitate a discussion on how to develop resources and collaborations between philosophers of science and scientists themselves. Session Landing Page: https://docs.google.com/document/d/15yZ9BGd3YD4KMSIi0-ulUGKqapztJh6m_NR3J0CR1P8/edit# Note: discussion and ideas raised during this workshop were integrated, along with feedback, into further development of these resources, see: https://osf.io/szq3c

Increasing confidence in pre-border risk management

International trade in aquatic animal products is governed by the World Trade Organisation’s SPS Agreement to which Australia is a signatory. The trade typically relies on certification undertaken by ‘Competent Authorities’ (CAs) to certify a product meets the importing requirements of a given country, including freedom from certain pests and diseases of concern to the importing country. For aquatic animals and aquatic animal goods, like ornamental finfish, salmon and prawns, DAWR will only accept trade from countries with a CA it has recognises and has approved. DAWR grants the more rigorous approved status to a CA once an evaluation occurs on the capacity for the agency to comply with Australia’s import requirements.The evaluation of a prospective CA by the department occurs only once, prior to establishment of the approval. There is currently no requirement for random or routine auditing of CA procedures by Australia once the initial agreement has been established. Non-compliance with certification requirements may be discovered during document assessments, or through routine physical inspections at the Australian border. In some cases, however, there is no requirement for a physical inspection of CA-certified goods – goods will be released from biosecurity control based only on an assessment of the documentation.In using CAs to manage its risk off-shore, the department is delegating certification authority to a third party. Economic theory on incentives, in particular delegation theory, suggest the scope and effectiveness of this delegation would usually be governed by how well-aligned the interests of the department and CA are. Theory also suggests the implementation of monitoring by the department is needed to ensure the actual decisions being taken by a CA fall within the rules and guidelines that have been prescribed.This project focuses on investigating the behaviour of CAs in undertaking their certification role and provides guidance on whether Australian border inspection policies should be modified in response. The analysis involves interviews with stakeholders, analysis of import inspection data and insights from economic theory. Two aquatic-animal pathways are used in the analysis, but methodology and findings are likely apply across a range of other pathways