Effects of interacting with a large language model compared with a human coach on the clinical diagnostic process and outcomes among fourth-year medical students: study protocol for a prospective, randomised experiment using patient vignettes.

INTRODUCTION Versatile large language models (LLMs) have the potential to augment diagnostic decision-making by assisting diagnosticians, thanks to their ability to engage in open-ended, natural conversations and their comprehensive knowledge access. Yet the novelty of LLMs in diagnostic decision-making introduces uncertainties regarding their impact. Clinicians unfamiliar with the use of LLMs in their professional context may rely on general attitudes towards LLMs more broadly, potentially hindering thoughtful use and critical evaluation of their input, leading to either over-reliance and lack of critical thinking or an unwillingness to use LLMs as diagnostic aids. To address these concerns, this study examines the influence on the diagnostic process and outcomes of interacting with an LLM compared with a human coach, and of prior training vs no training for interacting with either of these 'coaches'. Our findings aim to illuminate the potential benefits and risks of employing artificial intelligence (AI) in diagnostic decision-making. METHODS AND ANALYSIS We are conducting a prospective, randomised experiment with N=158 fourth-year medical students from Charité Medical School, Berlin, Germany. Participants are asked to diagnose patient vignettes after being assigned to either a human coach or ChatGPT and after either training or no training (both between-subject factors). We are specifically collecting data on the effects of using either of these 'coaches' and of additional training on information search, number of hypotheses entertained, diagnostic accuracy and confidence. Statistical methods will include linear mixed effects models. Exploratory analyses of the interaction patterns and attitudes towards AI will also generate more generalisable knowledge about the role of AI in medicine. ETHICS AND DISSEMINATION The Bern Cantonal Ethics Committee considered the study exempt from full ethical review (BASEC No: Req-2023-01396). All methods will be conducted in accordance with relevant guidelines and regulations. Participation is voluntary and informed consent will be obtained. Results will be published in peer-reviewed scientific medical journals. Authorship will be determined according to the International Committee of Medical Journal Editors guidelines

Material for "Collaboration during the diagnostic decision-making process: When does it help?"

When making complex decisions, such as a medical diagnosis, decision makers typically gather, analyse, and synthesize information. In a previous study, we showed that delegating such complex decisions to collaborating pairs increases decision quality substantially as compared to individuals, without resulting in different information search. Given the higher costs associated with teamwork, however, it is of great practical interest to understand when in the process performance benefits of teams may arise so to delegate single phases to teams where possible. We thus conducted an experimental study in which 4th-year medical students (n = 109) worked either in pairs or alone on two separate tasks: (1) analyzing diagnostic test results (e.g., X-rays) and (2) making diagnoses based on pre-interpreted test results. Linear mixed-effects models revealed a benefit of collaborating pairs over individuals across tasks, yet not for one task in particular (analysis: Mpairs = 0.51, SD = 0.08 vs. Mindividuals = 0.44, SD = 0.11; diagnoses: Mpairs = 0.49, SD = 0.17 vs. Mindividuals = 0.43, SD = 0.23), Estimates = 0.06, 95% CI 0.01 – 0.11, p = .031. We conclude that the possibility of team members to correct each other’s errors and to make use of a greater knowledge base may pay off both when analyzing information and when integrating it into a diagnosis. These findings encourage the strategic use of collaboration with a colleague when making complex decisions