Script Concordance Tests: Guidelines for Construction

International audienc

The script concordance test in radiation oncology: validation study of a new tool to assess clinical reasoning

<p>Abstract</p> <p>Background</p> <p>The Script Concordance test (SCT) is a reliable and valid tool to evaluate clinical reasoning in complex situations where experts' opinions may be divided. Scores reflect the degree of concordance between the performance of examinees and that of a reference panel of experienced physicians. The purpose of this study is to demonstrate SCT's usefulness in radiation oncology.</p> <p>Methods</p> <p>A 90 items radiation oncology SCT was administered to 155 participants. Three levels of experience were tested: medical students (n = 70), radiation oncology residents (n = 38) and radiation oncologists (n = 47). Statistical tests were performed to assess reliability and to document validity.</p> <p>Results</p> <p>After item optimization, the test comprised 30 cases and 70 questions. Cronbach alpha was 0.90. Mean scores were 51.62 (± 8.19) for students, 71.20 (± 9.45) for residents and 76.67 (± 6.14) for radiation oncologists. The difference between the three groups was statistically significant when compared by the Kruskall-Wallis test (p < 0.001).</p> <p>Conclusion</p> <p>The SCT is reliable and useful to discriminate among participants according to their level of experience in radiation oncology. It appears as a useful tool to document the progression of reasoning during residency training.</p

Online clinical reasoning assessment with the Script Concordance test: a feasibility study

BACKGROUND: The script concordance (SC) test is an assessment tool that measures capacity to solve ill-defined problems, that is, reasoning in context of uncertainty. This tool has been used up to now mainly in medicine. The purpose of this pilot study is to assess the feasibility of the test delivered on the Web to French urologists. METHODS: The principle of SC test construction and the development of the Web site are described. A secure Web site was created with two sequential modules: (a) The first one for the reference panel (n = 26) with two sub-tasks: to validate the content of the test and to elaborate the scoring system; (b) The second for candidates with different levels of experience in Urology: Board certified urologists, residents, medical students (5 or 6(th )year). Minimum expected number of participants is 150 for urologists, 100 for residents and 50 for medical students. Each candidate is provided with an individual access code to this Web site. He/she may complete the Script Concordance test several times during his/her curriculum. RESULTS: The Web site has been operational since April 2004. The reference panel validated the test in June of the same year during the annual seminar of the French Society of Urology. The Web site is available for the candidates since September 2004. In six months, 80% of the target figure for the urologists, 68% of the target figure for the residents and 20% of the target figure for the student passed the test online. During these six months, no technical problem was encountered. CONCLUSION: The feasibility of the web-based SC test is successful as two-thirds of the expected number of participants was included within six months. Psychometric properties (validity, reliability) of the test will be evaluated on a large scale (N = 300). If positive, educational impact of this assessment tool will be useful to help urologists during their curriculum for the acquisition of clinical reasoning skills, which is crucial for professional competence

Summative assessment of 5th year medical students' clinical reasoning by script concordance test: requirements and challenges

Background: The Script Concordance Test (SCT) has not been reported in summative assessment of students across the multiple domains of a medical curriculum. We report the steps used to build a test for summative assessment in a medical curriculum. Methods: A 51 case, 158-question, multidisciplinary paper was constructed to assess clinical reasoning in 5th-year. 10–16 experts in each of 7 discipline-based reference panels answered questions on-line. A multidisciplinary group considered reference panel data and data from a volunteer group of 6th Years, who sat the same test, to determine the passing score for the 5th Years. Results: The mean (SD) scores were 63.6 (7.6) and 68.6 (4.8) for the 6th Year (n = 23, alpha = 0.78) and and 5th Year (n = 132, alpha =0.62) groups (p < 0.05), respectively. The passing score was set at 4 SD from the expert mean. Four students failed. Conclusions: The SCT may be a useful method to assess clinical reasoning in medical students in multidisciplinary summative assessments. Substantial investment in training of faculty and students and in the development of questions is required.Paul Duggan and Bernard Charli

Professional approaches in clinical judgements among senior and junior doctors: implications for medical education

<p>Abstract</p> <p>Background</p> <p>Clinical experience has traditionally been highly valued in medical education and clinical healthcare. On account of its multi-faceted nature, clinical experience is mostly difficult to articulate, and is mainly expressed in clinical situations as professional approaches. Due to retirement, hospitals in Scandinavia will soon face a substantial decrease in the number of senior specialist doctors, and it has been discussed whether healthcare will suffer an immense loss of experienced-based knowledge when this senior group leaves the organization. Both senior specialists and junior colleagues are often involved in clinical education, but the way in which these two groups vary in professional approaches and contributions to clinical education has not been so well described. Cognitive psychology has contributed to the understanding of how experience may influence professional approaches, but such studies have not included the effect of differences in position and responsibilities that junior and senior doctors hold in clinical healthcare. In the light of the discussion above, it is essential to describe the professional approaches of senior doctors in relation to those of their junior colleagues. This study therefore aims to describe and compare the professional approaches of junior and senior doctors when making clinical judgements.</p> <p>Methods</p> <p>Critical incident technique was used in interviews with nine senior doctors and nine junior doctors in internal medicine. The interviews were subjected to qualitative content analysis.</p> <p>Result</p> <p>Senior and junior doctors expressed a variety of professional approaches in clinical judgement as follows: use of theoretical knowledge, use of prior experience of cases and courses of events, use of ethical and moral values, meeting and communicating with the patient, focusing on available information, relying on their own ability, getting support and guidance from others and being directed by the organization.</p> <p>Conclusion</p> <p>The most prominent varieties of professional approaches were seen in use of knowledge and work-related experience. Senior doctors know how the organization has worked in the past and have acquired techniques with respect to long-term decisions and their consequences. Junior doctors, on the other hand, have developed techniques and expertise for making decisions based on a restricted amount of information, in relation to patients' wellbeing as well as organizational opportunities and constraints. This study contributes to medical education by elucidating the variation in professional approaches among junior and senior doctors, which can be used as a basis for discussion about clinical judgement, in both pre-clinical and clinical education. Further research is required to explain how these professional approaches are expressed and used in clinical education.</p

General practitioners' reasoning when considering the diagnosis heart failure: a think-aloud study

BACKGROUND: Diagnosing chronic heart failure is difficult, especially in mild cases or early in the course of the disease, and guidelines are not easily implemented in everyday practice. The aim of this study was to investigate general practitioners' diagnostic reasoning about patients with suspected chronic heart failure in comparison with recommendations in European guidelines. METHODS: Think-aloud technique was used. Fifteen general practitioners reasoned about six case vignettes, representing authentic patients with suspected chronic heart failure. Information about each case was added successively in five steps. The general practitioners said their thoughts aloud while reasoning about the probability of the patient having chronic heart failure, and tried to decide about the diagnosis. Arguments for and against chronic heart failure were analysed and compared to recommendations in guidelines. RESULTS: Information about ejection fraction was the most frequent diagnostic argument, followed by information about cardiac enlargement or pulmonary congestion on chest X-ray. However, in a third of the judgement situations, no information about echocardiography was utilized in the general practitioners' diagnostic reasoning. Only three of the 15 doctors used information about a normal electrocardiography as an argument against chronic heart failure. Information about other cardio-vascular diseases was frequently used as a diagnostic argument. CONCLUSIONS: The clinical information was not utilized to the extent recommended in guidelines. Some implications of our study are that 1) general practitioners need more information about how to utilize echocardiography when diagnosing chronic heart failure, 2) guidelines ought to give more importance to information about other cardio-vascular diseases in the diagnostic reasoning, and 3) guidelines ought to treat the topic of diastolic heart failure in a clearer way