The reliability of in-training assessment when performance improvement is taken into account

During in-training assessment students are frequently assessed over a longer period of time and therefore it can be expected that their performance will improve. We studied whether there really is a measurable performance improvement when students are assessed over an extended period of time and how this improvement affects the reliability of the overall judgement. In-training assessment results were obtained from 104 students on rotation at our university hospital or at one of the six affiliated hospitals. Generalisability theory was used in combination with multilevel analysis to obtain reliability coefficients and to estimate the number of assessments needed for reliable overall judgement, both including and excluding performance improvement. Students’ clinical performance ratings improved significantly from a mean of 7.6 at the start to a mean of 7.8 at the end of their clerkship. When taking performance improvement into account, reliability coefficients were higher. The number of assessments needed to achieve a reliability of 0.80 or higher decreased from 17 to 11. Therefore, when studying reliability of in-training assessment, performance improvement should be considered

The Validation of Geriatric Cases for Interprofessional Education: A Consensus Method

Background: Case-based Interprofessional Education (IPE) can help students practise their roles and responsibilities. To construct these cases, input from experts in clinical practice is essential. Consensus between these experts can be facilitated using consensus methods. In this study, a geriatric focus for the cases was chosen because of the interprofessional nature of geriatrics and the ageing population in healthcare facilities. Methods: By combining the 3 most commonly used consensus methods, we developed a 6-step approach to validate cases for IPE. The 6 steps include 3 expert rounds (Steps 1, 3 and 5) and 2 rounds in which discussion points were formulated by the researcher (Steps 2 and 4). The cases were piloted with students as Step 6. Four facets of a case were included: the patient description, the complemented treatment plan, the difficulty of the case and the scoring of the treatment plan. Our educational setting required constructing 4 cases with increasing difficulty. Results: Step 1: 5 typical geriatric cases were assembled. Step 2: Similar characteristics were defined; 5 cases were merged into 4. The 4 cases showed increasing difficulty levels. Step 3: The constructed cases were validated for patient description authenticity, treatment plan adequacy, difficulty and scoring of the treatment plan. Step 4: The items for further discussion were defined. Step 5: Consensus was reached for all 4 cases through a face-to-face discussion. Step 6: The student pilot for Case 1 showed no significant adjustments. Conclusion: We developed a 6-step consensus method to validate cases for IPE, and we constructed 4 geriatric cases based on this method. While consensus about the patient descriptions and difficulty levels was reached easily, consensus on the treatment plans was more difficult to achieve. Validation of the scoring of the treatment plan was unsuccessful. Further research on this will be conducted

Feasibility and reliability of an in-training assessment programme in an undergraduate clerkship

INTRODUCTION: Structured assessment, embedded in a training programme, with systematic observation, feedback and appropriate documentation may improve the reliability of clinical assessment. This type of assessment format is referred to as in-training assessment (ITA). The feasibility and reliability of an ITA programme in an internal medicine clerkship were evaluated. The programme comprised 4 ward-based test formats and 1 outpatient clinic-based test format. Of the 4 ward-based test formats, 3 were single-sample tests, consisting of 1 student-patient encounter, 1 critical appraisal session and 1 case presentation. The other ward-based test and the outpatient-based test were multiple sample tests, consisting of 12 ward-based case write-ups and 4 long cases in the outpatient clinic. In all the ITA programme consisted of 19 assessments. METHODS: During 41 months, data were collected from 119 clerks. Feasibility was defined as over two thirds of the students obtaining 19 assessments. Reliability was estimated by performing generalisability analyses with 19 assessments as items and 5 test formats as items. RESULTS: A total of 73 students (69%) completed 19 assessments. Reliability expressed by the generalisability coefficients was 0.81 for 19 assessments and 0.55 for 5 test formats. CONCLUSIONS: The ITA programme proved to be feasible. Feasibility may be improved by scheduling protected time for assessment for both students and staff. Reliability may be improved by more frequent use of some of the test formats

A qualitative study of constructive clinical learning experiences.

Little is known about the effectiveness of clinical education. A more educational structure is considered to be potentially beneficial. The following structured components were added to a surgical clerkship: logbooks, an observed student-patient encounter, individual appraisals, feedback on patient notes, and (case) presentations by students. The authors organized two focus-group sessions in which 19 students participated to explore their perceptions about effective clinical learning experiences and the newly introduced structured components. The analysis of the transcripts showed that observation and constructive feedback are key features of clinical training. The structured activities were appreciated and the results show the direction to be taken for further improvement. Learning experiences depended vastly on individual clinicians' educational qualities. Students experienced being on call, assisting in theatre and time for self-study as instructive elements. Recommended clerkship components are: active involvement of students, direct observation, selection of teachers, a positive learning environment and time for self-study