Assessment, surgeon, and society

An increasing public demand to monitor and assure the quality of care provided by physicians and surgeons has been accompanied by a deepening appreciation within the profession of the demands of self-regulation and the need for accountability. To respond to these developments, the public and the profession have turned increasingly to assessment, both to establish initial competence and to ensure that it is maintained throughout a career. Fortunately, this comes at a time when there have been significant advances in the breadth and quality of the assessment tools available. This article provides an overview of the drivers of change in assessment which includes the educational outcomes movement, the development of technology, and advances in assessment. It then outlines the factors that are important in selecting assessment devices as well as a system for classifying the methods that are available. Finally, the drivers of change have spawned a number of trends in the assessment of competence as a surgeon. Three of them are of particular note, simulation, workplace-based assessment, and the assessment of new competences, and each is reviewed with a focus on its potential

Peer assessment of competence

Objective This instalment in the series on professional assessment summarises how peers are used in the evaluation process and whether their judgements are reliable and valid. Method The nature of the judgements peers can make, the aspects of competence they can assess and the factors limiting the quality of the results are described with reference to the literature. The steps in implementation are also provided. Results Peers are asked to make judgements about structured tasks or to provide their global impressions of colleagues. Judgements are gathered on whether certain actions were performed, the quality of those actions and ⁄ or their suitability for a particular purpose. Peers are used to assess virtually all aspects of professional competence, including technical and nontechnical aspects of proficiency. Factors influencing the quality of those assessments are reliability, relationships, stakes and equivalence. Conclusion Given the broad range of ways peer evaluators can be used and the sizeable number of competencies they can be asked to judge, generalisations are difficult to derive and this form of assessment can be good or bad depending on how it is carried out

Virtual patients in health professions education

La práctica clínica de hoy con hospitalizaciones cortas, riesgos de responsabilidad legal, exigentes normas de acreditación y la menor disponibilidad de docentes, han llevado a la búsqueda de nuevas opciones para la formación de los estudiantes en las profesiones de la salud. Los pacientes virtuales (PV) son programas de computador que simulan a un paciente real y están diseñados para la formación y evaluación del razonamiento clínico. Ofrecen un medio seguro para el aprendizaje de nuestros estudiantes. Se utilizan en evaluación sumativa y se requiere de 6-9 casos para que la evaluación tenga validez. El diseño del PV determina el tipo de evaluación que estamos realizando. Los pacientes electrónicos virtuales son complejos de diseñar lo que conlleva a un alto costo para su desarrollo. A pesar de lo anterior es una herramienta ideal para utilizarse en países en desarrollo y su diseño se ajusta a las necesidades particulares de cada país o institución. Presentamos dos experiencias exitosas en la aplicación de esta tecnología. Otra opción es repotenciar un PV que ya está terminado, haciendo los ajustes necesarios para la nueva situación donde se va a aplicar. Describimos el proceso que es más rápido y menos dispendioso que construir un PV de novo. Existen un banco de la Comunidad Europea, eVIP, donde se encuentran aproximadamente 340 casos que se pueden repotenciar de acuerdo a las necesidades de la institución o país. En resumen, se trata de una excelente herramienta para promover razonamiento clínico accesible a todos.201-209Hospital practice today is characterized by hospitalizations, legal liabilities, strict norms of accreditation, and faculty at short hand. This situation has led to the search of new options for the education of our students in health professions. Electronic Virtual Patients (VP) are computer programs that simulate a real patient, which are designed for the formation and assessment of clinical reasoning and knowledge, depending on their design. They offer a secure scenario for the education of our students. They can be used for assessment but they require 6-9 cases to have content validity. The VP design determines the kind of evaluation to be applied. VPs are very complex to design and they entail a high development cost. Regardless of this situation, they are an ideal tool to be used in developing countries, responding to the socioeconomic situation of the institution or country. We present two successful experiences in two different continents. Another option is to refurbish the existing VP to local needs. We describe the process, which is faster and more efficient than building the case again. In the European Union we find the eVIP program, which has a repertory on its website with 340 cases that can be refurbished by anyone according to the institution or local requirements. In summary, VPs are an excellent resource for clinical reasoning, available to everybody

Setting school-level outcome standards

To establish international standards for medical schools, an appropriate panel of experts must decide on performance standards. A pilot test of such standards was set in the context of a multidimensional (multiple-choice question examination, objective structured clinical examination, faculty observation) examination at 8 leading schools in China. Methods  A group of 16 medical education leaders from a broad array of countries met over a 3-day period. These individuals considered competency domains, examination items, and the percentage of students who could fall below a cut-off score if the school was still to be considered as meeting competencies. This 2-step process started with a discussion of the borderline school and the relative difficulty of a borderline school in achieving acceptable standards in a given competency domain. Committee members then estimated the percentage of students falling below the standard that is tolerable at a borderline school and were allowed to revise their ratings after viewing pilot data. Results  Tolerable failure rates ranged from 10% to 26% across competency domains and examination types. As with other standard-setting exercises, standard deviations from initial to final estimates of the tolerable failure rates fell, but the cut-off scores did not change significantly. Final, but not initial cut-off scores were correlated with student failure rates ( r =  0.59, P  = 0.03). Discussion  This paper describes a method to set school-level outcome standards at an international level based on prior established standard-setting methods. Further refinement of this process and validation using other examinations in other countries will be needed to achieve accurate international standards.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/71572/1/j.1365-2929.2005.02374.x.pd

Assessing clinical reasoning skills using Script Concordance Test (SCT) and extended matching questions (EMQs): A pilot for urology trainees

Introduction: Clinical reasoning skill is the core of medical competence. Commonly used assessment methods for medical competence have limited ability to evaluate critical thinking and reasoning skills. Script Concordance Test (SCT) and Extended Matching Questions(EMQs) are the evolving tests which are considered to be valid and reliable tools for assessing clinical reasoning and judgment. We performed this pilot study to determine whether SCT and EMQs can differentiate clinical reasoning ability among urology residents, interns and medical students.Methods: This was a cross-sectional study in which an examination with 48 SCT-based items on eleven clinical scenarios and four themed EMQs with 21 items were administered to a total of 27 learners at three differing levels of experience i.e. 9 urology residents, 6 interns and 12 fifth year medical students. A non-probability convenience sampling was done. The SCTs and EMQs were developed from clinical situations representative of urological practice by 5 content experts (urologists) and assessed by a medical education expert. Learners\u27 responses were scored using the standard and the graduated key. A one way analysis of variance (ANOVA) was conducted to compare the mean scores across the level of experience. A p-value of \u3c 0.05 was considered statistically significant. Test reliability was estimated by Cronbach α. A focused group discussion with candidates was done to assess their perception of test.Results: Both SCT and EMQs successfully differentiated residents from interns and students. Statistically significant difference in mean score was found for both SCT and EMQs among the 3 groups using both the standard and the graduated key. The mean scores were higher for all groups as measured by the graduated key compared to the standard key. The internal consistency (Cronbach\u27s α) was 0.53 and 0.6 for EMQs and SCT, respectively. Majority of the participants were satisfied with regard to time, environment, instructions provided and the content covered and nearly all felt that the test helped them in thinking process particularly clinical reasoning.Conclusions: Our data suggest that both SCT and EMQs are capable of discriminating between learners according to their clinicalexperience in urology. As there is a wide acceptability by all candidates, these tests could be used to assess and enhance clinical reasoningskills. More research is needed to prove validity of these tests

Interprofessional communication in a sociohierarchical culture: development of the TRI-O guide

Objectives: Interprofessional education (IPE) and collaborative practice are essential for patient safety. Effective teamwork starting with partnership-based communications should be introduced early in the educational process. Many societies in the world hold socio-hierarchical culture with a wide power distance, which makes collaboration among health professionals challenging. Since an appropriate communication framework for this context is not yet available, this study filled that gap by developing a guide for interprofessional communication, which is best suited to the socio-hierarchical and socio-cultural contexts. Materials and methods: The draft of the guide was constructed based on previous studies of communication in health care in a socio-hierarchical context, referred to international IPE literature, and refined by focus group discussions among various health professionals. Nominal group technique, also comments from national and international experts of communication skills in health care, was used to validate the guide. A pilot study with a pre–posttest design was conducted with 53 first- and 107 fourth-year undergraduate medical, nursing, and health nutrition students. Results: We developed the “TRI-O” guide of interprofessional communication skills, emphasizing “open for collaboration, open for information, open for discussion”, and found that the application of the guide during training was feasible and positively influenced students’ perceptions. Conclusion: The findings suggest that the TRI-O guide is beneficial to help students initiate partnership-based communication and mutual collaboration among health professionals in the socio-hierarchical and socio-cultural context

Learning to mark: a qualitative study of the experiences and concerns of medical markers

BACKGROUND: Although there is published research on the methods markers use in marking various types of assessment, there is relatively little information on the processes markers use in approaching a marking exercise. This qualitative paper describes the preparation and experiences of general practice (GP) teachers who undertake marking a written assessment in an undergraduate medical course. METHODS: Semi-structured interviews were conducted with seven of the 16 GP tutors on an undergraduate course. The purposive sample comprised two new markers, two who had marked for a couple of years and three experienced markers. Each respondent was interviewed twice, once following a formative assessment of a written case study, and again after a summative assessment. All interviews were audio-taped and analysed for emerging themes. A respondent validation exercise was conducted with all 16 GP tutors. RESULTS: Markers had internal concerns about their ability to mark fairly and made considerable efforts to calibrate their marking. They needed guidance and coaching when marking for the first time and adopted a variety of marking styles, reaching a decision through a number of routes. Dealing with pass/fail borderline scripts and the consequences of the mark on the student were particular concerns. Even experienced markers felt the need to calibrate their marks both internally and externally CONCLUSION: Previous experience of marking appears to improve markers' confidence and is a factor in determining the role which markers adopt. Confidence can be improved by giving clear instructions, along with examples of marking. The authors propose that one method of providing this support and coaching could be by a process of peer review of a selection of papers prior to the main marking. New markers in particular would benefit from further guidance, however they are influenced by others early on in their marking career and course organisers should be mindful of this when arranging double marking

Predicting fitness to practise events in international medical graduates who registered as UK doctors via the Professional and Linguistic Assessments Board (PLAB) system: a national cohort study

Background International medical graduates working in the UK are more likely to be censured in relation to fitness to practise compared to home graduates. Performance on the General Medical Council’s (GMC’s) Professional and Linguistic Assessments Board (PLAB) tests and English fluency have previously been shown to predict later educational performance in this group of doctors. It is unknown whether the PLAB system is also a valid predictor of unprofessional behaviour and malpractice. The findings would have implications for regulatory policy. Methods This was an observational study linking data relating to fitness to practise events (referral or censure), PLAB performance, demographic variables and English language competence, as evaluated via the International English Language Test System (IELTS). Data from 27,330 international medical graduates registered with the GMC were analysed, including 210 doctors who had been sanctioned in relation to at least one fitness to practise issue. The main outcome was risk of eventual censure (including a warning). Results The significant univariable educational predictors of eventual censure (versus no censures or referrals) were lower PLAB part 1 (hazard ratio [HR], 0.99; 95% confidence interval, 0.98 to 1.00) and part 2 scores (HR, 0.94; 0.91 to 0.97) at first sitting, multiple attempts at both parts of the PLAB, lower IELTS reading (HR, 0.79; 0.65 to 0.94) and listening scores (HR, 0.76; 0.62 to 0.93) and higher IELTS speaking scores (HR, 1.28; 1.04 to 1.57). Multiple resits at either part of the PLAB and higher IELTS speaking score (HR, 1.49; 1.20 to 1.84) were also independent predictors of censure. We estimated that the proposed limit of four attempts at both parts of the PLAB would reduce the risk in this entire group by only approximately two censures per 5 years in this group of doctors. Conclusions Making the PLAB, or any replacement assessment, more stringent and raising the required standards of English reading and listening may result in fewer fitness to practice events in international medical graduates. However, the number of PLAB resits permitted would have to be further capped to meaningfully impact the risk of sanctions in this group of doctor