Workplace-based assessment: effects of rater expertise

Traditional psychometric approaches towards assessment tend to focus exclusively on quantitative properties of assessment outcomes. This may limit more meaningful educational approaches towards workplace-based assessment (WBA). Cognition-based models of WBA argue that assessment outcomes are determined by cognitive processes by raters which are very similar to reasoning, judgment and decision making in professional domains such as medicine. The present study explores cognitive processes that underlie judgment and decision making by raters when observing performance in the clinical workplace. It specifically focuses on how differences in rating experience influence information processing by raters. Verbal protocol analysis was used to investigate how experienced and non-experienced raters select and use observational data to arrive at judgments and decisions about trainees’ performance in the clinical workplace. Differences between experienced and non-experienced raters were assessed with respect to time spent on information analysis and representation of trainee performance; performance scores; and information processing––using qualitative-based quantitative analysis of verbal data. Results showed expert-novice differences in time needed for representation of trainee performance, depending on complexity of the rating task. Experts paid more attention to situation-specific cues in the assessment context and they generated (significantly) more interpretations and fewer literal descriptions of observed behaviors. There were no significant differences in rating scores. Overall, our findings seemed to be consistent with other findings on expertise research, supporting theories underlying cognition-based models of assessment in the clinical workplace. Implications for WBA are discussed

Gut Feelings as a Third Track in General Practitioners’ Diagnostic Reasoning

BACKGROUND: General practitioners (GPs) are often faced with complicated, vague problems in situations of uncertainty that they have to solve at short notice. In such situations, gut feelings seem to play a substantial role in their diagnostic process. Qualitative research distinguished a sense of alarm and a sense of reassurance. However, not every GP trusted their gut feelings, since a scientific explanation is lacking. OBJECTIVE: This paper explains how gut feelings arise and function in GPs' diagnostic reasoning. APPROACH: The paper reviews literature from medical, psychological and neuroscientific perspectives. CONCLUSIONS: Gut feelings in general practice are based on the interaction between patient information and a GP's knowledge and experience. This is visualized in a knowledge-based model of GPs' diagnostic reasoning emphasizing that this complex task combines analytical and non-analytical cognitive processes. The model integrates the two well-known diagnostic reasoning tracks of medical decision-making and medical problem-solving, and adds gut feelings as a third track. Analytical and non-analytical diagnostic reasoning interacts continuously, and GPs use elements of all three tracks, depending on the task and the situation. In this dual process theory, gut feelings emerge as a consequence of non-analytical processing of the available information and knowledge, either reassuring GPs or alerting them that something is wrong and action is required. The role of affect as a heuristic within the physician's knowledge network explains how gut feelings may help GPs to navigate in a mostly efficient way in the often complex and uncertain diagnostic situations of general practice. Emotion research and neuroscientific data support the unmistakable role of affect in the process of making decisions and explain the bodily sensation of gut feelings.The implications for health care practice and medical education are discussed

Does Experience Matter?: A Comparison of the Practice of Attendings and Residents

OBJECTIVE: To compare the utilization of health care resources and patterns of chronic disease care by patients of medical residents and patients of their attending physicians. MATERIALS AND METHODS: This study involved a longitudinal cohort of 14,554 patients seen over a 1-year period by 149 residents and 36 attendings located in an urban academic medical center. Data were acquired prospectively through a practice management system used to order tests, write prescriptions, and code ambulatory visits. We assessed resource utilization by measuring the total direct costs of care over a 1-year period, including ambulatory and inpatient costs, and the numbers and types of resources used. RESULTS: Residents' patients were similar to attendings' patients in age and gender, but residents' patients were more likely to have Medicaid or Medicare and to have a higher burden of comorbidity. Total annual ambulatory care costs were almost 60% higher for residents' patients than for attendings' patients in unadjusted analyses, and 30% higher in analyses adjusted for differences in case mix (adjusted mean $888 vs$750; P=.0001). The primary cost drivers on the outpatient side were consultations and radiological procedures. Total inpatient costs were almost twice as high for residents' patients compared to attendings' patients in unadjusted analyses, but virtually identical in analyses adjusted for case mix differences (adjusted mean of $849 vs$860). Admission rates were almost double for residents' patients. Total adjusted costs for residents' patients were slightly, but not significantly, higher than for attendings' patients (adjusted mean $1,651 vs$1,540; P>.05). Residents' and attendings' patients generally did not differ in the patterns of care for diabetes, asthma/chronic obstructive pulmonary disease (COPD), congestive heart failure, ischemic heart disease, and depression, except that residents' patients with asthma/COPD, ischemic heart disease, and diabetes were admitted more frequently than attendings' patients. CONCLUSIONS: Our results indicate that residents' patients had higher costs than attendings' patients, but the differences would have been seriously overestimated without adjustment. We conclude that it costs about 7% more for residents to manage patients than for attendings. On the ambulatory side, the larger number of procedures and consults ordered for residents' patients appears to drive the higher costs