Effect of a metacognitive intervention on cognitive heuristic use during diagnostic reasoning

Medical judgment and decision-making frequently occur under conditions of uncertainty. In order to reduce the complexity of diagnosis, physicians often rely on cognitive heuristics. Use of heuristics during clinical reasoning can be effective; however when used inappropriately the result can be flawed reasoning, medical errors and patient harm. Many researchers have attempted to debias individuals from inappropriate heuristic use by designing interventions based on normative theories of decision-making. There have been few attempts to debias individuals using interventions based on descriptive decision-making theories. Objectives: (1) Assess use of Anchoring and Adjustment and Confirmation Bias during diagnostic reasoning; (2) Investigate the impact of heuristic use on diagnostic accuracy; (3) Determine the impact of a metacognitive intervention based on the Mental Model Theory designed to reduce biased judgment by inducing physicians to 'think about how they think'; and (4) Test a novel technique using eye-tracking to determine heuristic use and diagnostic accuracy within mode of thinking as defined by the Dual Process Theory. Methods: Medical students and residents assessed clinical scenarios using a computer system, specified a diagnosis, and designated the data used to arrive at the diagnosis. During case analysis, subjects either verbalized their thoughts or wore eye-tracking equipment to capture eye movements and pupil size as they diagnosed cases. Diagnostic data specified by the subject was used to measure heuristic use and assess the impact of heuristic use on diagnostic accuracy. Eye-tracking data was used to determine the frequency of heuristic use (Confirmation Bias only) and mode of thinking. Statistic models were executed to determine the effect of the metacognitive intervention. Results: Use of cognitive heuristics during diagnostic reasoning was common for this subject population. Logistic regression showed case difficulty to be an important factor contributing to diagnostic error. The metacognitive intervention had no effect on heuristic use and diagnostic accuracy. Eye-tracking data reveal this subject population infrequently assess cases in the Intuitive mode of thinking; spend more time in the Analytical mode of thinking, and switches between the two modes frequently as they reason through a case to arrive at a diagnosis

Assessing the use of cognitive heuristic representativeness in clinical reasoning.

We performed a pilot study to investigate use of the cognitive heuristic Representativeness in clinical reasoning. We tested a set of tasks and assessments to determine whether subjects used the heuristics in reasoning, to obtain initial frequencies of heuristic use and related cognitive errors, and to collect cognitive process data using think-aloud techniques. The study investigates two aspects of the Representativeness heuristic - judging by perceived frequency and representativeness as causal beliefs. Results show that subjects apply both aspects of the heuristic during reasoning, and make errors related to misapplication of these heuristics. Subjects in this study rarely used base rates, showed significant variability in their recall of base rates, demonstrated limited ability to use provided base rates, and favored causal data in diagnosis. We conclude that the tasks and assessments we have developed provide a suitable test-bed to study the cognitive processes underlying heuristic errors

Effect of a limited-enforcement intelligent tutoring system in dermatopathology on student errors, goals and solution paths

OBJECTIVES: Determine effects of a limited-enforcement intelligent tutoring system in dermatopathology on student errors, goals and solution paths. Determine if limited enforcement in a medical tutoring system inhibits students from learning the optimal and most efficient solution path. Describe the type of deviations from the optimal solution path that occur during tutoring, and how these deviations change over time. Determine if the size of the problem-space (domain scope), has an effect on learning gains when using a tutor with limited enforcement.\ud \ud METHODS: Analyzed data mined from 44 pathology residents using SlideTutor-a Medical Intelligent Tutoring System in Dermatopathology that teaches histopathologic diagnosis and reporting skills based on commonly used diagnostic algorithms. Two subdomains were included in the study representing sub-algorithms of different sizes and complexities. Effects of the tutoring system on student errors, goal states and solution paths were determined.\ud \ud RESULTS: Students gradually increase the frequency of steps that match the tutoring system's expectation of expert performance. Frequency of errors gradually declines in all categories of error significance. Student performance frequently differs from the tutor-defined optimal path. However, as students continue to be tutored, they approach the optimal solution path. Performance in both subdomains was similar for both errors and goal differences. However, the rate at which students progress toward the optimal solution path differs between the two domains. Tutoring in superficial perivascular dermatitis, the larger and more complex domain was associated with a slower rate of approximation towards the optimal solution path.\ud \ud CONCLUSIONS: Students benefit from a limited-enforcement tutoring system that leverages diagnostic algorithms but does not prevent alternative strategies. Even with limited enforcement, students converge toward the optimal solution path.\ud \u

Factors affecting feeling-of-knowing in a medical intelligent tutoring system: the role of immediate feedback as a metacognitive scaffold

Previous studies in our laboratory have shown the benefits of immediate feedback on cognitive performance for pathology residents using an intelligent tutoring system (ITS) in pathology. In this study, we examined the effect of immediate feedback on metacognitive performance, and investigated whether other metacognitive scaffolds will support metacognitive gains when immediate feedback is faded. Twenty-three participants were randomized into intervention and control groups. For both groups, periods working with the ITS under varying conditions were alternated with independent computer-based assessments. On day 1, a within-subjects design was used to evaluate the effect of immediate feedback on cognitive and metacognitive performance. On day 2, a between-subjects design was used to compare the use of other metacognitive scaffolds (intervention group) against no metacognitive scaffolds (control group) on cognitive and metacognitive performance, as immediate feedback was faded. Measurements included learning gains (a measure of cognitive performance), as well as several measures of metacognitive performance, including Goodman-Kruskal gamma correlation (G), bias, and discrimination. For the intervention group, we also computed metacognitive measures during tutoring sessions. Results showed that immediate feedback in an intelligent tutoring system had a statistically significant positive effect on learning gains, G and discrimination. Removal of immediate feedback was associated with decreasing metacognitive performance, and this decline was not prevented when students used a version of the tutoring system that provided other metacognitive scaffolds. Results obtained directly from the ITS suggest that other metacognitive scaffolds do have a positive effect on G and discrimination, as immediate feedback is faded. We conclude that immediate feedback had a positive effect on both metacognitive and cognitive gains in a medical tutoring system. Other metacognitive scaffolds were not sufficient to replace immediate feedback in this study. However, results obtained directly from the tutoring system are not consistent with results obtained from assessments. In order to facilitate transfer to real-world tasks, further research will be needed to determine the optimum methods for supporting metacognition as immediate feedback is faded.\ud \u