Measuring Harm in Health Care: Optimizing Adverse Event Review

OBJECTIVE: The objective of this study was to identify modifiable factors that improve the reliability of ratings of severity of health care-associated harm in clinical practice improvement and research. METHODS: A diverse group of clinicians rated 8 types of adverse events: blood product, device or medical/surgical supply, fall, health care-associated infection, medication, perinatal, pressure ulcer, surgery. We used a generalizability theory framework to estimate the impact of number of raters, rater experience, and rater provider type on reliability. RESULTS: Pharmacists were slightly more precise and consistent in their ratings than either physicians or nurses. For example, to achieve high reliability of 0.83, 3 physicians could be replaced by 2 pharmacists without loss in precision of measurement. If only 1 rater was available for rating, approximately 5% of the reviews for severe harm would have been incorrectly categorized. Reliability was greatly improved with 2 reviewers. CONCLUSIONS: We identified factors that influence the reliability of clinician reviews of health care-associated harm. Our novel use of generalizability analyses improved our understanding of how differences affect reliability. This approach was useful in optimizing resource utilization when selecting raters to assess harm and may have similar applications in other settings in health care

Design and implementation of a decision aid for juvenile idiopathic arthritis medication choices

Abstract Background Randomized trials have demonstrated the efficacy of patient decision aids to facilitate shared decision making in clinical situations with multiple medically reasonable options for treatment. However, little is known about how best to implement these tools into routine clinical practice. In addition, reliable implementation of decision aids has been elusive and spread within pediatrics has been slow. We sought to develop and reliably implement a decision aid for treatment of children with juvenile idiopathic arthritis. Methods To design our decision aid, we partnered with patient, parent, and clinician stakeholders from the Pediatric Rheumatology Care and Outcomes Improvement Network. Six sites volunteered to use quality improvement methods to implement the decision aid. Four of these sites collected parent surveys following visits to assess outcomes. Parents reported on clinician use of the decision aid and the amount of shared decision making and uncertainty they experienced. We used chi-square tests to compare eligible visits with and without use of the decision aid on the experience of shared decision making and uncertainty. Results After 18 rounds of testing and revision, stakeholders approved the decision aid design for regular use. Qualitative feedback from end-users was positive. During the implementation project, the decision aid was used in 35% of visits where starting or switching medication was discussed. Clinicians used the decision aid as intended in 68% of these visits. The vast majority of parents reported high levels of shared decision making following visits with (64/76 = 84%) and without (80/95 = 84%) use of the decision aid (p = 1). Similarly, the vast majority of parents reported no uncertainty following visits with (74/76 = 97%) and without (91/95 = 96%) use of the decision aid (p = 0.58). Conclusions Although user acceptability of the decision aid was high, reliable implementation in routine clinical care proved challenging. Our parsimonious approach to outcome assessment failed to detect a difference between visits with and without use of our aid. Innovative approaches are needed to facilitate use of decision aids and the assessment of outcomes