Anderson, Deverick J

Brown, Sarah R

Bulger, Amy L

Hamad, Yasir

Jones, Travis M

Klompas, Michael

Massaro, Anthony F

O\u27Brien, Cara

Pande, Anupam

Rhee, Chanu

Tobin, Kathleen A

Warren, David K

English

Digital Commons@Becker

Washington University School of MedicineDigital Commons@BeckerOpen Access Publications2018Variability in determining sepsis time zero andbundle compliance rates for the centers formedicare and medicaid services SEP-1 measureChanu RheeSarah R. BrownTravis M. JonesCara O'BrienAnupam PandeSee next page for additional authorsFollow this and additional works at: https://digitalcommons.wustl.edu/open_access_pubsAuthorsChanu Rhee, Sarah R. Brown, Travis M. Jones, Cara O'Brien, Anupam Pande, Yasir Hamad, Amy L. Bulger,Kathleen A. Tobin, Anthony F. Massaro, Deverick J. Anderson, David K. Warren, and Michael KlompasInfection Control & Hospital Epidemiology (2018), 39, 994–996doi:10.1017/ice.2018.134Concise CommunicationVariability in determining sepsis time zero and bundle compliancerates for the Centers for Medicare and Medicaid Services SEP-1measureChanu Rhee MD, MPH1,2, Sarah R. Brown MD2, Travis M. Jones PharmD3, Cara O’Brien MD4,Anupam Pande MD, MPH5, Yasir Hamad MD5, Amy L. Bulger RN, MPH6, Kathleen A. Tobin RN7,Anthony F. Massaro MD2, Deverick J. Anderson MD, MPH3, David K. Warren MD, MPH5 andMichael Klompas MD, MPH1,2 for the CDC Prevention Epicenters Program1Department of Population Medicine, Harvard Medical School/Harvard Pilgrim Health Care Institute, Boston, Massachusetts, 2Department of Medicine, Brighamand Women’s Hospital, Boston, Massachusetts, 3Duke Center for Antimicrobial Stewardship and Infection Prevention, Duke University School of Medicine,Durham, North Carolina, 4Department of Medicine, Duke University Medical Center, Durham, North Carolina, 5Department of Medicine, Washington UniversitySchool of Medicine, St Louis, Missouri, 6Department of Quality and Safety, Brigham and Women’s Hospital, Boston, Massachusetts and 7Lawrence Center forQuality and Safety, Massachusetts General Hospital, Boston, MassachusettsAbstractWe compared sepsis “time zero” and Centers for Medicare and Medicaid Services (CMS) SEP-1 pass rates among 3 abstractors in 3hospitals. Abstractors agreed on time zero in 29 of 80 (36%) cases. Perceived pass rates ranged from 9 of 80 cases (11%) to 19 of 80 cases(23%). Variability in time zero and perceived pass rates limits the utility of SEP-1 for measuring quality.(Received 19 March 2018; accepted 9 May 2018; electronically published June 22, 2018)In October 2015, the Centers for Medicare and Medicaid Services(CMS) implemented the “SEP-1” sepsis core measure requiringUS hospitals to report compliance with 3- and 6-hour bundles ofcare for patients with severe sepsis or septic shock.1 Hospitals arenow devoting substantial resources to measuring and improvingSEP-1 adherence, which requires all bundle components be metto “pass.”2,3SEP-1 bundle adherence is measured relative to sepsis “timezero,” defined as the first point at which there is documentation ofsuspected or confirmed infection, 2 or more systemic inflamma-tory response syndrome criteria, and one or more organ dys-function criteria within a 6-hour window.1 Time zero is alsotriggered if a clinician explicitly documents severe sepsis or septicshock. Given this complex definition, different abstractors mayreach different conclusions about time zero, which, in turn, couldlead to different impressions on whether cases passed or failedSEP-1.4,5We compared time-zero determinations and SEP-1 pass ratesamong hospital abstractors and clinicians in 3 US hospitals.We also examined clinical factors associated with lower agree-ment rates.MethodsWe randomly selected 80 SEP-1 cases discharged between July 1and December 31, 2016, at 3 US tertiary-care hospitals: Brighamand Women’s Hospital in Boston, Massachusetts; Barnes-JewishHospital in St Louis, Missouri; and Duke University Hospital inDurham, North Carolina. Each case was reviewed by the officialhospital SEP-1 abstractor and by 2 clinicians at each hospital(either internal medicine physicians or clinical pharmacists) forall SEP-1 components, including time zero (Table 1) and whethercases passed. Abstractors were blinded to one another’s deter-minations. Clinician reviewers underwent 1 hour of training onSEP-1 abstraction by the lead investigator (C.R.) to encouragestandardization using the CMS specification in place during thestudy period.1 We applied CMS exclusion criteria prior toselecting cases for review (ie, outside hospital transfer, severesepsis criteria not met on chart review, goals of care limitations,and antibiotic administration prior to 24 hours before time zero).1We compared agreement on time-zero and SEP-1 pass-versus-fail rates among the 3 abstractors at each site. Time zero wasconsidered concordant between abstractors if they were within±1 minute of each other. We also conducted sensitivity analysesallowing for agreement if time-zero determinations were within1 hour and 3 hours of each other.We calculated interobserver variability on whether cases pas-sed or failed SEP-1 using the Fleiss κ for 3-abstractor comparisonsand the Cohen κ for 2-abstractor comparisons.6 We consideredκ> 0.75 to be strong agreement, κ= 0.40–0.75 to be moderateagreement, and κ< 0.40 to be poor agreement.7Cite this article: Rhee C, et al. (2018). Variability in determining sepsis time zero andbundle compliance rates for the Centers for Medicare and Medicaid Services SEP-1 measure.Infection Control & Hospital Epidemiology 2018, 39, 994–996. doi: 10.1017/ice.2018.134Author for correspondence: Chanu Rhee, MD, MPH, Department of PopulationMedicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, 401 ParkDrive, Suite 401, Boston, MA 02215. E-mail: crhee@bwh.harvard.edu© 2018 by The Society for Healthcare Epidemiology of America. All rights reserved.We conducted a multivariate analysis to identify factorsassociated with disagreement on time zero. Covariates includedage >65, sex, hospital length-of-stay >7 days, sepsis time zerooccurring after hospital admission (per the hospital abstractor),and which organ dysfunction criteria triggered time zero per thehospital abstractor (ie, hypotension, lactate >2.0mmol/L, provi-der documentation of severe sepsis/septic shock, or other organdysfunction).Statistical analyses were performed using SAS version 9.3software (SAS Institute, Cary, NC) and an online softwarepackage for interrater reliability calculations.8 The study wasapproved by the institutional review boards at HarvardPilgrim Health Care Institute, Partners Healthcare, WashingtonUniversity School of Medicine, and Duke University HealthSystem.ResultsOf the 80 study cases, all 3 abstractors agreed on time zero in 29cases (36.3%). Agreement rates by hospital are shown in Table 2.Among the 51 cases for which there was a discrepancy, the mediantime-zero difference between clinician abstractors and hospitalabstractors was 40 minutes (interquartile range [IQR], 0–70 minutes;range, 0 minutes to 11.6 days). Agreement on time zero was betterbut still marginal when the window for concordance was expandedto 1 hour (47 of 80 cases, 58.9%) or 3 hours (54 of 80 cases, 67.5%).Hospital abstractors identified time zero as occurring in theemergency department or day of admission in 55 of 80 cases (68.8%).Agreement among the 3 abstractors was better in these cases (25 of55 cases, 45.5%) than in cases in which time zero occurred afterhospital admission (4 of 25 cases, 16%; P= .01). On multivariateanalysis, hospital-onset of sepsis was independently associated with atleast 1 abstractor disagreeing on time zero (odds ratio [OR], 8.2; 95%confidence interval [CI], 1.6–40.7; P= .01), whereas age, sex, length ofstay>7 days, and organ dysfunction criteria were not.Overall, hospital abstractors identified 19 of 80 cases (23.8%)as passing SEP-1. Among the clinician abstractors, 9 cases(11.3%) passed when using the abstractor with the strictestassessments at each hospital. When using the highest pass ratesper clinician abstractor at each hospital, 15 cases (18.8%) passed.Interrater reliability among the 3 abstractors for determiningSEP-1 compliance was poor (Fleiss κ, 0.39).When assessing agreement by at least 1 clinician abstractoridentifying the same time zero as the hospital abstractor, agree-ment increased to 56 of 80 cases (70.0%), and interrater reliabilityfor determining SEP-1 compliance was better but still onlymoderate (Cohen κ, 0.67). When examining agreement onlybetween the 2 clinician abstractors at each hospital, agreement ontime zero occurred in 34 of 80 cases (42.5%) and interraterreliability for SEP-1 compliance was poor (Cohen κ, 0.28).DiscussionWe found poor agreement between abstractors for identifying sepsistime zero and whether cases passed the CMS SEP-1 measure.Table 2. Agreement in Determining Sepsis Time Zero and SEP-1 Compliance by HospitalaOverall (N= 80),n/N (%)Hospital 1 (N= 29),n/N (%)Hospital 2 (N= 21),n/N (%)Hospital 3 (N= 30),n/N (%)Agreement (All 3 Abstractors)Exact (±1 minute) 29/80 (36.3) 15/29 (51.7) 8/21 (38.1) 6/30 (20)± 1 hour 47/80 (58.8) 20/29 (69) 12/21 (57.1) 15/30 (50)± 3 hours 54/80 (67.5) 24/29 (82.8) 12/21 (57.1) 18/30 (60)Median difference in time zero for clinician vs hospital abstractors (IQR)b 40min (0–70) 41min (0–139) 13min (1–568) 49min (0–210)Agreement for sepsis occurring in emergency department (within ±1min)c 25/55 (45.5) 13/23 (56.5) 7/11 (63.6) 5/21 (23.8)Agreement for sepsis occurring after admissionb (within ±1min) 4/25 (16) 2/6 (33.3) 1/10 (10) 1/9 (11.1)Interrater reliability for overall SEP-1 pass vs fail (Fleiss κ)d 0.39 0.39 0.24 0.37Note. IQR, interquartile range.aData for each hospital are presented in no specific order.bRepresents the median difference in time zero determined by both clinician abstractors compared to each hospital’s official quality abstractor. Only cases where there was at least 1disagreement were included in the analysis.cThe timing of sepsis onset was determined using the official hospital abstractor’s time zero.dInterrater reliability calculations included all 3 abstractors at each hospital.Table 1. SEP-1 Criteria for Severe Sepsis “Time Zero”All three of the following within a 6-hour windowa:1. Documentation of suspected or confirmed infection2. ≥2 Systemic Inflammatory Response Syndrome criteria:∙ Temperature> 38.3°C or< 36.0°C; heart rate >90 beats per minute;respirations >20 per minute; white blood cell count >12 or <4 × 103/µLor >10% bands3. ≥1 Organ dysfunctionb:∙ Systolic blood pressure< 90mmHg (or decrease by >40mmHg) ormean arterial blood pressure <65mmHg∙ Lactate >2.0mmol/L∙ Initiation of mechanical ventilation or noninvasive positive pressureventilation∙ Creatinine >2.0mg/dL, or urine output <0.5mL/kg/hour for 2 hours∙ Total bilirubin >2.0mg/dL∙ Platelet count <100,000 × 109/L∙ International normalized ratio >1.5 or aPTT >60 secondsNote. aPPT, activated partial thromboplastin time.aTime zero is the time at which the last sign of severe sepsis (documentation of suspectedinfection, ≥2 systemic inflammatory response syndrome criteria, and organ dysfunction)within that 6-hour window is noted. Alternatively, severe sepsis criteria are met if providerdocumentation of suspected or confirmed severe sepsis or septic shock is present.bExcludes organ dysfunction explicitly documented as chronic.Infection Control & Hospital Epidemiology 995Agreement improved to only a moderate rating when requiring just1 of 2 clinician abstractors to agree with a hospital’s officialabstractor. Sepsis onset after hospital admission was associated withlower agreement rates compared to sepsis present on admission.The SEP-1 measure relies on determining sepsis time zero tocalculate 3- and 6-hour bundle compliance rates, but severalpotential sources of error as well as subjectivity may have affectedthe results. Abstractors need to assess many different parts of thechart (eg, vital signs, laboratory tests, clinical notes, and medi-cation administration records) to determine time zero and overallSEP-1 compliance. Abstractors must exercise judgment to decidewhether clinicians suspect infection, whether organ dysfunction ispresent, and whether organ dysfunction is new or chronic.Reviewers may also need to review dozens of progress notes,including multiple versions of the same note that have beencopied and pasted, to find the first documentation of suspectedinfection, particularly when sepsis occurs after hospital admission.More broadly, sepsis is an elusive entity to define and identify.There is no gold standard for sepsis, and even expert cliniciansusing common definitions often disagree on whether sepsis ispresent or absent.9,10Our study has several limitations. Clinicians may be less adept atabstracting data for quality measures than trained hospital abstrac-tors. We focused on agreement for sepsis time zero and overallSEP-1 pass rates, but variability in abstracting individual bundlecomponents could also contribute to disagreements in perceived passrates. Our study was conducted in academic hospitals and may notbe generalizable to community hospitals, where sepsis cases maydiffer in their level of complexity. Finally, the CMS specification forSEP-1 continues to change over time, and we were unable to eval-uate the impact of recent changes on interrater reliability.In conclusion, there is significant variability between differentabstractors in determining severe sepsis time zero and SEP-1 com-pliance rates. These findings underscore the importance of ensuringadequate standardization of quality measures, especially complex oneslike SEP-1, that require substantial judgment for implementation.Acknowledgments. The content is solely the responsibility of the authorsand does not necessarily represent the official views of the Centers for DiseaseControl and Prevention or the Agency for Healthcare Research and Quality.Financial support. This study was funded by the Prevention EpicentersProgram of the Centers for Disease Control and Prevention (grant no.U54CK000484). C.R. received support from the Agency for HealthcareResearch and Quality (grant no. K08HS025008).Potential conflicts of interest. None of the authors have any conflicts todisclose.References1. Centers for Medicare and Medicaid Services. QualityNet—InpatientHospitals Specifications Manual. Quality website. https://www.qualitynet.org. Accessed March 19, 2018.2. Venkatesh AK, Slesinger T, Whittle J, et al. Preliminary performance onthe new CMS Sepsis-1 national quality measure: early insights from theemergency quality network (E-QUAL). Ann Emerg Med 2018;71:10–15.3. Barbash IJ, Rak KJ, Kuza CC, Kahn JM. Hospital perceptions of Medicare’ssepsis quality reporting initiative. J Hosp Med 2017;12:963–968.4. Klompas M, Rhee C. The CMS sepsis mandate: right disease, wrongmeasure. Ann Intern Med 2016;165:517–518.5. Aaronson EL, Filbin MR, Brown DF, Tobin K, Mort EA. New mandatedCenters for Medicare and Medicaid Services requirements for sepsisreporting: caution from the field. J Emerg Med 2017;52:109–116.6. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med(Zagreb) 2012;22:276–282.7. Stevens JP, Kachniarz B, Wright SB, et al. When policy gets it right:variability in US hospitals’ diagnosis of ventilator-associated pneumonia*.Crit Care Med 2014;42:497–503.8. ReCal3: Reliability for 3 + coders. dfreelon.org website. http://dfreelon.org/utils/recalfront/recal3/. Accessed March 19, 2018.9. Rhee C, Kadri SS, Danner RL, et al. Diagnosing sepsis is subjective and highlyvariable: a survey of intensivists using case vignettes. Crit Care 2016;20:89.10. Angus DC, Seymour CW, Coopersmith CM, et al. A framework for thedevelopment and interpretation of different sepsis definitions and clinicalcriteria. Crit Care Med 2016;44:e113–e121.996 Chanu Rhee et al

Variability in determining sepsis time zero and bundle compliance rates for the centers for medicare and medicaid services SEP-1 measure

https://digitalcommons.wustl.edu/cgi/viewcontent.cgi?article=8000&amp;context=open_access_pubs

Variability in determining sepsis time zero and bundle compliance rates for the centers for medicare and medicaid services SEP-1 measure

Abstract

Similar works

Full text

Available Versions

Digital Commons@Becker