Evaluation of an automated safety surveillance system using risk adjusted sequential probability ratio testing

<p>Abstract</p> <p>Background</p> <p>Automated adverse outcome surveillance tools and methods have potential utility in quality improvement and medical product surveillance activities. Their use for assessing hospital performance on the basis of patient outcomes has received little attention. We compared risk-adjusted sequential probability ratio testing (RA-SPRT) implemented in an automated tool to Massachusetts public reports of 30-day mortality after isolated coronary artery bypass graft surgery.</p> <p>Methods</p> <p>A total of 23,020 isolated adult coronary artery bypass surgery admissions performed in Massachusetts hospitals between January 1, 2002 and September 30, 2007 were retrospectively re-evaluated. The RA-SPRT method was implemented within an automated surveillance tool to identify hospital outliers in yearly increments. We used an overall type I error rate of 0.05, an overall type II error rate of 0.10, and a threshold that signaled if the odds of dying 30-days after surgery was at least twice than expected. Annual hospital outlier status, based on the state-reported classification, was considered the gold standard. An event was defined as at least one occurrence of a higher-than-expected hospital mortality rate during a given year.</p> <p>Results</p> <p>We examined a total of 83 hospital-year observations. The RA-SPRT method alerted 6 events among three hospitals for 30-day mortality compared with 5 events among two hospitals using the state public reports, yielding a sensitivity of 100% (5/5) and specificity of 98.8% (79/80).</p> <p>Conclusions</p> <p>The automated RA-SPRT method performed well, detecting all of the true institutional outliers with a small false positive alerting rate. Such a system could provide confidential automated notification to local institutions in advance of public reporting providing opportunities for earlier quality improvement interventions.</p

Near real-time vaccine safety surveillance using electronic health records-a systematic review of the application of statistical methods.

PURPOSE: Pre-licensure studies have limited ability to detect rare adverse events (AEs) to vaccines, requiring timely post-licensure studies. With the increasing availability of electronic health records (EHR) near real-time vaccine safety surveillance using these data has emerged as an option. We reviewed methods currently used to inform development of similar systems for countries considering their introduction. METHODS: Medline, EMBASE and Web of Science were searched, with additional searches of conference abstract books. Questionnaires were sent to organizations worldwide to ascertain unpublished studies. Eligible studies used EHR and regularly assessed pre-specified AE to vaccine(s). Key features of studies were compared descriptively. RESULTS: From 2779 studies, 31 were included from the USA (23), UK (6), and Taiwan and New Zealand (1 each). These were published/conducted between May 2005 and April 2015. Thirty-eight different vaccines were studied, focusing mainly on influenza (47.4%), especially 2009 H1N1 vaccines. Forty-six analytic approaches were used, reflecting frequency of EHR updates and the AE studied. Poisson-based maximized sequential probability ratio test was the most common (43.5%), followed by its binomial (23.9%) and conditional versions (10.9%). Thirty-seven of 49 analyses (75.5%) mentioned control for confounding, using an adjusted expected rate (51.4% of those adjusting), stratification (16.2%) or a combination of a self-controlled design and stratification (13.5%). Guillain-Barré syndrome (11.9%), meningitis/encephalitis/myelitis (11.9%) and seizures (10.8%) were studied most often. CONCLUSIONS: Near real-time vaccine safety surveillance using EHR has developed over the past decade but is not yet widely used. As more countries have access to EHR, it will be important that appropriate methods are selected, considering the data available and AE of interest

Improved diagnosis and management of sepsis and bloodstream infection

Sepsis is a severe organ dysfunction triggered by infections, and a leading cause of hospitalization and death. Concurrent bloodstream infection (BSI) is common and around one third of sepsis patients have positive blood cultures. Prompt diagnosis and treatment is crucial, but there is a trade-off between the negative effects of over diagnosis and failure to recognize sepsis in time. The emerging crisis of antimicrobial resistance has made bacterial infections more difficult to treat, especially gram-negative pathogens such as Pseudomonas aeruginosa. The overall aim with this thesis was to improve diagnosis, assess the influence of time to antimicrobial treatment and explore prognostic bacterial virulence markers in sepsis and BSI. The papers are based on observational data from 7 cohorts of more than 100 000 hospital episodes. In addition, whole genome sequencing has been performed on approximately 800 invasive P. aeruginosa isolates collected from centers in Europe and Australia. Paper I showed that automated surveillance of sepsis incidence using the Sepsis-3 criteria is feasible in the non-ICU setting, with examples of how implementing this model generates continuous epidemiological data down to the ward level. This information can be used for directing resources and evaluating quality-of-care interventions. In Paper II, evidence is provided for using peripheral oxygen saturation (SpO2) to diagnose respiratory dysfunction in sepsis, proposing the novel thresholds 94% and 90% to get 1 and 2 SOFA points, respectively. This has important implications for improving sepsis diagnosis, especially when conventional arterial blood gas measurements are unavailable. Paper III verified that sepsis surveillance data can be utilized to develop machine learning screening tools to improve early identification of sepsis. A Bayesian network algorithm trained on routine electronic health record data predicted sepsis onset within 48 hours with better discrimination and earlier than conventional NEWS2 outside the ICU. The results suggested that screening may primarily be suited for the early admission period, which have broader implications also for other sepsis screening tools. Paper IV demonstrated that delays in antimicrobial treatment with in vitro pathogen coverage in BSI were associated with increased mortality after 12 hours from blood culture collection, but not at 1, 3, and 6 hours. This indicates a time window where clinicians should focus on the diagnostic workup, and proposes a target for rapid diagnostics of blood cultures. Finally, Paper V showed that the virulence genotype had some influence on mortality and septic shock in P. aeruginosa BSI, however, it was not a major prognostic determinant. Together these studies contribute to better understanding of the sepsis and BSI populations, and provide several suggestions to improve diagnosis and timing of treatment, with implications for clinical practice. Future works should focus on the implementation of sepsis surveillance, clinical trials of time to antimicrobial treatment and evaluating the prognostic importance of bacterial genotype data in larger populations from diverse infection sources and pathogens

Postmarket sequential database surveillance of medical products

Thesis (Ph. D.)--Massachusetts Institute of Technology, Engineering Systems Division, 2013.Cataloged from PDF version of thesis.Includes bibliographical references (p. 193-212).This dissertation focuses on the capabilities of a novel public health data system - the Sentinel System - to supplement existing postmarket surveillance systems of the U.S. Food and Drug Administration (FDA). The Sentinel System is designed to identify and assess safety risks associated with drugs, therapeutic biologics, vaccines, and medical devices that emerge post-licensure. Per the initiating legislation, the FDA must complete a priori evaluations of the Sentinel System's technical capabilities to support regulatory decision-making. This research develops qualitative and quantitative tools to aid the FDA in such evaluations, particularly with regard to the Sentinel System's novel sequential database surveillance capabilities. Sequential database surveillance is a "near real-time" sequential statistical method to evaluate pre-specified exposure-outcome pairs. A "signal" is detected when the data suggest an excess risk that is statistically significant. The qualitative tool - the Sentinel System Pre- Screening Checklist - is designed to determine whether the Sentinel System is well suited, on its face, to evaluate a pre-specified exposure-outcome pair. The quantitative tool - the Sequential Database Surveillance Simulator - allows the user to explore virtually whether sequential database surveillance of a particular exposure-outcome pair is likely to generate evidence to identify and assess safety risks in a timely manner to support regulatory decision-making. Particular attention is paid to accounting for uncertainties including medical product adoption and utilization, misclassification error, and the unknown true excess risk in the environment. Using vaccine examples and the simulator to illustrate, this dissertation first demonstrates the tradeoffs associated with sample size calculations in sequential statistical analysis, particularly the tradeoff between statistical power and median sample size. Second, it demonstrates differences in performance between various surveillance configurations when using distributed database systems. Third, it demonstrates the effects of misclassification error on sequential database surveillance, and specifically how such errors may be accounted for in the design of surveillance. Fourth, it considers the complexities of modeling new medical product adoption, and specifically, the existence of a "dual market" phenomenon for these new medical products. This finding raises non-trivial generalizability concerns regarding evidence generated via sequential database surveillance when performed immediately post-licensure.by Judith C. Maro.Ph.D

Epidemiological surveillance of drug safety using cumulative sequential analysis in electronic healthcare data

Background: Methods for safety signal detection in electronic healthcare data analysing data sequentially are being developed to meet the limitations of spontaneous reporting systems. Objectives: This study aims to provide an overview of the literature on sequential analysis of electronic healthcare data and describe the development and testing of a novel epidemiological surveillance system. Methods: We searched Medline, Embase, PubMed, Scopus, Web of Science, and the Cochrane Library applying similar in- and exclusion criteria as those of a previous systematic review. The proposed system consisted of repeated cohort studies and was tested in an emulated prospective setting. Two signal evaluations were performed with several sensitivity analyses and a target trial emulation. Findings: In the literature, 11 studies analysed the data sequentially of which two applied traditional epidemiological methods. Epidemiological surveillance of several exposures and outcomes can be successfully conducted with the newly proposed sequential analysis of electronic healthcare data. Signal evaluation studies confirmed the results of the system. Conclusions: Very few studies in the literature analysed data at multiple time points, although this seems to be a prerequisite for testing the methods in a realistic setting. We demonstrated the feasibility of a sequential surveillance system using electronic healthcare data

Topics in construction safety and health : struck-by and caught-in hazards : an interdisciplinary annotated bibliography

"These referenced articles provide literature on the dangers to construction workers from job hazards in their occupations including the equipment they use and the type of work environment they are working in" - NIOSHTIC-2NIOSHTIC no. 20068258Production of this document was supported by cooperative agreement OH 009762 from the National Institute for Occupational Safety and Health (NIOSH). The contents are solely the responsibility of the authors and do not necessarily represent the official views of NIOSH.Struck-by-and-Caught-in-Hazards-annotated-bibliography.pdfcooperative agreement OH 009762 from the National Institute for Occupational Safety and Healt

Safety‐oriented discrete event model for airport A‐SMGCS reliability assessment

A detailed analysis of State of the Art Technologies and Procedures into Airport Advanced-Surface Movement Guidance and Control Systems has been provided in this thesis, together with the review ofStatistical Monte Carlo Analysis, Reliability Assessment and Petri Nets theories. This practical and theoretical background has lead the author to the conclusion that there is a lack of linkage in between these fields. At the same of time the rapid increasing of Air Traffic all over the world, has brought in evidence the urgent need of practical instruments able to identify and quantify the risks connected with Aircraft operations on the ground, since the Airport has shown to be the actual ‘bottle neck’ of the entire Air Transport System. Therefore, the only winning approach to such a critical matter has to be multi-disciplinary, sewing together apparently different subjects, coming from the most disparate areas of interest and trying to fulfil the gap. The result of this thesis work has come to a start towards the end, when a Timed Coloured Petri Net (TCPN) model of a ‘sample’ Airport A-SMGCS has been developed, that is capable of taking into account different orders of questions arisen during these recent years and tries to give them some good answers. The A-SMGCS Airport model is, in the end, a parametric tool relying on Discrete Event System theory, able to perform a Reliability Analysis of the system itself, that: • uses a Monte Carlo Analysis applied to a Timed Coloured Petri Net, whose purpose is to evaluate the Safety Level of Surface Movements along an Airport • lets the user to analyse the impact of Procedures and Reliability Indexes of Systems such as Surface Movement Radars, Automatic Dependent Surveillance-Broadcast, Airport Lighting Systems, Microwave Sensors, and so on… onto the Safety Level of Airport Aircraft Transport System • not only is a valid instrument in the Design Phase, but it is useful also into the Certifying Activities an in monitoring the Safety Level of the above mentioned System with respect to changes to Technologies and different Procedures.This TCPN model has been verified against qualitative engineering expectations by using simulation experiments and occupancy time schedules generated a priori. Simulation times are good, and since the model has been written into Simulink/Stateflow programming language, it can be compiled to run real-time in C language (Real-time workshop and Stateflow Coder), thus relying on portable code, able to run virtually on any platform, giving even better performances in terms of execution time. One of the most interesting applications of this work is the estimate, for an Airport, of the kind of A-SMGCS level of implementation needed (Technical/Economical convenience evaluation). As a matter of fact, starting from the Traffic Volume and choosing the kind of Ground Equipment to be installed, one can make predictions about the Safety Level of the System: if the value is compliant with the TLS required by ICAO, the A-SMGCS level of Implementation is sufficiently adequate. Nevertheless, even if the Level of Safety has been satisfied, some delays due to reduced or simplified performances (even if Safety is compliant) of some of the equipment (e.g. with reference to False Alarm Rates) can lead to previously unexpected economical consequences, thus requiring more accurate systems to be installed, in order to meet also Airport economical constraints. Work in progress includes the analysis of the effect of weather conditions and re-sequencing of a given schedule. The effect of re-sequencing a given schedule is not yet enough realistic since the model does not apply inter arrival and departure separations. However, the model might show some effect on different sequences based on runway occupancy times. A further developed model containing wake turbulence separation conditions would be more sensitive for this case. Hence, further work will be directed towards: • The development of On-Line Re-Scheduling based on the available actual runway/taxiway configuration and weather conditions. • The Engineering Safety Assessment of some small Italian Airport A-SMGCSs (Model validation with real data). • The application of Stochastic Differential Equations systems in order to evaluate the collision risk on the ground inside the Place alone on the Petri Net, in the event of a Short Term Conflict Alert (STCA), by adopting Reich Collision Risk Model. • Optimal Air Traffic Control Algorithms Synthesis (Adaptive look-ahead Optimization), by Dynamically Timed Coloured Petri Nets, together with the implementation of Error-Recovery Strategies and Diagnosis Functions