Performance of scoring systems in selecting short stay medical admissions suitable for assessment in same day emergency care:an analysis of diagnostic accuracy in a UK hospital setting

OBJECTIVES: To assess the performance of the Amb score and Glasgow Admission Prediction Score (GAPS) in identifying acute medical admissions suitable for same day emergency care (SDEC) in a large urban secondary centre. DESIGN: Retrospective assessment of routinely collected data from electronic healthcare records. SETTING: Single large urban tertiary care centre. PARTICIPANTS: All unplanned admissions to general medicine on Monday–Friday, episodes starting 08:00–16:59 hours and lasting up to 48 hours, between 1 April 2019 and 9 March 2020. MAIN OUTCOME MEASURES: Sensitivity, specificity, positive and negative predictive value of the Amb score and GAPS in identifying patients discharged within 12 hours of arrival. RESULTS: 7365 episodes were assessed. 94.6% of episodes had an Amb score suggesting suitability for SDEC. The positive predictive value of the Amb score in identifying those discharged within 12 hours was 54.5% (95% CI 53.3% to 55.8%). The area under the receiver operating characteristic curve (AUROC) for the Amb score was 0.612 (95% CI 0.599 to 0.625). 42.4% of episodes had a GAPS suggesting suitability for SDEC. The positive predictive value of the GAPS in identifying those discharged within 12 hours was 50.5% (95% CI 48.4% to 52.7%). The AUROC for the GAPS was 0.606 (95% CI 0.590 to 0.622). 41.4% of the population had both an Amb and GAPS score suggestive of suitability for SDEC and 5.7% of the population had both and Amb and GAPS score suggestive of a lack of suitability for SDEC. CONCLUSIONS: The Amb score and GAPS had poor discriminatory ability to identify acute medical admissions suitable for discharge within 12 hours, limiting their utility in selecting patients for assessment within SDEC services within this diverse patient population

Variability and performance of NHS England's 'reason to reside' criteria in predicting hospital discharge in acute hospitals in England:a retrospective, observational cohort study

OBJECTIVES: NHS England (NHSE) advocates ‘reason to reside’ (R2R) criteria to support discharge planning. The proportion of patients without R2R and their rate of discharge are reported daily by acute hospitals in England. R2R has no interoperable standardised data model (SDM), and its performance has not been validated. We aimed to understand the degree of intercentre and intracentre variation in R2R-related metrics reported to NHSE, define an SDM implemented within a single centre Electronic Health Record to generate an electronic R2R (eR2R) and evaluate its performance in predicting subsequent discharge. DESIGN: Retrospective observational cohort study using routinely collected health data. SETTING: 122 NHS Trusts in England for national reporting and an acute hospital in England for local reporting. PARTICIPANTS: 6 602 706 patient-days were analysed using 3-month national data and 1 039 592 patient-days, using 3-year single centre data. MAIN OUTCOME MEASURES: Variability in R2R-related metrics reported to NHSE. Performance of eR2R in predicting discharge within 24 hours. RESULTS: There were high levels of intracentre and intercentre variability in R2R-related metrics (p<0.0001) but not in eR2R. Informedness of eR2R for discharge within 24 hours was low (J-statistic 0.09–0.12 across three consecutive years). In those remaining in hospital without eR2R, 61.2% met eR2R criteria on subsequent days (76% within 24 hours), most commonly due to increased NEWS2 (21.9%) or intravenous therapy administration (32.8%). CONCLUSIONS: Reported R2R metrics are highly variable between and within acute Trusts in England. Although case-mix or community care provision may account for some variability, the absence of a SDM prevents standardised reporting. Following the development of a SDM in one acute Trust, the variability reduced. However, the performance of eR2R was poor, prone to change even when negative and unable to meaningfully contribute to discharge planning

Road map for clinicians to develop and evaluate AI predictive models to inform clinical decision-making

Background Predictive models have been used in clinical care for decades. They can determine the risk of a patient developing a particular condition or complication and inform the shared decision-making process. Developing artificial intelligence (AI) predictive models for use in clinical practice is challenging; even if they have good predictive performance, this does not guarantee that they will be used or enhance decision-making. We describe nine stages of developing and evaluating a predictive AI model, recognising the challenges that clinicians might face at each stage and providing practical tips to help manage them.Findings The nine stages included clarifying the clinical question or outcome(s) of interest (output), identifying appropriate predictors (features selection), choosing relevant datasets, developing the AI predictive model, validating and testing the developed model, presenting and interpreting the model prediction(s), licensing and maintaining the AI predictive model and evaluating the impact of the AI predictive model. The introduction of an AI prediction model into clinical practice usually consists of multiple interacting components, including the accuracy of the model predictions, physician and patient understanding and use of these probabilities, expected effectiveness of subsequent actions or interventions and adherence to these. Much of the difference in whether benefits are realised relates to whether the predictions are given to clinicians in a timely way that enables them to take an appropriate action.Conclusion The downstream effects on processes and outcomes of AI prediction models vary widely, and it is essential to evaluate the use in clinical practice using an appropriate study design

How far back do we need to look to capture diagnoses in electronic health records? A retrospective observational study of hospital electronic health record data

Objectives: Analysis of routinely collected electronic health data is a key tool for long-term condition research and practice for hospitalised patients. This requires accurate and complete ascertainment of a broad range of diagnoses, something not always recorded on an admission document at a single point in time. This study aimed to ascertain how far back in time electronic hospital records need to be interrogated to capture long-term condition diagnoses. Design: Retrospective observational study of routinely collected hospital electronic health record data. Setting: Queen Elizabeth Hospital Birmingham (UK)-linked data held by the PIONEER acute care data hub. Participants: Patients whose first recorded admission for chronic obstructive pulmonary disease (COPD) exacerbation (n=560) or acute stroke (n=2142) was between January and December 2018 and who had a minimum of 10 years of data prior to the index date. Outcome measures: We identified the most common International Classification of Diseases version 10-coded diagnoses received by patients with COPD and acute stroke separately. For each diagnosis, we derived the number of patients with the diagnosis recorded at least once over the full 10-year lookback period, and then compared this with shorter lookback periods from 1 year to 9 years prior to the index admission. Results: Seven of the top 10 most common diagnoses in the COPD dataset reached &gt;90% completeness by 6 years of lookback. Atrial fibrillation and diabetes were &gt;90% coded with 2–3 years of lookback, but hypertension and asthma completeness continued to rise all the way out to 10 years of lookback. For stroke, 4 of the top 10 reached 90% completeness by 5 years of lookback; angina pectoris was &gt;90% coded at 7 years and previous transient ischaemic attack completeness continued to rise out to 10 years of lookback. Conclusion: A 7-year lookback captures most, but not all, common diagnoses. Lookback duration should be tailored to the conditions being studied

Examining organisational responses to performance-based financial incentive systems : a case study using NHS staff influenza vaccination rates from 2012/13 to 2019/20

Objective: Financial incentives are often applied to motivate desirable performance across organisations in healthcare systems. In the 2016/17 financial year, the National Health Service (NHS) in England set a national performance-based incentive to increase uptake of the influenza vaccination amongst front-line staff. Since then, the threshold levels needed for hospital trusts to achieve the incentive (i.e., the targets) have ranged from 70% to 80%. The present study examines the impact of this financial incentive across eight vaccination seasons. Design: A retrospective observational study examining routinely recorded rates of influenza vaccination amongst staff in all acute NHS hospital trusts across eight vaccination seasons (2012/13-2019/20). The number of trusts included varied per year, from 127 to 137, due to organisational changes. McCrary’s density test is conducted to determine if the number of hospital trusts narrowly achieving the target by the end of each season is higher than would be expected in the absence of any responsiveness to the target. We refer to this bunching above the target threshold as a “threshold effect”. Results: In the years before a national incentive was set, 9%-31% of NHS Trusts reported achieving the target, compared with 43%-74% in the four years after. Threshold effects did not emerge before the national incentive for payment was set; however, since then, threshold effects have appeared every year. Some trusts report narrowly achieving the target each year, both as the target rises and falls. Threshold effects were not apparent at targets for partial payments. Conclusions: We provide compelling evidence that performance-based financial incentives produced threshold effects. Policymakers who set such incentives are encouraged to track threshold effects since they contain information on how organisations are responding to an incentive, what enquiries they may wish to make, how the incentive may be improved and what unintended effects it may be having