Road map for clinicians to develop and evaluate AI predictive models to inform clinical decision-making

Background Predictive models have been used in clinical care for decades. They can determine the risk of a patient developing a particular condition or complication and inform the shared decision-making process. Developing artificial intelligence (AI) predictive models for use in clinical practice is challenging; even if they have good predictive performance, this does not guarantee that they will be used or enhance decision-making. We describe nine stages of developing and evaluating a predictive AI model, recognising the challenges that clinicians might face at each stage and providing practical tips to help manage them.Findings The nine stages included clarifying the clinical question or outcome(s) of interest (output), identifying appropriate predictors (features selection), choosing relevant datasets, developing the AI predictive model, validating and testing the developed model, presenting and interpreting the model prediction(s), licensing and maintaining the AI predictive model and evaluating the impact of the AI predictive model. The introduction of an AI prediction model into clinical practice usually consists of multiple interacting components, including the accuracy of the model predictions, physician and patient understanding and use of these probabilities, expected effectiveness of subsequent actions or interventions and adherence to these. Much of the difference in whether benefits are realised relates to whether the predictions are given to clinicians in a timely way that enables them to take an appropriate action.Conclusion The downstream effects on processes and outcomes of AI prediction models vary widely, and it is essential to evaluate the use in clinical practice using an appropriate study design

Eliciting willingness-to-pay to prevent hospital medication administration errors in the UK: a contingent valuation survey.

Medication errors are common in hospitals. These errors can result in adverse drug events (ADEs), which can reduce the health and well-being of patients', and their relatives and caregivers. Interventions have been developed to reduce medication errors, including those that occur at the administration stage. OBJECTIVE: We aimed to elicit willingness-to-pay (WTP) values to prevent hospital medication administration errors. DESIGN AND SETTING: An online, contingent valuation (CV) survey was conducted, using the random card-sort elicitation method, to elicit WTP to prevent medication errors. PARTICIPANTS: A representative sample of the UK public. METHODS: Seven medication error scenarios, varying in the potential for harm and the severity of harm, were valued. Scenarios were developed with input from: clinical experts, focus groups with members of the public and piloting. Mean and median WTP values were calculated, excluding protest responses or those that failed a logic test. A two-part model (logit, generalised linear model) regression analysis was conducted to explore predictive characteristics of WTP. RESULTS: Responses were collected from 1001 individuals. The proportion of respondents willing to pay to prevent a medication error increased as the severity of the ADE increased and was highest for scenarios that described actual harm occurring. Mean WTP across the scenarios ranged from £45 (95% CI £36 to £54) to £278 (95% CI £200 to £355). Several factors influenced both the value and likelihood of WTP, such as: income, known experience of medication errors, sex, field of work, marriage status, education level and employment status. Predictors of WTP were not, however, consistent across scenarios. CONCLUSIONS: This CV study highlights how the UK public value preventing medication errors. The findings from this study could be used to carry out a cost-benefit analysis which could inform implementation decisions on the use of technology to reduce medication administration errors in UK hospitals

Understanding physicians’ behavior toward alerts about nephrotoxic medications in outpatients: a cross-sectional analysis

Background: Although most outpatients are relatively healthy, many have chronic renal insufficiency, and high override rates for suggestions on renal dosing have been observed. To better understand the override of renal dosing alerts in an outpatient setting, we conducted a study to evaluate which patients were more frequently prescribed contraindicated medications, to assess providers’ responses to suggestions, and to examine the drugs involved and the reasons for overrides. Methods: We obtained data on renal alert overrides and the coded reasons for overrides cited by providers at the time of prescription from outpatient clinics and ambulatory hospital-based practices at a large academic health care center over a period of 3 years, from January 2009 to December 2011. For detailed chart review, a group of 6 trained clinicians developed the appropriateness criteria with excellent inter-rater reliability (κ = 0.93). We stratified providers by override frequency and then drew samples from the high- and low-frequency groups. We measured the rate of total overrides, rate of appropriate overrides, medications overridden, and the reason(s) for override. Results: A total of 4120 renal alerts were triggered by 584 prescribers in the study period, among which 78.2% (3,221) were overridden. Almost half of the alerts were triggered by 40 providers and one-third was triggered by high-frequency overriders. The appropriateness rates were fairly similar, at 28.4% and 31.6% for high- and low-frequency overriders, respectively. Metformin, glyburide, hydrochlorothiazide, and nitrofurantoin were the most common drugs overridden. Physicians’ appropriateness rates were higher than the rates for nurse practitioners (32.9% vs. 22.1%). Physicians with low frequency override rates had higher levels of appropriateness for metformin than the high frequency overriders (P = 0.005). Conclusion: A small number of providers accounted for a large fraction of overrides, as was the case with a small number of drugs. These data suggest that a focused intervention targeting primarily these providers and medications has the potential to improve medication safety

On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop

Background: Clinical decision support (CDS) for electronic prescribing systems (computerized physician order entry) should help prescribers in the safe and rational use of medicines. However, the best ways to alert users to unsafe or irrational prescribing are uncertain. Specifically, CDS systems may generate too many alerts, producing unwelcome distractions for prescribers, or too few alerts running the risk of overlooking possible harms. Obtaining the right balance of alerting to adequately improve patient safety should be a priority. Methods: A workshop funded through the European Regional Development Fund was convened by the University Hospitals Birmingham NHS Foundation Trust to assess current knowledge on alerts in CDS and to reach a consensus on a future research agenda on this topic. Leading European researchers in CDS and alerts in electronic prescribing systems were invited to the workshop. Results: We identified important knowledge gaps and suggest research priorities including (1) the need to determine the optimal sensitivity and specificity of alerts; (2) whether adaptation to the environment or characteristics of the user may improve alerts; and (3) whether modifying the timing and number of alerts will lead to improvements. We have also discussed the challenges and benefits of using naturalistic or experimental studies in the evaluation of alerts and suggested appropriate outcome measures. Conclusions: We have identified critical problems in CDS, which should help to guide priorities in research to evaluate alerts. It is hoped that this will spark the next generation of novel research from which practical steps can be taken to implement changes to CDS systems that will ultimately reduce alert fatigue and improve the design of future systems

A patient safety toolkit for family practices

Objectives: Major gaps remain in our understanding of primary care patient safety. We describe a toolkit for measuring patient safety in family practices. Methods: Six tools were used in 46 practices. These tools were: NHS Education for Scotland Trigger Tool, NHS Education for Scotland Medicines Reconciliation Tool, Primary Care Safequest, Prescribing Safety Indicators, PREOS-PC, and Concise Safe Systems Checklist. Results: PC-Safequest showed that most practices had a well-developed safety climate. However, the Trigger Tool revealed that a quarter of events identified were associated with moderate or substantial harm, with a third originating in primary care and avoidable. Although medicines reconciliation was undertaken within 2 days in >70% of cases, necessary discussions with a patient/carer did not always occur. The prescribing safety indicators identified 1,435 instances of potentially hazardous prescribing or lack of recommended monitoring (from 92,649 patients). The Concise Safe Systems Checklist found that 25% of staff thought their practice provided inadequate follow-up for vulnerable patients discharged from hospital and inadequate monitoring of non-collection of prescriptions. Most patients had a positive perception of the safety of their practice although 45% identified at least one safety problem in the past year. Conclusions: Patient safety is complex and multidimensional. The Patient Safety Toolkit is easy to use and hosted on a single platform with a collection of tools generating practical and actionable information. It enables family practices to identify safety deficits that they can review and change procedures to improve their patient safety across a key sets of patient safety issues

Electronic prescribing systems in hospitals to improve medication safety: a multimethods research programme.

Electronic prescribing (ePrescribing) systems allow health-care professionals to enter prescriptions and manage medicines using a computer. We set out to find out how these ePrescribing systems are chosen, set up and used in English hospitals. Given that these systems are designed to improve medication safety, we looked at whether or not these systems affected the number of prescribing errors made (mistakes such as ordering the wrong dose of medication). We also tried to see whether or not the systems were good value for money (or more cost-effective). Finally, we made recommendations to help hospitals choose, set up and use ePrescribing systems. We found that setting up ePrescribing systems was very difficult because there is a need to take into consideration how different pharmacists, nurses and doctors work, and the different work that needs to be carried out for different diseases and medical conditions. We recorded a link between the implementation of ePrescribing systems and a reduction in some high-risk prescribing errors in two out of three study sites. Given that the error reductions corresponded to the warnings triggered by the system, we concluded that the system is likely to have caused the error reduction. Prescribing errors may lead to adverse events that lead to death, impaired quality of life and longer hospital stays. The cost of an ePrescribing system increased in proportion to reduced errors, reaching £4.31 per patient per year for the site that experienced the greatest reduction in prescribing errors (i.e. site S). This estimate is based on assumptions in the model and how much a health service is willing to pay for a unit of health benefit. To help professionals choose, set up and use ePrescribing systems in the future, we produced an online ePrescribing Toolkit (www.eprescribingtoolkit.com/; accessed 21 December 2019) that, with support from NHS England, is becoming widely used internationally

Patients’ evaluations of patient safety in English general practices: a cross-sectional study

Background: The frequency and nature of safety problems and harm in general practices has previously relied on information supplied by health professionals, and scarce attention has been paid to experiences of patients. Aim: To examine patient-reported experiences and outcomes of patient safety in Primary Care in England. Design and Setting: Cross-sectional study in 45 general practices. Method: A postal version of the Patient Reported Experiences and Outcomes of Safety in Primary Care (PREOS-PC) questionnaire was sent to a random sample of 6,736 patients. Main outcome measures included “practice activation” (what does the practice do to create a safe environment); “patient activation” (how pro-active are patients in ensuring safe healthcare delivery); “experiences of safety events” (safety errors); “outcomes of safety” (harm); and “overall perception of safety” (how safe do patients rate their practice). Results: 1,244 patients (18.4%) returned completed questionnaires. Scores were high for “practice activation” (mean (standard error) = 80.4 out of 100 (2.0)) and low for “patient activation” (26.3 out of 100 (2.6)). A substantial proportion of patients (45%) reported having experienced at least one safety problem in the previous 12 months, mostly related to appointments (33%), diagnosis (17%), patient-provider communication (15%), and coordination between providers (14%). 221 patients (23%) reported some degree of harm in the previous 12 months. The overall assessment of the level of safety of their practices was generally high (86.0 out of 100 (16.8)). Conclusion: Priority areas for patient safety improvement in general practices in England include appointments, diagnosis, communication, coordination and patient activation

A complex ePrescribing-based Anti-Microbial Stewardship (ePAMS+) intervention for hospitals combining technological and behavioural components : protocol for a feasibility trial

BackgroundAntimicrobial resistance is a leading global public health threat, with inappropriate use of antimicrobials in healthcare contributing to its development. Given this urgent need, we developed a complex ePrescribing-based Anti-Microbial Stewardship intervention (ePAMS+).MethodsePAMS+ includes educational and organisational behavioural elements, plus guideline-based clinical decision support to aid optimal antimicrobial use in hospital inpatients. ePAMS+ particularly focuses on prompt initiation of antimicrobials, followed by early review once test results are available to facilitate informed decision-making on stopping or switching where appropriate. A mixed-methods feasibility trial of ePAMS+ will take place in two NHS acute hospital care organisations. Qualitative staff interviews and observation of practice will respectively gather staff views on the technical component of ePAMS+ and information on their use of ePAMS+ in routine work. Focus groups will elicit staff and patient views on ePAMS+; one-to-one interviews will discuss antimicrobial stewardship with staff and will record patient experiences of receiving antibiotics and their thoughts on inappropriate prescribing. Qualitative data will be analysed thematically. Fidelity Index development will enable enactment of ePAMS+ to be measured objectively in a subsequent trial assessing the effectiveness of ePAMS+. Quantitative data collection will determine the feasibility of extracting data and deriving key summaries of antimicrobial prescribing; we will quantify variability in the primary outcome, number of antibiotic defined daily doses, to inform the future larger-scale trial design.DiscussionThis trial is essential to determine the feasibility of implementing the ePAMS+ intervention and measuring relevant outcomes, prior to evaluating its clinical and cost-effectiveness in a full scale hybrid cluster-randomised stepped-wedge clinical trial. Findings will be shared with study sites and with qualitative research participants and will be published in peer-reviewed journals and presented at academic conferences.Trial registrationThe qualitative and Fidelity Index research were approved by the Health and Research Authority and the North of Scotland Research Ethics Service (ref: 19/NS/0174). The feasibility trial and quantitative analysis (protocol v1.0, 15 December 2021) were approved by the London South East Research Ethics Committee (ref: 22/LO/0204) and registered with ISRCTN (ISRCTN 13429325) on 24 March 202