Automation bias in electronic prescribing

© 2017 The Author(s). Background: Clinical decision support (CDS) in e-prescribing can improve safety by alerting potential errors, but introduces new sources of risk. Automation bias (AB) occurs when users over-rely on CDS, reducing vigilance in information seeking and processing. Evidence of AB has been found in other clinical tasks, but has not yet been tested with e-prescribing. This study tests for the presence of AB in e-prescribing and the impact of task complexity and interruptions on AB. Methods: One hundred and twenty students in the final two years of a medical degree prescribed medicines for nine clinical scenarios using a simulated e-prescribing system. Quality of CDS (correct, incorrect and no CDS) and task complexity (low, low + interruption and high) were varied between conditions. Omission errors (failure to detect prescribing errors) and commission errors (acceptance of false positive alerts) were measured. Results: Compared to scenarios with no CDS, correct CDS reduced omission errors by 38.3% (p < .0001, n = 120), 46.6% (p < .0001, n = 70), and 39.2% (p < .0001, n = 120) for low, low + interrupt and high complexity scenarios respectively. Incorrect CDS increased omission errors by 33.3% (p < .0001, n = 120), 24.5% (p < .009, n = 82), and 26.7% (p < .0001, n = 120). Participants made commission errors, 65.8% (p < .0001, n = 120), 53.5% (p < .0001, n = 82), and 51.7% (p < .0001, n = 120). Task complexity and interruptions had no impact on AB. Conclusions: This study found evidence of AB omission and commission errors in e-prescribing. Verification of CDS alerts is key to avoiding AB errors. However, interventions focused on this have had limited success to date. Clinicians should remain vigilant to the risks of CDS failures and verify CDS

Using multiclass classification to automate the identification of patient safety incident reports by type and severity

Background: Approximately 10% of admissions to acute-care hospitals are associated with an adverse event. Analysis of incident reports helps to understand how and why incidents occur and can inform policy and practice for safer care. Unfortunately our capacity to monitor and respond to incident reports in a timely manner is limited by the sheer volumes of data collected. In this study, we aim to evaluate the feasibility of using multiclass classification to automate the identification of patient safety incidents in hospitals. Methods: Text based classifiers were applied to identify 10 incident types and 4 severity levels. Using the one-versus-one (OvsO) and one-versus-all (OvsA) ensemble strategies, we evaluated regularized logistic regression, linear support vector machine (SVM) and SVM with a radial-basis function (RBF) kernel. Classifiers were trained and tested with “balanced” datasets (n_ Type  = 2860, n_ SeverityLevel  = 1160) from a state-wide incident reporting system. Testing was also undertaken with imbalanced “stratified” datasets (n_ Type  = 6000, n_ SeverityLevel =5950) from the state-wide system and an independent hospital reporting system. Classifier performance was evaluated using a confusion matrix, as well as F-score, precision and recall. Results: The most effective combination was a OvsO ensemble of binary SVM RBF classifiers with binary count feature extraction. For incident type, classifiers performed well on balanced and stratified datasets (F-score: 78.3, 73.9%), but were worse on independent datasets (68.5%). Reports about falls, medications, pressure injury, aggression and blood products were identified with high recall and precision. “Documentation” was the hardest type to identify. For severity level, F-score for severity assessment code (SAC) 1 (extreme risk) was 87.3 and 64% for SAC4 (low risk) on balanced data. With stratified data, high recall was achieved for SAC1 (82.8–84%) but precision was poor (6.8–11.2%). High risk incidents (SAC2) were confused with medium risk incidents (SAC3). Conclusions: Binary classifier ensembles appear to be a feasible method for identifying incidents by type and severity level. Automated identification should enable safety problems to be detected and addressed in a more timely manner. Multi-label classifiers may be necessary for reports that relate to more than one incident type.Ying Wang, Enrico Coiera, William Runciman and Farah Magrab

Conversational agents in healthcare: a systematic review.

Objective: Our objective was to review the characteristics, current applications, and evaluation measures of conversational agents with unconstrained natural language input capabilities used for health-related purposes. Methods: We searched PubMed, Embase, CINAHL, PsycInfo, and ACM Digital using a predefined search strategy. Studies were included if they focused on consumers or healthcare professionals; involved a conversational agent using any unconstrained natural language input; and reported evaluation measures resulting from user interaction with the system. Studies were screened by independent reviewers and Cohen's kappa measured inter-coder agreement. Results: The database search retrieved 1513 citations; 17 articles (14 different conversational agents) met the inclusion criteria. Dialogue management strategies were mostly finite-state and frame-based (6 and 7 conversational agents, respectively); agent-based strategies were present in one type of system. Two studies were randomized controlled trials (RCTs), 1 was cross-sectional, and the remaining were quasi-experimental. Half of the conversational agents supported consumers with health tasks such as self-care. The only RCT evaluating the efficacy of a conversational agent found a significant effect in reducing depression symptoms (effect size d = 0.44, p = .04). Patient safety was rarely evaluated in the included studies. Conclusions: The use of conversational agents with unconstrained natural language input capabilities for health-related purposes is an emerging field of research, where the few published studies were mainly quasi-experimental, and rarely evaluated efficacy or safety. Future studies would benefit from more robust experimental designs and standardized reporting. Protocol Registration: The protocol for this systematic review is registered at PROSPERO with the number CRD42017065917

How to Teach Health IT Evaluation: Recommendations for Health IT Evaluation Courses

Systematic health IT evaluation studies are needed to ensure system quality and safety and to provide the basis for evidence-based health informatics. Well-trained health informatics specialists are required to guarantee that health IT evaluation studies are conducted in accordance with robust standards. Also, policy makers and managers need to appreciate how good evidence is obtained by scientific process and used as an essential justification for policy decisions. In a consensus-based approach with over 80 experts in health IT evaluation, recommendations for the structure, scope and content of health IT evaluation courses on the master or postgraduate level have been developed, supported by a structured analysis of available courses and of available literature. The recommendations comprise 15 mandatory topics and 15 optional topics for a health IT evaluation course

The Role of Formative Evaluation in Promoting Digitally-based Health Equity and Reducing Bias for Resilient Health Systems: The Case of Patient Portals.

OBJECTIVES: Patient portals are increasingly implemented to improve patient involvement and engagement. We here seek to provide an overview of ways to mitigate existing concerns that these technologies increase inequity and bias and do not reach those who could benefit most from them. METHODS: Based on the current literature, we review the limitations of existing evaluations of patient portals in relation to addressing health equity, literacy and bias; outline challenges evaluators face when conducting such evaluations; and suggest methodological approaches that may address existing shortcomings. RESULTS: Various stakeholder needs should be addressed before deploying patient portals, involving vulnerable groups in user-centred design, and studying unanticipated consequences and impacts of information systems in use over time. CONCLUSIONS: Formative approaches to evaluation can help to address existing shortcomings and facilitate the development and implementation of patient portals in an equitable way thereby promoting the creation of resilient health systems

Artificial Intelligence in Clinical Decision Support : Challenges for Evaluating AI and Practical Implications

OBJECTIVES This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance. METHOD A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. RESULTS There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed. CONCLUSION Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application

The impact of electronic records on patient safety : a qualitative study

BACKGROUND: Our aim was to explore NHS staff perceptions and experiences of the impact on patient safety of introducing a maternity system. METHODS: Qualitative semi-structured interviews were conducted with 19 members of NHS staff who represented a variety of staff groups (doctors, midwives, health care assistants), staff grades (consultant and midwife grades) and wards within a maternity unit. Participants represented a single maternity unit at a NHS teaching hospital in the North of England. Interviews were conducted during the first 12 months of the system being implemented and were analysed thematically. RESULTS: Participants perceived there to be an elevated risk to patient safety during the system's implementation. The perceived risks were attributed to a range of social and technical factors. For example, poor system design and human error which resulted in an increased potential for missing information and inputting error. CONCLUSIONS: The first 12 months of introducing the maternity system was perceived to and in some cases had already caused actual risk to patient safety. Trusts throughout the NHS are facing increasing pressure to become paperless and should be aware of the potential adverse impacts on patient safety that can occur when introducing electronic systems. Given the potential for increased risk identified, recommendations for further research and for NHS trusts introducing electronic systems are proposed

Implementing evidence-based medicine in general practice: a focus group based study

BACKGROUND: Over the past years concerns are rising about the use of Evidence-Based Medicine (EBM) in health care. The calls for an increase in the practice of EBM, seem to be obstructed by many barriers preventing the implementation of evidence-based thinking and acting in general practice. This study aims to explore the barriers of Flemish GPs (General Practitioners) to the implementation of EBM in routine clinical work and to identify possible strategies for integrating EBM in daily work. METHODS: We used a qualitative research strategy to gather and analyse data. We organised focus groups between September 2002 and April 2003. The focus group data were analysed using a combined strategy of 'between-case' analysis and 'grounded theory approach'. Thirty-one general practitioners participated in four focus groups. Purposeful sampling was used to recruit participants. RESULTS: A basic classification model documents the influencing factors and actors on a micro-, meso- as well as macro-level. Patients, colleagues, competences, logistics and time were identified on the micro-level (the GPs' individual practice), commercial and consumer organisations on the meso-level (institutions, organisations) and health care policy, media and specific characteristics of evidence on the macro-level (policy level and international scientific community). Existing barriers and possible strategies to overcome these barriers were described. CONCLUSION: In order to implement EBM in routine general practice, an integrated approach on different levels needs to be developed

Drug information resources used by nurse practitioners and collaborating physicians at the point of care in Nova Scotia, Canada: a survey and review of the literature

BACKGROUND: Keeping current with drug therapy information is challenging for health care practitioners. Technologies are often implemented to facilitate access to current and credible drug information sources. In the Canadian province of Nova Scotia, legislation was passed in 2002 to allow nurse practitioners (NPs) to practice collaboratively with physician partners. The purpose of this study was to determine the current utilization patterns of information technologies by these groups of practitioners. METHODS: Nurse practitioners and their collaborating physician partners in Nova Scotia were sent a survey in February 2005 to determine the frequency of use, usefulness, accessibility, credibility, and current/timeliness of personal digital assistant (PDA), computer, and print drug information resources. Two surveys were developed (one for PDA users and one for computer users) and revised based on a literature search, stakeholder consultation, and pilot-testing results. A second distribution to nonresponders occurred two weeks following the first. Data were entered and analysed with SPSS. RESULTS: Twenty-seven (14 NPs and 13 physicians) of 36 (75%) recipients responded. 22% (6) returned personal digital assistant (PDA) surveys. Respondents reported print, health professionals, and online/electronic resources as the most to least preferred means to access drug information, respectively. 37% and 35% of respondents reported using "both print and electronic but print more than electronic" and "print only", respectively, to search monograph-related drug information queries whereas 4% reported using "PDA only". Analysis of respondent ratings for all resources in the categories print, health professionals and other, and online/electronic resources, indicated that the Compendium of Pharmaceuticals and Specialties and pharmacists ranked highly for frequency of use, usefulness, accessibility, credibility, and current/timeliness by both groups of practitioners. Respondents' preferences and resource ratings were consistent with self-reported methods for conducting drug information queries. Few differences existed between NP and physician rankings of resources. CONCLUSION: The use of computers and PDAs remains limited, which is also consistent with preferred and frequent use of print resources. Education for these practitioners regarding available electronic drug information resources may facilitate future computer and PDA use. Further research is needed to determine methods to increase computer and PDA use and whether these technologies affect prescribing and patient outcomes