Adaptive design of a clinical decision support tool: What the impact on utilization rates means for future CDS research

© The Author(s) 2019. OBJECTIVE: We employed an agile, user-centered approach to the design of a clinical decision support tool in our prior integrated clinical prediction rule study, which achieved high adoption rates. To understand if applying this user-centered process to adapt clinical decision support tools is effective in improving the use of clinical prediction rules, we examined utilization rates of a clinical decision support tool adapted from the original integrated clinical prediction rule study tool to determine if applying this user-centered process to design yields enhanced utilization rates similar to the integrated clinical prediction rule study. MATERIALS & METHODS: We conducted pre-deployment usability testing and semi-structured group interviews at 6 months post-deployment with 75 providers at 14 intervention clinics across the two sites to collect user feedback. Qualitative data analysis is bifurcated into immediate and delayed stages; we reported on immediate-stage findings from real-time field notes used to generate a set of rapid, pragmatic recommendations for iterative refinement. Monthly utilization rates were calculated and examined over 12 months. RESULTS: We hypothesized a well-validated, user-centered clinical decision support tool would lead to relatively high adoption rates. Then 6 months post-deployment, integrated clinical prediction rule study tool utilization rates were substantially lower than anticipated based on the original integrated clinical prediction rule study trial (68%) at 17% (Health System A) and 5% (Health System B). User feedback at 6 months resulted in recommendations for tool refinement, which were incorporated when possible into tool design; however, utilization rates at 12 months post-deployment remained low at 14% and 4% respectively. DISCUSSION: Although valuable, findings demonstrate the limitations of a user-centered approach given the complexity of clinical decision support. CONCLUSION: Strategies for addressing persistent external factors impacting clinical decision support adoption should be considered in addition to the user-centered design and implementation of clinical decision support

Design of a tablet computer app for facilitation of a molecular blood culture test in clinical microbiology and preliminary usability evaluation

BACKGROUND: User mobility is an important aspect of the development of clinical information systems for health care professionals. Mobile phones and tablet computers have obtained widespread use by health care professionals, offering an opportunity for supporting the access to patient information through specialized applications (apps) while supporting the mobility of the users. The use of apps for mobile phones and tablet computers may support workflow of complex tasks, for example, molecular-based diagnostic tests in clinical microbiology. Multiplex Blood Culture Test (MuxBCT) is a molecular-based diagnostic test used for rapid identification of pathogens in positive blood cultures. To facilitate the workflow of the MuxBCT, a specialized tablet computer app was developed as an accessory to the diagnostic test. The app aims to reduce the complexity of the test by step-by-step guidance of microscopy and to assist users in reaching an exact bacterial or fungal diagnosis based on blood specimen observations and controls. Additionally, the app allows for entry of test results, and communication thereof to the laboratory information system (LIS). OBJECTIVE: The objective of the study was to describe the design considerations of the MuxBCT app and the results of a preliminary usability evaluation. METHODS: The MuxBCT tablet app was developed and set up for use in a clinical microbiology laboratory. A near-live simulation study was conducted in the clinical microbiology laboratory to evaluate the usability of the MuxBCT app. The study was designed to achieve a high degree of realism as participants carried out a scenario representing the context of use for the MuxBCT app. As the MuxBCT was under development, the scenario involved the use of molecular blood culture tests similar to the MuxBCT for identification of microorganisms from positive blood culture samples. The study participants were observed, and their interactions with the app were recorded. After the study, the participants were debriefed to clarify observations. RESULTS: Four medical laboratory technicians, for example, representative of end users of the app, participated in the clinical simulation study. Using the MuxBCT app, the study participants successfully identified and reported all microorganisms from the positive blood cultures examined. Three of the four participants reported that they found the app useful, while one study participant reported that she would prefer to make notes on paper and later enter them into the LIS. CONCLUSIONS: The preliminary usability evaluation results indicate that use of the MuxBCT tablet app can facilitate the workflow of the MuxBCT diagnostic test

Live usability testing of two complex clinical decision support tools: Observational study

© 2019 Journal of Medical Internet Research. All rights reserved. Background: Potential of the electronic health records (EHR) and clinical decision support (CDS) systems to improve the practice of medicine has been tempered by poor design and the resulting burden they place on providers. CDS is rarely tested in the real clinical environment. As a result, many tools are hard to use, placing strain on providers and resulting in low adoption rates. The existing CDS usability literature relies primarily on expert opinion and provider feedback via survey. This is the first study to evaluate CDS usability and the provider-computer-patient interaction with complex CDS in the real clinical environment. Objective: This study aimed to further understand the barriers and facilitators of meaningful CDS usage within a real clinical context. Methods: This qualitative observational study was conducted with 3 primary care providers during 6 patient care sessions. In patients with the chief complaint of sore throat, a CDS tool built with the Centor Score was used to stratify the risk of group A Streptococcus pharyngitis. In patients with a chief complaint of cough or upper respiratory tract infection, a CDS tool built with the Heckerling Rule was used to stratify the risk of pneumonia. During usability testing, all human-computer interactions, including audio and continuous screen capture, were recorded using the Camtasia software. Participants\u27 comments and interactions with the tool during clinical sessions and participant comments during a postsession brief interview were placed into coding categories and analyzed for generalizable themes. Results: In the 6 encounters observed, primary care providers toggled between addressing either the computer or the patient during the visit. Minimal time was spent listening to the patient without engaging the EHR. Participants mostly used the CDS tool with the patient, asking questions to populate the calculator and discussing the results of the risk assessment; they reported the ability to do this as the major benefit of the tool. All providers were interrupted during their use of the CDS tool by the need to refer to other sections of the chart. In half of the visits, patients\u27 clinical symptoms challenged the applicability of the tool to calculate the risk of bacterial infection. Primary care providers rarely used the incorporated incentives for CDS usage, including progress notes and patient instructions. Conclusions: Live usability testing of these CDS tools generated insights about their role in the patient-provider interaction. CDS may contribute to the interaction by being simultaneously viewed by the provider and patient. CDS can improve usability and lessen the strain it places on providers by being short, flexible, and customizable to unique provider workflow. A useful component of CDS is being as widely applicable as possible and ensuring that its functions represent the fastest way to perform a particular task

User-centred design of a clinical decision support system for palliative care: Insights from healthcare professionals

[EN] Objective:Although clinical decision support systems (CDSS) have many benefits for clinical practice, they also have several barriers to their acceptance by professionals. Our objective in this study was to design and validate The Aleph palliative care (PC) CDSS through a user-centred method, considering the predictions of the artificial intelligence (AI) core, usability and user experience (UX). Methods:We performed two rounds of individual evaluation sessions with potential users. Each session included a model evaluation, a task test and a usability and UX assessment. Results:The machine learning (ML) predictive models outperformed the participants in the three predictive tasks. System Usability Scale (SUS) reported 62.7 +/- 14.1 and 65 +/- 26.2 on a 100-point rating scale for both rounds, respectively, while User Experience Questionnaire - Short Version (UEQ-S) scores were 1.42 and 1.5 on the -3 to 3 scale. Conclusions:The think-aloud method and including the UX dimension helped us to identify most of the workflow implementation issues. The system has good UX hedonic qualities; participants were interested in the tool and responded positively to it. Performance regarding usability was modest but acceptable.The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the InAdvance project (H2020-SC1-BHC-2018¿2020 grant number 825750) and the CANCERLESS project (H2020-SC1-2020-Single-Stage-RTD grant number 965351), both funded by the European Union¿s Horizon 2020 research and innovation programme. Also, it was partially supported by the ALBATROSS project (National Plan for Scientific and Technical Research and Innovation 2017¿ 2020, grant number PID2019-104978RB-I00)Blanes-Selva, V.; Asensio-Cuesta, S.; Doñate-Martínez, A.; Pereira Mesquita, F.; Garcia-Gomez, JM. (2023). User-centred design of a clinical decision support system for palliative care: Insights from healthcare professionals. Digital Health. 9:1-13. https://doi.org/10.1177/20552076221150735113

Combining multivariate statistics and the think-aloud protocol to assess Human-Computer Interaction barriers in symptom checkers

[EN] Symptom checkers are software tools that allow users to submit a set of symptoms and receive advice related to them in the form of a diagnosis list, health information or triage. The heterogeneity of their potential users and the number of different components in their user interfaces can make testing with end-users unaffordable. We designed and executed a two-phase method to test the respiratory diseases module of the symptom checker Erdusyk. Phase I consisted of an online test with a large sample of users (n = 53). In Phase I, users evaluated the system remotely and completed a questionnaire based on the Technology Acceptance Model. Principal Component Analysis was used to correlate each section of the interface with the questionnaire responses, thus identifying which areas of the user interface presented significant contributions to the technology acceptance. In the second phase, the think-aloud procedure was executed with a small number of samples (n = 15), focusing on the areas with significant contributions to analyze the reasons for such contributions. Our method was used effectively to optimize the testing of symptom checker user interfaces. The method allowed kept the cost of testing at reasonable levels by restricting the use of the think-aloud procedure while still assuring a high amount of coverage. The main barriers detected in Erdusyk were related to problems understanding time repetition patterns, the selection of levels in scales to record intensities, navigation, the quantification of some symptom attributes, and the characteristics of the symptoms. (C) 2017 Elsevier Inc. All rights reserved.This work was supported by Helse Nord [grant HST1121-13], the Faculty of Health Sciences from UIT The Arctic University of Norway [researcher code 1108], and The Research Council of Norway [grant 248150/O70]. We thank Professor Emeritus Rafael Romero-Villafranca for reviewing the statistical analysis of this paper.Marco-Ruiz, L.; Bones, E.; De La Asuncion, E.; Gabarron, E.; Aviles-Solis, JC.; Lee, E.; Traver Salcedo, V.... (2017). Combining multivariate statistics and the think-aloud protocol to assess Human-Computer Interaction barriers in symptom checkers. Journal of Biomedical Informatics. 74:104-122. https://doi.org/10.1016/j.jbi.2017.09.002S1041227

Usability Textual Data Analysis: A Formulaic Coding Think-Aloud Protocol Method for Usability Evaluation

Think-aloud protocols are among the most standard methods for usability evaluation, which help to discover usability problems and to examine improvements because they provide direct information on a user's thinking and cognitive processes; however, it is often difficult to determine how to analyze the data to identify usability problems because there is no formulaic analysis procedure for textual data. Therefore, the analysis is time-consuming, and the quality of the results varies depending on an analyst's skills. In the present study, the author proposes a formulaic analysis think-aloud protocol method that specifies the procedure for analyzing participants' verbal responses during usability tests. The aim of the proposed think-aloud protocol method was to deliver an explicit procedure using step coding (SCAT) and 70 design items for textual data analysis, and then, the method was applied to a case study of usability evaluation to confirm that the method could extract the target system's problems. By using step coding and 70 design items, the process of extracting usability problems from textual data was made explicit, and the problems were extracted analytically. In other words, the proposed method was less ambiguous. Once a formulaic analysis procedure was established, textual data analysis could be performed easily and efficiently. The analysis could be performed without hesitation after data acquisition, and there were fewer omissions. In addition, it is expected that the procedure would be easy to use, even for novice designers

Measures of User experience in a Streptococcal pharyngitis and Pneumonia Clinical Decision Support Tools

Objective: To understand clinician adoption of CDS tools as this may provide important insights for the implementation and dissemination of future CDS tools. Materials and Methods: Clinicians (n=168) at a large academic center were randomized into intervention and control arms to assess the impact of strep and pneumonia CDS tools. Intervention arm data were analyzed to examine provider adoption and clinical workflow. Electronic health record data were collected on trigger location, the use of each component and whether an antibiotic, other medication or test was ordered. Frequencies were tabulated and regression analyses were used to determine the association of tool component use and physician orders. Results: The CDS tool was triggered 586 times over the study period. Diagnosis was the most frequent workflow trigger of the CDS tool (57%) as compared to chief complaint (30%) and diagnosis/antibiotic combinations (13%). Conversely, chief complaint was associated with the highest rate (83%) of triggers leading to an initiation of the CDS tool (opening the risk prediction calculator). Similar patterns were noted for initiation of the CDS bundled ordered set and completion of the entire CDS tool pathway. Completion of risk prediction and bundled order set components were associated with lower rates of antibiotic prescribing (OR 0.5; CI 0.2-1.2 and OR 0.5; CI 0.3-0.9, respectively). Discussion: Different CDS trigger points in the clinician user workflow lead to substantial variation in downstream use of the CDS tool components. These variations were important as they were associated with significant differences in antibiotic ordering. Conclusions: These results highlight the importance of workflow integration and flexibility for CDS success

Innovative Simulation of Health Care Services in the Usability Laboratory: Experiences from the Model for Telecare Alarm Services-project

In Norway, a recent health reform urged municipalities to prepare for telecare alarm services to handle alarms associated to welfare technology and telecare technology in citizens’ homes. That requires a re-organisation of health and social services in many municipalities and several are preparing to establish new telecare alarm services operated in inter-municipal response centres. In this context, the research project “Model for Telecare Alarm Services” aims to study how existing telecare alarm services in Norwegian municipalities are organised and operated, and identify critical factors when designing new models for future services. This paper presents how an innovative simulation of health care services was used in the research project, when key informants from several municipalities, research partners and industry tested different models of telecare alarm services in a usability laboratory. The lessons learned by the research group showed that laboratory simulation was an efficient way of testing different scenarios of new telecare service models, together with key informants from heterogenous end-user groups.publishedVersionNivå