Comparison of a prototype for indications-based prescribing with 2 commercial prescribing systems

Importance: The indication (reason for use) for a medication is rarely included on prescriptions despite repeated recommendations to do so. One barrier has been the way existing electronic prescribing systems have been designed. Objective: To evaluate, in comparison with the prescribing modules of 2 leading electronic health record prescribing systems, the efficiency, error rate, and satisfaction with a new computerized provider order entry prototype for the outpatient setting that allows clinicians to initiate prescribing using the indication. Design, Setting, and Participants: This quality improvement study used usability tests requiring internal medicine physicians, residents, and physician assistants to enter prescriptions electronically, including indication, for 8 clinical scenarios. The tool order assignments were randomized and prescribers were asked to use the prototype for 4 of the scenarios and their usual system for the other 4. Time on task, number of clicks, and order details were captured. User satisfaction was measured using posttask ratings and a validated system usability scale. The study participants practiced in 2 health systems\u27 outpatient practices. Usability tests were conducted between April and October of 2017. Main Outcomes and Measures: Usability (efficiency, error rate, and satisfaction) of indications-based computerized provider order entry prototype vs the electronic prescribing interface of 2 electronic health record vendors. Results: Thirty-two participants (17 attending physicians, 13 residents, and 2 physician assistants) used the prototype to complete 256 usability test scenarios. The mean (SD) time on task was 1.78 (1.17) minutes. For the 20 participants who used vendor 1\u27s system, it took a mean (SD) of 3.37 (1.90) minutes to complete a prescription, and for the 12 participants using vendor 2\u27s system, it took a mean (SD) of 2.93 (1.52) minutes. Across all scenarios, when comparing number of clicks, for those participants using the prototype and vendor 1, there was a statistically significant difference from the mean (SD) number of clicks needed (18.39 [12.62] vs 46.50 [27.29]; difference, 28.11; 95% CI, 21.47-34.75; P \u3c .001). For those using the prototype and vendor 2, there was also a statistically significant difference in number of clicks (20.10 [11.52] vs 38.25 [19.77]; difference, 18.14; 95% CI, 11.59-24.70; P \u3c .001). A blinded review of the order details revealed medication errors (eg, drug-allergy interactions) in 38 of 128 prescribing sessions using a vendor system vs 7 of 128 with the prototype. Conclusions and Relevance: Reengineering prescribing to start with the drug indication allowed indications to be captured in an easy and useful way, which may be associated with saved time and effort, reduced medication errors, and increased clinician satisfaction

Comparison of a prototype for indications-based prescribing with 2 commercial prescribing systems

Importance: The indication (reason for use) for a medication is rarely included on prescriptions despite repeated recommendations to do so. One barrier has been the way existing electronic prescribing systems have been designed. Objective: To evaluate, in comparison with the prescribing modules of 2 leading electronic health record prescribing systems, the efficiency, error rate, and satisfaction with a new computerized provider order entry prototype for the outpatient setting that allows clinicians to initiate prescribing using the indication. Design, Setting, and Participants: This quality improvement study used usability tests requiring internal medicine physicians, residents, and physician assistants to enter prescriptions electronically, including indication, for 8 clinical scenarios. The tool order assignments were randomized and prescribers were asked to use the prototype for 4 of the scenarios and their usual system for the other 4. Time on task, number of clicks, and order details were captured. User satisfaction was measured using posttask ratings and a validated system usability scale. The study participants practiced in 2 health systems\u27 outpatient practices. Usability tests were conducted between April and October of 2017. Main Outcomes and Measures: Usability (efficiency, error rate, and satisfaction) of indications-based computerized provider order entry prototype vs the electronic prescribing interface of 2 electronic health record vendors. Results: Thirty-two participants (17 attending physicians, 13 residents, and 2 physician assistants) used the prototype to complete 256 usability test scenarios. The mean (SD) time on task was 1.78 (1.17) minutes. For the 20 participants who used vendor 1\u27s system, it took a mean (SD) of 3.37 (1.90) minutes to complete a prescription, and for the 12 participants using vendor 2\u27s system, it took a mean (SD) of 2.93 (1.52) minutes. Across all scenarios, when comparing number of clicks, for those participants using the prototype and vendor 1, there was a statistically significant difference from the mean (SD) number of clicks needed (18.39 [12.62] vs 46.50 [27.29]; difference, 28.11; 95% CI, 21.47-34.75; P \u3c .001). For those using the prototype and vendor 2, there was also a statistically significant difference in number of clicks (20.10 [11.52] vs 38.25 [19.77]; difference, 18.14; 95% CI, 11.59-24.70; P \u3c .001). A blinded review of the order details revealed medication errors (eg, drug-allergy interactions) in 38 of 128 prescribing sessions using a vendor system vs 7 of 128 with the prototype. Conclusions and Relevance: Reengineering prescribing to start with the drug indication allowed indications to be captured in an easy and useful way, which may be associated with saved time and effort, reduced medication errors, and increased clinician satisfaction

Designing Clinical Data Presentation Using Cognitive Task Analysis Methods

Despite the many decades of research on effective use of clinical systems in medicine, the adoption of health information technology to improve patient care continues to be slow especially in ambulatory settings. This applies to dentistry as well, a primary care discipline with approximately 137,000 practicing dentists in the United States. One critical reason is the poor usability of clinical systems, which makes it difficult for providers to navigate through the system and obtain an integrated view of patient data during patient care. Cognitive science methods have shown significant promise to meaningfully inform and formulate the design, development and assessment of clinical information systems. Most of these methods were applied to evaluate the design of systems after they have been developed. Very few studies, on the other hand, have used cognitive engineering methods to inform the design process for a system itself. It is this gap in knowledge – how cognitive engineering methods can be optimally applied to inform the system design process – that this research seeks to address through this project proposal. This project examined the cognitive processes and information management strategies used by dentists during a typical patient exam and used the results to inform the design of an electronic dental record interface. The resulting 'proof of concept' was evaluated to determine the effectiveness and efficiency of such a cognitively engineered and application flow design. The results of this study contribute to designing clinical systems that provide clinicians with better cognitive support during patient care. Such a system will contribute to enhancing the quality and safety of patient care, and potentially to reducing healthcare costs

Perception gaps and the adoption of information technology in the clinical healthcare environment

Implementation of information systems has lagged in many areas of clinical healthcare for a variety of reasons. Economics, data complexity and resistance are among the often quoted roadblocks. Research suggests that physicians play a major part in the adoption, use and diffusion of information technology (IT) in clinical settings. There are also other healthcare professionals, clinical and non-clinical, who play important roles in making decisions about the acquisition of information technology. In addition to these groups there are information technology professionals providing the services required within the healthcare field. Finally within this group are those IT professionals who have sufficient cross training to understand specific needs. Each member of these groups brings a different perspective to both needs assessments as well as implementation of clinical systems. This study considers the idea that there are preconceived differences of opinion of the information needs of clinical healthcare by the clinical community and the information technology professionals. Are these differences significant enough to create a barrier to implementation? A questionnaire was developed from preliminary data to assess multiple parameters which could impact implementation of a clinical information technology solution. A Web of System Performance (WOSP) model was created to map each of the following eight areas of concern: functionality, usability, extendibility, connectivity, flexibility, reliability, privacy and security. Responses to the questions were related to professional roles, age and experience. There were no differences seen in the perceived need for secure systems by either healthcare workers or IT professionals. The variance of perceived need was greatest among the various non-physician healthcare workers when compared to physicians or information technology professions. This was a consistent pattern for the otherparameters with the exception of the usability of the electronic health record. In this area all groups disagreed significantly. The study, though limited by its small sample, still suggests that the resistance by healthcare professionals is not a significant barrier to successful information technology implementation

Systematic Heuristic Evaluation of Computerized Consultation Order Templates: Clinicians’ and Human Factors Engineers’ Perspectives

We assessed the usability of consultation order templates and identified problems to prioritize in design efforts for improving referral communication. With a sample of 26 consultation order templates, three evaluators performed a usability heuristic evaluation. The evaluation used 14 domain-independent heuristics and the following three supplemental references: 1 new domain-specific heuristic, 6 usability goals, and coded clinicians’ statements regarding ease of use for 10 sampled templates. Evaluators found 201 violations, a mean of 7.7 violations per template. Minor violations outnumbered major violations almost twofold, 115 (57%) to 62 (31%). Approximately 68% of violations were linked to 5 heuristics: aesthetic and minimalist design (17%), error prevention (16%), consistency and standards (14%), recognition rather than recall (11%), and meet referrers’ information needs (10%). Severe violations were attributed mostly to meet referrers’ information needs and recognition rather than recall. Recorded violations yielded potential negative consequences for efficiency, effectiveness, safety, learnability, and utility. Evaluators and clinicians demonstrated 80% agreement in usability assessment. Based on frequency and severity of usability heuristic violations, the consultation order templates reviewed may impede clinical efficiency and risk patient safety. Results support the following design considerations: communicate consultants’ requirements, facilitate information seeking, and support communication. While the most frequent heuristic violations involved interaction design and presentation, the most severe violations lacked information desired by referring clinicians. Violations related to templates’ inability to support referring clinicians’ information needs had the greatest potential negative impact on efficiency and safety usability goals. Heuristics should be prioritized in future design efforts

Master of Science

thesisThe Residency Review Committee (RRC) requires that general surgery residents document their Surgical Intensive Care Unit (SICU) experiences. To satisfy these requirements we created a web based intranet log to make it easier for residents to track their patients and determine when these requirements were complete. A premium was put on usability to promote acceptance by surgical residents. A prototype web site was designed with input from an attending general surgeon. Three general surgery residents were selected to participate in the iterative design phase. They went through three iterations using a "think-aloud" method while performing tasks on the prototype web site. Each iteration led to improvements to the web site. In a comparison test, a group of seven medical students performed 14 typical web site tasks using both the prototype and the final versions. They were asked to complete a Questionnaire for User Interaction Satisfaction (QUIS) for each version. The time for completion of these tasks was also recorded. The user interaction satisfaction did not show any improvement (F(1,6)=0.13, p=0.912). Similarly, there was no improvement in times for delete and add tasks ( Delete F(1,5) = 0.949, p=0.375, Add F(1,5)=0.267, p=0.628 ); however, the time to complete edit tasks was faster for the final version of the web site (F (1,5)= 14.3, p=0.013). The primary reason for not detecting other differences between the two web sites is likely that the comparison study did not have sufficient power. This was suggested by the participants whose comments favored the final version over the prototype as well as a trend of consistently higher mean subset scores in the final version. The results indicate that differences may be seen when more complex tasks are completed (editing information) versus the two simpler tasks (adding or deleting a patient record in a web site). Future studies should focus on the impact of navigation strategies on speed and data warehouse approaches to creating the application. This study shows the benefits of using an iterative design approach to create a usable web site and demonstrates the importance of further research in the field of usability

Improving Data Quality in Primary Care: Modelling, Measurement, and the Design of Interventions

In an era where governments around the world invest heavily in data collection and data management, poor-quality data is expensive and has many direct and indirect costs. While there are different types of data quality challenges, some of the more complex data quality problems depend on the design and production processes involved in generating data. Therefore, it is important to design systems that support better data quality. This involves understanding what quality means in a specific context, understanding how it can be measured, and identifying ways to encourage better data quality behaviours. Healthcare is not immune to the challenges of data quality and can be classified as a complex socio-technical system by virtue of its characteristics. As such, the study of healthcare data quality and its improvement is well suited for the domain of systems design and human factors engineering. Cognitive Work Analysis (CWA) is especially well suited for this task, as it can be used to better understand the context and workflow of users in complex socio-technical domains. It is a conceptual framework that facilitates the analysis of factors that shape human-information interaction and has been used in healthcare for over 20 years. The approach is work-centred, rather than user-centred, and it analyses the constraints and goals that shape information behaviour in the work environment. I used CWA as a framework to help me analyse the problem of data quality in healthcare. My research uses an instrumental case study approach to understand data quality in primary care. My goal was to answer three questions: In primary care, how are individual users influenced by their environment to input high-quality data? What techniques could be used to design systems that persuade users to enter higher-quality data? Is it possible to improve data quality in primary care by persuading users with the user interface of information systems in these complex socio-technical systems? The scope of work included modelling data quality, defining and measuring data quality in a primary care system, establishing design concepts that could improve data quality through persuasion, and testing the viability of some design concepts. I began analysing this problem by creating an abstraction hierarchy of patient treatment with medical records. This model can be used to represent patient treatment from a primary care perspective. The model helped explain the patient treatment ecosystem and how data is generated through patient encounters. After creating my model to represent patient treatment, I incorporated it into two CWAs of data quality and data codification. The first model represented codification in the primary care ecosystem, whereas the second model represented codification in community hospitals. After developing abstraction hierarchies for both domains, I analysed similar tasks from each system with control task analysis, strategies analysis, and worker competencies analysis. The tasks that I analysed related specifically to data codification: in primary care, I modelled the record encounter task performed by clinicians at a Family Health Team (FHT), and in the community hospital, I modelled the abstract task performed by health information management professionals. I used the same record encounter task at the FHT as a continuing focus of my case study. I used both models of codification to perform a comparison. My goal was to identify the differences between the ecosystems and tasks that were present in primary care and the community hospital. Comparing CWA models is not a well-defined process in the literature, and I developed an approach to conduct this comparison based on seminal works. I used the approach to systematically compare each phase of my CWA models. I found that the analysis of both system domains in parallel enabled a richer understanding of each environment that may not have been achieved independently. In addition, I discovered that a rich environment exists around data codification processes, and this context influences and distinguishes the actions of users. While the tasks in both domains were seemingly similar, they took place with different priorities and required different competencies. After building and comparing models, I investigated the summarizing task in primary care more closely by analysing data within a FHT’s reporting database. The goal of this study was to understand data quality tradeoffs between timeliness, validity, completeness, and use in primary care users. Data quality measures and metrics were developed through interviews with a focus group of managers. After analysing data quality measures for 196,967 patient encounters, I created baselines, modelled each measure with logit binomial regression to show correlations, characterized tradeoffs, and investigated data quality interactions. Based on the analysis, I found a positive relationship between validity and completeness, and a negative relationship between timeliness and use. Use of data and reductions in entry delay were positively associated with completeness and validity. These results suggested that if users are not provided with sufficient time to record data as part of their regular workflow, they will prioritize their time to spend more time with patients. As a measurement of the effectiveness of a system, the negative correlation between use and timeliness points to a self-reinforcing data repository that provides users with little external value. These findings were consistent with the modelling work and also provided useful insight to study data quality improvements within the system. I used my measures from the data analysis to select design priorities and behaviour changes that should, according to my ongoing case study, improve data quality. Then I developed several design concepts by combining CWA, a framework for behaviour change, and a design framework for persuasive systems. The design concepts adopted different persuasion principles to change specific behaviours. To test the validity of my design concepts, I worked with a FHT to implement some of my proposed interventions during a field study. This involved the introduction of a non-invasive summary screen into the user workflow. After the summary screen had been deployed for eight weeks, I received secondary data from the FHT to analyse. First, I performed a pre-post measurement of several data quality measures by doing a simple paired t-test. To further understand the results, I borrowed from healthcare quality improvement methodologies and used statistical process control charts to understand the overall context of the measures. The average delay per entry was reduced by 3.35 days, and the percentage of same-day entries increased by 10.3%. The number of records that were complete dropped by 4.8%. Changes to entry accuracy and report generation were not significant. Several additional insights could be extracted by looking at each the XmR chart for each variable and discussing the trends with the FHT. Feedback was also collected from users through an online survey. Through the use of a case study spanning several years, I was able to reach the following conclusions: data codification and data quality are manufactured within complex socio-technical systems and users are heavily influenced by a variety of factors within their ecosystem; persuasive design, informed with data from a CWA, is an effective technique for creating ecologically relevant persuasive designs; and data quality in primary care can be improved through the use of these designs in the system’s user interface. There are interesting opportunities to apply the results of my work to other jurisdictions. A strength of this work lies in its usefulness for international readers to draw comparisons between different systems and health care environments throughout the world

Understanding Providers’ Interaction with Graphical User Interface Pertaining to Clinical Document Usage in an Electronic Health Record System

University of Minnesota Ph.D. dissertation. 2017. Major: Health Informatics. Advisors: Genevieve Melton-Meaux, Terrence Adam. 1 computer file (PDF); 110 pages.Despite high Electronic Health Record (EHR) system adoption rates by hospital and office-based practices, many users remain highly dissatisfied with the current state of EHRs. Sub-optimal EHR usability as a result of insufficient incorporation of User-Centered Design (UCD) approach during System Development Life Cycle process (SDLC) is considered as a leading factor for this observed discrepancy. The purpose of this study is to expand our knowledge around physicians’ interaction with in-patient EHRs around clinical documentation work flow processes by examining various tasks related to not entry and related information-seeking. The knowledge gained could be applied in tailoring physicians’ EHR training and designing a future EHR leading to more efficient and effective task performances, ultimately resulting in greater user satisfaction and enhanced EHR usability

A Multi-Method Evaluation Of A Guideline Based Clinical Decision Support Intervention On Provider Ordering Behavior, System Acceptance And Inter-Professional Communication

Background and aims: Unnecessary variation in the delivery of patient care is well documented in the medical literature; evidence-based clinical practice is critical for improving the quality of care. Clinical decision support systems (CDSS) are promising tools for improving the systematic integration of evidence into clinical practice. This study evaluated a CDSS in a domain of care that had not yet been explored—namely, decision support for venous catheter selection. This dissertation study aimed to (1) evaluate the effect of this CDSS on provider ordering behavior before and after implementation and explore the differential impact of this tool by provider type and service and (2) identify organizational, individual, usability, and workflow factors that impact CDSS acceptance by physicians and advanced practice nurses and to elicit information about the impact of this system on communication between providers and the nurse-led vascular access team. Methods: This was a multi-method study. Aim one was single group pre-post analysis of longitudinal data. Variables included those related to patient and provider level factors. The main analysis was conducted with linear regression models with random effects to account for clustering of data. We conducted semi-structured interviews for aim two and use conventional qualitative content analysis to identify themes. Results: We found mixed results in the impact of the CDSS on provider ordering behavior. While the CDSS did not have an impact on the number of venous catheters ordered, we saw a statistically and clinically significant decrease in the proportion of double lumen catheters ordered. Findings for the qualitative aim showed that the CDSS improved process efficiency and inter-professional communication. We found that it also facilitated education for evidence based practice for novice providers. Discussion: This dissertation study showed a clear impact of the CDSS on double lumen catheter ordering, which has implications for patient outcomes. Furthermore, we found impacts by provider type. Additional work is needed to evaluate this CDSS in other settings and to further assess differential impacts by provider type

A Task-based Support Architecture for Developing Point-of-care Clinical Decision Support Systems for the Emergency Department

Objectives: The purpose of this study was to create a task-based support architecture for developing clinical decision support systems (CDSSs) that assist physicians in making decisions at the point-of-care in the emergency department (ED). The backbone of the proposed architecture was established by a task-based emergency workflow model for a patient-physician encounter. Methods: The architecture was designed according to an agent-oriented paradigm. Specifically, we used the O-MaSE (Organization-based Multi-agent System Engineering) method that allows for iterative translation of functional requirements into architectural components (e.g., agents). The agent-oriented paradigm was extended with ontology-driven design to implement ontological models representing knowledge required by specific agents to operate. Results: The task-based architecture allows for the creation of a CDSS that is aligned with the task-based emergency workflow model. It facilitates decoupling of executable components (agents) from embedded domain knowledge (ontological models), thus supporting their interoperability, sharing, and reuse. The generic architecture was implemented as a pilot system, MET3-AE – a CDSS to help with the management of pediatric asthma exacerbation in the ED. The system was evaluated in a hospital ED. Conclusions: The architecture allows for the creation of a CDSS that integrates support for all tasks from the task-based emergency workflow model, and interacts with hospital information systems. Proposed architecture also allows for reusing and sharing system components and knowledge across disease-specific CDSSs