Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system

Objectives: To assess the effectiveness of two usability evaluation methods, cognitive walkthrough (CW) and think aloud (TA), for identifying usability problems and to compare the performance of CW and TA in identifying different types of usability problems. Methods: A CWwas performed by two usability evaluators and 10 physicians were recruited to perform a TA usability testing of a CPOE system (Medicator). The severity of identified usability problems was determined and the usability problems were categorized based on the User Action Framework (UAF). The potential of usability problems to cause medication errors was also determined. The thoroughness, validity and effectiveness of the two methods were compared. Results: Fifty seven unique usability problems of different severity, spread over the four phases of interaction as defined by the UAF, were identified. The effectiveness of the TA method for identifying usability problemswas 0.08 higher than that of theCW(0.70 vs. 0.62). The thoroughness (the extent to which a method can identify existing usability problems) of the TA was higher for the “Planning” and “Assessment” phases and lower for the “Translation” phase (as defined by UAF). The thoroughness of TA for identifying problems that may potentially result in medication errors was higher than that of CW (0.81 vs. 0.68). The number of usability problems identified by each of the methods was significantly less than the total number of detected real usability problems in Medicator (p < 0.001). The observed differences between the number of real usability problems identified by CW and TA (38 vs. 41), the difference between the average severity of the detected problems by CW and TA (2.37 vs. 2.41), and the difference for identifying problems potentially resulting in medication errors (15 vs. 18) were not statistically significant (p > 0.4). Conclusions: This study shows that although TA showed a slightly better effectiveness, there is no significant difference between the performance of the CW and the TA methods in terms of number of usability problems identified and the mean severity of these problems. Since no single evaluation method will uncover all of the usability problems a combination of methods is advised as the most appropriate approach, especially if usability problems can lead to potentially fatal outcomes

The Web-Based Usability Heuristic Survey Supports User Satisfaction

Competition between companies is now getting stronger strongly related to the company's mechanism in providing the services provided. Companies can build ways that can maintain customer loyalty. The approach can be done with web-based Usability Testing adopting an application user acceptance model, a usability aspect analysis that acts as a customer who enjoys a company's products and services. At present, there are already many theories of user acceptance models for an application. One of them is the user satisfaction model (User Satisfaction). This review specifically also identifies that content content has a significant positive effect on website visitor satisfaction

Using Think Aloud Protocols to Assess E-Prescribing in Community Pharmacies

Introduction: Think aloud protocol has rarely been used as a method of data collection in community pharmacies.Purpose: The aim of the report is to describe how think aloud protocols were used to identify issues that arise when using e-prescribing technology in pharmacies. In this paper, we report on the benefits and challenges of using think aloud protocols in pharmacies to examine the use of e-prescribing systems.Methods: Sixteen pharmacists and pharmacy technicians were recruited from seven community pharmacies in Wisconsin. Data were collected using direct observation alongside think aloud protocol. Direct observations and think aloud protocols took place between January-February, 2011. Participants were asked to verbalize their thoughts as they process electronic prescriptions.Results: Participants identified weaknesses in e-prescribing that they had previously not conceived. This created heightened awareness for vigilance when processing e-prescriptions. The main challenge with using think aloud protocols was due to interruptions in the pharmacies. Also, a few participants found it challenging to remember to continue verbalizing their thought process during think aloud sessions.Conclusion: The use of think aloud protocols as method of data collection is a new way for understanding the issues related to technology use in community pharmacy practice. Think aloud protocol was beneficial in providing objective information on e-prescribing use not solely based on pharmacist’s or technician’s opinion of the technology. This method provided detailed information on a wide variety of real time challenges with e-prescribing technology use in community pharmacies. Using this data collection method can help identify potential patient safety issues when using e-prescribing and suggestions for redesign

When to ask participants to think aloud:A comparative study of concurrent and retrospective think-aloud methods

This paper presents the results of a study that compared two think-aloud usability testing methods: the concurrent think-aloud and the retrospective think-aloud methods. Data from task performance, testing experience, and usability problems were collected from 30 participants equally distributed between the two think-aloud conditions. The results suggest that while the thinking aloud method had no impact on task performance and testing experience, participants using the concurrent think-aloud method reported a larger number of problems with the test interface than participants using the retrospective think-aloud method. These findings suggest a reason for preferring the concurrent think-aloud method to the retrospective one

Usability Assessment of an EHR’s Clinical Notes Interface from Attendings and Residents Vision: An Exploratory Study

Background: Usability gaps between current and future improved Electronic Health Record (EHR) system designs exist due to insufficient incorporation of User-Centered Design (UCD) principles during System Development Life Cycle (SDLC).Objectives: To evaluate the usability aspects of a commonly used EHR system specific to clinical notes usage from attendings’ and residents’ standpoints by analyzing objective measures of users’ performance and their subjective perceptions employing mixed methods approach.Methods: Usability of a commercial, inpatient EHR clinical notes documentation interface was analyzed from standpoints of two provider groups employing two standardized patient cases. Both objective and subjective data were collected from attending (n=6) and resident physicians (n=8) through usability testing employing a mixed method approach.Results: The study results suggested that (i) EHR usability and desirability is influenced by user characteristics, (ii) workloads associated with H&amp;P and progress notes writing are perceived differently between two groups, (iii) repeated task performance improves user efficiency and (iv) user performance is correlated to their subjective system assessments.Conclusion: Understanding usability of clinical documentation interface from perspectives of two different user groups, provides interface designers with an opportunity to develop an EHR system centered on UCD principles

A Heuristic Evaluation of Iraq E-Portal

The integration of Information Technology (IT) and Communication Technology (CT) creates Information and Communication Technologies (ICTs), which has changed the citizens’ lives and the manner they interact. Therefore, the governments around the world took serious steps to adopt and implement ICT to provide the best services to their citizens. However, these governments have focused only on adoption and implementation of these new technologies and not so much on their usability aspect. This paper aims to investigate the usability of e-portal in Iraq based on the citizens’ standpoint. The e-portal in Iraq has seen a decline of the number of citizens who visit the portal to conduct transactions electronically, which is due to the improper design that does not attract the citizens and stakeholders to browse and use it. Moreover, this study is based on a heuristic evaluation, which aims to capture citizens’ standpoint of usability on this e-portal. The results indicated that out of 10 principles, 5 and 7 were inappropriate design

Effect of user sessions on the heuristic usability method

Heuristic evaluation (HE) is a widely used method for assessing software systems. Several studies have sought to improve the effectiveness of HE by developing its heuristics and procedures. However, few studies have involved the end-user, and to the best of the authors' knowledge, no HE studies involving end-users with non-expert evaluators have been reported. Therefore, the aim of this study is to investigate the impact of end-users on the results obtained by a non-expert evaluator within the HE process, and through that, to explore the number of usability problems and their severity. This article proposes introducing two sessions within the HE process: a user exploration session (UES-HE) and a user review session (URS-HE). The outcomes are compared with two solid benchmarks in the usability-engineering field: the traditional HE and the usability testing (UT) methods. The findings show that the end-user has a significant impact on non-expert evaluator results in both sessions. In the UES-HE method, the results outperformed all usability evaluation methods (UEMs) regarding the usability problems identified, and it tended to identify more major, minor, and cosmetic problems than other methods.</jats:p

Assessing the reliability, validity and acceptance of a classification scheme of usability problems (CUP)

Post-print (lokagerð höfunda)The aim of this study was to evaluate the Classification of Usability Problems (CUP) scheme. The goal of CUP is to classify usability problems further to give user interface developers better feedback to improve their understanding of usability problems, help them manage usability maintenance, enable them to find effective fixes for UP, and prevent such problems from reoccurring in the future. First, reliability was evaluated with raters of different levels of expertise and experience in using CUP. Second, acceptability was assessed with a questionnaire. Third, validity was assessed by developers in two field studies. An analytical comparison was also made to three other classification schemes. CUP reliability results indicated that the expertise and experience of raters are critical factors for assessing reliability consistently, especially for the more complex attributes. Validity analysis results showed that tools used by developers must be tailored to their working framework, knowledge and maturity. The acceptability study showed that practitioners are concerned with the effort spent in applying any tool. To understand developers’ work and the implications of this study two theories are presented for understanding and prioritising UP. For applying classification schemes, the implications of this study are that training and context are needed.Peer Reviewe

Measuring Learnability in Human-Computer Interaction

It is well accepted that learnability is a crucial attribute of usability that should be considered in almost every software system. A good learnability leads within a short time and with minimal effort to a high level of proficiency of the user. Therefore, expensive training time of complex systems is reduced. However, there is only few consensus on how to define and evaluate learnability. In addition, gathering detailed information on learnability is quite difficult. In todays books on usability evaluation, learnability gets only few attention, research publications are spread to several other fields and the term learnability is also used in other context. The objective of this thesis is to give an structured overview of learnability and methods for evaluation and additionally assist in the evaluator’s individual choice of an appropriate method. First of all, several definitions of learnability are discussed. For a deeper understanding psychological background knowledge is provided. Afterwards, methods to asses learnability are presented. This comprises nine methods that seem particularly appropriate to measure learnability. As this methods are very diverse, a framework based on analytical hierarchy process is provided. This framework aims to classify presented methods with respect to certain criteria and assess practitioners in selecting an appropriate method to measure learnability