Methods of Usability Testing in the Development of eHealth Applications: A Scoping Review

Background The number of eHealth applications has exponentially increased in recent years, with over 325,000 health apps now available on all major app stores. This is in addition to other eHealth applications available on other platforms such as PC software, web sites and even gaming consoles. As with other digital applications, usability is one of the key factors in the successful implementation of eHealth apps. Reviews of the literature on empirical methods of usability testing in eHealth were last published in 2015. In the context of an exponentially increasing rate of App development year on year, an updated review is warranted. Objective To identify, explore, and summarize the current methods used in the usability testing of eHealth applications. Methods A scoping review was conducted on literature available from April 2014 up to October 2017. Four databases were searched. Literature was considered for inclusion if it was (1) focused on an eHealth application (which includes websites, PC software, smartphone and tablet applications), (2) provided information about usability of the application, (3) provided empirical results of the usability testing, (4) a full or short paper (not an abstract) published in English after March 2014. We then extracted data pertaining to the usability evaluation processes described in the selected studies. Results 133 articles met the inclusion criteria. The methods used for usability testing, in decreasing order of frequency were: questionnaires (n = 105), task completion (n = 57), ‘Think-Aloud’ (n = 45), interviews (n = 37), heuristic testing (n = 18) and focus groups (n = 13). Majority of the studies used one (n = 45) or two (n = 46) methods of testing. The rest used a combination of three (n = 30) or four (n = 12) methods of testing usability. None of the studies used automated mechanisms to test usability. The System Usability Scale (SUS) was the most frequently used questionnaire (n = 44). The ten most frequent health conditions or diseases where eHealth apps were being evaluated for usability were the following: mental health (n = 12), cancer (n = 10), nutrition (n = 10), child health (n = 9), diabetes (n = 9), telemedicine (n = 8), cardiovascular disease (n = 6), HIV (n = 4), health information systems (n = 4) and smoking (n = 4). Further iterations of the app were reported in a minority of the studies (n = 41). The use of the ‘Think-Aloud’ (Pearson Chi-squared test: χ2 = 11.15, p < 0.05) and heuristic walkthrough (Pearson Chi-squared test: χ2 = 4.48, p < 0.05) were significantly associated with at least one further iteration of the app being developed. Conclusion Although there has been an exponential increase in the number of eHealth apps, the number of studies that have been published that report the results of usability testing on these apps has not increased at an equivalent rate. The number of digital health applications that publish their usability evaluation results remains only a small fraction. Questionnaires are the most prevalent method of evaluating usability in eHealth applications, which provide an overall measure of usability but do not pinpoint the problems that need to be addressed. Qualitative methods may be more useful in this regard. The use of multiple evaluation methods has increased. Automated methods such as eye tracking have not gained traction in evaluating health apps. Further research is needed into which methods are best suited for the different types of eHealth applications, according to their target users and the health conditions being addressed.Background The number of eHealth applications has exponentially increased in recent years, with over 325,000 health apps now available on all major app stores. This is in addition to other eHealth applications available on other platforms such as PC software, web sites and even gaming consoles. As with other digital applications, usability is one of the key factors in the successful implementation of eHealth apps. Reviews of the literature on empirical methods of usability testing in eHealth were last published in 2015. In the context of an exponentially increasing rate of App development year on year, an updated review is warranted. Objective To identify, explore, and summarize the current methods used in the usability testing of eHealth applications. Methods A scoping review was conducted on literature available from April 2014 up to October 2017. Four databases were searched. Literature was considered for inclusion if it was (1) focused on an eHealth application (which includes websites, PC software, smartphone and tablet applications), (2) provided information about usability of the application, (3) provided empirical results of the usability testing, (4) a full or short paper (not an abstract) published in English after March 2014. We then extracted data pertaining to the usability evaluation processes described in the selected studies. Results 133 articles met the inclusion criteria. The methods used for usability testing, in decreasing order of frequency were: questionnaires (n = 105), task completion (n = 57), ‘Think-Aloud’ (n = 45), interviews (n = 37), heuristic testing (n = 18) and focus groups (n = 13). Majority of the studies used one (n = 45) or two (n = 46) methods of testing. The rest used a combination of three (n = 30) or four (n = 12) methods of testing usability. None of the studies used automated mechanisms to test usability. The System Usability Scale (SUS) was the most frequently used questionnaire (n = 44). The ten most frequent health conditions or diseases where eHealth apps were being evaluated for usability were the following: mental health (n = 12), cancer (n = 10), nutrition (n = 10), child health (n = 9), diabetes (n = 9), telemedicine (n = 8), cardiovascular disease (n = 6), HIV (n = 4), health information systems (n = 4) and smoking (n = 4). Further iterations of the app were reported in a minority of the studies (n = 41). The use of the ‘Think-Aloud’ (Pearson Chi-squared test: χ2 = 11.15, p < 0.05) and heuristic walkthrough (Pearson Chi-squared test: χ2 = 4.48, p < 0.05) were significantly associated with at least one further iteration of the app being developed. Conclusion Although there has been an exponential increase in the number of eHealth apps, the number of studies that have been published that report the results of usability testing on these apps has not increased at an equivalent rate. The number of digital health applications that publish their usability evaluation results remains only a small fraction. Questionnaires are the most prevalent method of evaluating usability in eHealth applications, which provide an overall measure of usability but do not pinpoint the problems that need to be addressed. Qualitative methods may be more useful in this regard. The use of multiple evaluation methods has increased. Automated methods such as eye tracking have not gained traction in evaluating health apps. Further research is needed into which methods are best suited for the different types of eHealth applications, according to their target users and the health conditions being addressed

Usability analysis of contending electronic health record systems

In this paper, we report measured usability of two leading EHR systems during procurement. A total of 18 users participated in paired-usability testing of three scenarios: ordering and managing medications by an outpatient physician, medicine administration by an inpatient nurse and scheduling of appointments by nursing staff. Data for audio, screen capture, satisfaction rating, task success and errors made was collected during testing. We found a clear difference between the systems for percentage of successfully completed tasks, two different satisfaction measures and perceived learnability when looking at the results over all scenarios. We conclude that usability should be evaluated during procurement and the difference in usability between systems could be revealed even with fewer measures than were used in our study. © 2019 American Psychological Association Inc. All rights reserved.Peer reviewe

Wearables at work:preferences from an employee’s perspective

This exploratory study aims to obtain a first impression of the wishes and needs of employees on the use of wearables at work for health promotion. 76 employ-ees with a mean age of 40 years old (SD ±11.7) filled in a survey after trying out a wearable. Most employees see the potential of using wearable devices for workplace health promotion. However, according to employees, some negative aspects should be overcome before wearables can effectively contribute to health promotion. The most mentioned negative aspects were poor visualization and un-pleasantness of wearing. Specifically for the workplace, employees were con-cerned about the privacy of data collection

Acceptance of Health Services on Mobile Phones: A Study of Consumer Perceptions

CONTEXT: Mobile phones are a popular technology capable of portable computing and data access. These features could be helpful for delivering health care services. OBJECTIVE: This study examined health consumers' perceptions of mobile phone health services and their intentions to adopt the technology. DESIGN: Participants viewed a presentation introducing mobile phone services for interacting with health professionals, taking health actions, delivering health information, and managing health care services. Afterwards, semi-structured interviews inquired about their perceptions and acceptance of the technology. ANALYSIS: Qualitative analysis identified emerging themes from interview transcripts. PARTICIPANTS: Forty adult health consumers who were healthy individuals or patients with chronic health conditions - additionally, they had experience with health services delivered through information technology, or they did not. RESULTS: All participants intended to use the technology: thirty were ready to adopt immediately, while ten intended to adopt later upon the need or when particular conditions were met. Among all respondents, sixteen were interested in adopting a selection of the services only. Diverse motivations and pathways may shape health consumers' adoption decision. However, the general process has potential adopters considering their health status and health environment, their personality, and the perceived helpfulness of the technology. Helpfulness was determined by reflection upon positive and negative perceptions of the technology and consideration of usability and safety. Participants had positive impressions that the technology supports healthy behaviors and fosters quality and efficiency of care. There were also preferences for mobile and digital health services. Furthermore, participants believed the technology could support health information behaviors. The negative perceptions were concerns that the technology requires immense resources and skills, is stressful to use, neglects the social dimension of health care, and threatens the quality of care. Consequently, there was interest in the development of safe, responsive, and inclusive mobile health systems. Implementation of mobile phone health services should respond to health consumers' interests and concerns. CONCLUSION: Health consumers are interested in using mobile phone health services

Examining the Design and Usability of Telemedicine Communications: A Mixed-methods Study

This dissertation describes a mixed-methods study that examines the usability of telemedicine provider interfaces. This study consisted of content analysis, survey, and think aloud methodologies, which afford a multifaceted corpus of data for which to draw inferences and identify design features and functions that negatively impact usability. Usability is a critical component of the user experience with a telemedicine provider interface and can suede or impede the acceptance and adoption of telemedicine. Telemedicine has the potential to increase quality healthcare access and positive health outcomes for individuals who use it, and usability is a key component of technology acceptance and effective use. Empirical testing of health information technology (HIT) and telemedicine is advocated for as it is the most valuable method of research to understand humans\u27 cognitive processing of information as they interact with technology. In addition, using activity theory and mobile interface theory as a lens in which to understand human activities and interaction with telemedicine provider interfaces, including the telemedicine provider websites and their mobile-responsive websites in this study, is an effective tool for drawing reasonable inferences regarding the usability of telemedicine communications. Considering the rate at which an unprecedented amount of health information becomes available online and HIT facilitates the delivery of healthcare, usability testing and user-centered, iterative design practices become increasingly essential in order to design effective—and safe—health information and technology that enhance the patient-experience, the affordability and accessibility of healthcare, health literacy and patient empowerment, and positive health outcomes. Usability testing plays an increasingly important role in characterizing obstacles to achieving these initiatives of the modern patient-centered health paradigm and telemedicine. The mixed-methods usability testing performed in this study offers a principled approach to usability testing and is ecologically valid because it involves real human subjects. This study fulfills a void in research on the usability of telemedicine communications and reveals usability problems that may not be anticipated by designers of HIT and health information providers. Drawing from the insight gained from this mixed-method study, design features and functions that enhance the usability of health communications are offered. This study draws insight from the human factors, technical communication, and health and medical fields to develop systematic, practical usability testing methods that can be replicated and applied in many fields. The design recommendations resulting from this study will be valuable to programmers; systems analysts; clinicians and nurses; technical communicators; information architects; visual designers; and others in similar roles