Much effort has been devoted to evaluating the usability of web-based system.With the increase in the mobile-based applications coupled with the limitations and challenges of mobile devices, it becomes mandatory to evaluate the web-based systems in the context of smartphone usability.In addition, a number of international standards/models on usability are available, but seldom used for practical usability evaluation. In this study, the popular ISO 9241-11 standard was used to evaluate a web-based health awareness portal within the smartphone mobile context.The results reveal some pointers to usability issues as well as confirmation that the web-based awareness portal is relatively usable on smartphone devices within the components defined in the models

Hussain, Azham

Hussain, Zakaria

Mkpojiogu, Emmanuel O.C.

UUM Repository

 77:4 (2015) 1–5| www.jurnalteknologi.utm.my | eISSN 2180–3722 |   Jurnal Teknologi   Full Paper       USABILITY EVALUATION OF A WEB-BASED HEALTH AWARENESS PORTAL ON SMARTPHONE DEVICES USING ISO 9241- 11 MODEL    Azham Hussaina*, Emmanuel O.C. Mkpojiogua, Zakaria Hussainb  aSchool of Computing, Universiti Utara Malaysia, 06010 UUM Sintok, Malaysia bFaculty of Electrical Engineering, Universiti Teknologi MARA, 13500 Permatang Pauh, Pulau Pinang, Malaysia  Received  02 June 2015 Received in revised form  09 August 2015 Accepted  1 September 2015  *Corresponding author azham.h@uum.edu.my  Graphical abstract  Abstract  Much efforthas been devoted to evaluating the usability of web-based system. With the increase in the mobile-based applications coupled with the limitations and challenges of mobile devices, it becomes mandatory to evaluate the web-based systems in the context of smartphone usability. In addition, a number of international standards/models on usability are available, but seldom used for practical usability evaluation. In this study, the popular ISO 9241-11 standard was used to evaluate a web-based health awareness portal within the smartphone mobile context. The results reveal some pointers to usability issues as well as confirmation that the web-based awareness portal is relatively usable on smartphone devices within the components defined in the models.    Keywords:Usability, web-based, health portal, mobile devices, smartphones, ISO 9241-11   © 2015 Penerbit UTM Press. All rights reserved     1.0 INTRODUCTION  The popularity of mobile devices is on the rise. There are over one billion mobile subscribers [5]. Mobile devices have become an everyday commodity amongthe users. A recent estimate reveals that by the year 2017 the number of mobile devices per capita will be 1.4 [17]. Smartphones are the most popular mobile devices. In fact, worldwide statistics show that one in every five persons possesses a smartphone [17]. However, smartphones have some limitations and challenges in its interface due to the peculiar characteristics of mobile devices such as, low display resolutions, smallness of screen size, navigations difficulties, and non-conventional input methods [12]. Because of these inherent challenges and the fact that mobile applications lack robustness, and flexibility, and remain difficult to use, therefore usability evaluation becomes a very important issue for mobile devices and in particular, smartphones [5] [17]. Smartphones are considered very personal and their usability affects the users that use them [9].   Online users of websites/portals have many choices in finding information on websites. If users cannot find information and/or do so with difficulty, they certainly will go elsewhere for such information. They might also inform their friends and colleagues about their frustrating experience and this will affect loyalty to the web-based product [15]. This underscores the importance of usability within the context of websites. The ISO 9241-11 standard defines usability as the combination and convergence of effectiveness, efficiency and satisfaction [10] [14] [15]. This definition is about the most popular and the most widely used definitions of usability [4] [15]. The framework of the ISO 9142-11 model prompts for its 2                                Azham, Emmanuel & Zakaria / Jurnal Teknologi (Sciences & Engineering) 77:4 (2015) 1–5   use in usability evaluation. The standard defines usability thus as: “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in the specified context of use” [10]. Usability is an abstract construct which can be measured within the above stated dimensions. Usability is both a performance as well as a perception, (an aptitude (ability) as well as an attitude) that is, it has to do with users using the system (doing something in the system) and them ex-pressing how they felt about the system they used. Effectiveness and efficiency are performance dimensions of usability while satisfaction is the perception dimension of usability. Brooke [3] as cited Speicher [16] argued that “usability does not exist in an absolute sense; it can only be defined with reference to particular context” [16]. This implies that to speak about usability, the characteristics that underline it must be specified [16]. Effectiveness relates to whether users can actually complete their tasks and achieve their goals by using a given system. Efficiency refers to the extent to which users expend their resources in accomplishing their goals of using a system/product. Satisfaction is the degree of comfort and delight that users experience while achieving their goals or as they use the software [4]. As authoritative as this international standards for usability is, many practitioners/researchers still do not use it in usability evaluation [2]. So in this study, a usability evaluation will be carried out within the context of a web-based health awareness portal used/accessed on smartphones using the ISO 9241-11 framework.  The remaining part of this paper will address the following sub-sections: related works, methods, results, and discussion and conclusion  2.0 RELATED WORKS  This subsection addresses the some of the related usability evaluation models and the need for usability evaluation of web-based health awareness portals. Coursaris and Kim [5] stated that although a considerable volume of research on general usability exists, relatively, few studies have been done with focus on mobile technology. They also added that only 41% of mobile usability research studies are empirically inclined [5]. Furthermore, Coursaris and Kim [5] also argued that there were no usability evaluation frameworks that exist at the time in the context of mobile computing environment [5]. So, in light of this, they proposed a framework for the evaluation of mobile usability. The framework consists of three components: the first outer layer includes four factors, namely: user, technology, task/activity, and environment. The second inner layer has the key usability dimensions that is, effectiveness, efficiency, satisfaction, learnability, flexibility, attitude, operation-ability, etc. Lastly, the component includes the consequences like a system integration improvement, retention loyalty, and trust, etc. Also, in the context of website usability, Agarwal and Venkatesh [1] proposed five categories/ constituents: ease of use, content, promotion, made-for-the-medium, and emotion. Their proposal also has subcategories like, relevance, media use, depth/breadth, feedback, structure, community, challenge, plot, personality, etc. [1]. As can be observed, there are specific frameworks for mobile and website usability context. However, this study requires that the evaluation be done with the two contexts in view, therefore a context agnostic and more general model is required. The ISO 9241-11 fits well in this scenario.  2.1  The Need for the Usability Evaluation of Web-Based Health Awareness Systems  Health is a top browsing content on the Internet today. In a survey work carried out by the Pew Internet and American Life Project, there is a rise in the popularity and option of the Internet as a source and resource for obtaining information on health related issues [7][13]. Back in 2005, 8 in 10 (80%) of Internet users browsed the Internet for health related contents [7]. In addition, as Freudenheim [8] stressed, a more recent survey results indicate that 4 in 5 (80%) of Internet users seek healthcare related information on the Internet [8]. Also, Samuel and Zaiane [13] observed that online health sites provide a great range of topics on health, ranging from general topics, to specialized ones. They observed that from statistics provided from a survey, 51% of health sites address general health related topics. These resources are provided via an owner generated content that accounts for 46% of the Websites surveyed. In addition, they also showed that 56% of health content among the surveyed websites is for broadcast-to-any, that is, content delivered to any user [13]. In 2012, a study was conducted that showed that users who need self-help are increasingly having access to healthcare on the Internet [6]. On the Internet, there is a large content on health related information that can be accessed free of charge by patients [11]. This surge in the use and search for health related information on the internet makes the usability of web-based systems an imperative, thus, making their evaluation mandatory.     3.0 METHODS  A lab-based usability testing method was employed to test the usability and user experience of the e-Ebola Awareness System for Smartphones. The Smartphones used in the testing include: Samsung and Lenovo. A sample size of 9 (with an expected 27% margin of error at 95% level of confidence) was used for the Smartphone testing. Four task scenarios were made use of by users, namely: task 1: Open three news contents on Ebola in new tab and write out the name of the news media; task 2: Find three 3                                Azham, Emmanuel & Zakaria / Jurnal Teknologi (Sciences & Engineering) 77:4 (2015) 1–5   tweets on Ebola and write down the name of the source of the tweets; task 3: Search for information on Ebola symptom and Ebola prevention and write out one symptom and prevention each; task 4: View the content on Ebola causes and Ebola treatment in any language of your choice other than English. The following performance metrics were collected during the usability testing: Task time, task completion rates, and task errors. These performance metrics cover the effectiveness and efficiency dimensions of the ISO 9241-11 standard/model and as suggested by [18]. Task completion rate and task errors measure effectiveness, while task time measures efficiency. The perception metrics collected included: Task ease and the overall system satisfaction. Task ease and overall system satisfaction measure satisfaction. Task ease was measured using a single easy question, a single 7-point questionnaire that ranges from ‘1=very difficult’ to ‘7=very easy’ (Overall, how difficult or easy did you find this task). The overall satisfaction was measured using the System Usability Scale (SUS). Apart from measuring satisfaction (that is, in terms of usability), this questionnaire also measures learnability as a sub-dimension. SUS is a 5-point Likert-type questionnaire with 10 item questions and with options ranging from strongly disagree to strongly agree. The items alternate between positive and negative questions, implying that there are 5 positive and negative questions respectively. SUS is reliable and valid. Its scores have a modest correlation with task performance. It measures both learnability and usability [14][4]. The test protocol for the lab-based usability testing is as follows: 1. System setup & Internet connection, 2. In-briefing, 3. Pre-test questionnaire, 4. Test session (about 45 minutes), 5. Post-testquestionnaire, and 6. Debriefing. The Single Ease Question was administered after every task scenario. The pre-test questionnaire administered before a test session, is used to collect demographics (like gender, age, marital status, average time spends on the internet daily etc.) from users/testers. The SUS questionnaire is administered as a post-test questionnaire at the end of the test session.   4.0 RESULTS  4.1 Effectiveness  The results of the usability test show that there was no significant difference in the task completion rates for all tasks, except for task 4, where all users failed the task. The highest task completion rate is in task 3, with a 100% success. Others ranged from 78 to 89% completion rate. Task 4 was a total failure (0%). Users had a usability problem with task 4. They could not translate the content to the language of their choice. On the whole, the overall completion rate was 67% for the entire test.  Figure 1 Task completion rate  From the result obtained from the test, there was no significant difference in the task error rates for all tasks. However, task 3 and 4 had no error (0% error) (note that all users failed task 4). Task 2 had the highest error rate (33%), followed by task 1 (11%). This suggests that users had some difficulties doing these tasks. The overall error rate for the test was 11%.  Figure 2 Task error rate  4                                Azham, Emmanuel & Zakaria / Jurnal Teknologi (Sciences & Engineering) 77:4 (2015) 1–5   4.2 Efficiency  Figure 3 Task time  The highest task time was in task 3 with an average of 1.198 minutes. Though all users were able to successfully complete the task, however, they expended more time to achieve that. Task 2 had the least task time. The overall completion time is 1.085 minutes. On the average, lest than 2 minutes were spent by users in each of the tasks.   4.3 Satisfaction  Figure 4 Task ease   Figure 5 System satisfaction  As could be seen in Figure 4, all the tasks were relatively easy for all users on the average; however, there was no significant difference in ‘task ease’. Task 3 was the easiest task, followed by task 1 and then task 2. The most difficult task is task 4. However, user rating of task ease of task 4 is somewhat exaggerated, but it shows that they felt the task was 5                                Azham, Emmanuel & Zakaria / Jurnal Teknologi (Sciences & Engineering) 77:4 (2015) 1–5   easy with their rating of 5.22, even though, they all failed the task. The overall task easy is 5.75. Figure 5 presents the SUS scores for the test. The perceived usability score for the portal is 55.33, while its learnability score is 14.72. The overall perceived satisfaction score is 68.06. All scores range from 0 to 100. There is no significant difference between the perceived usability and the overall system satisfaction. However, there is a significant difference between the perceived learnability and the perceived usability and overall satisfaction.   4.0 DISCUSSION AND CONCLUSION  From the usability testing, though there were some indicators of usability issues observed, it can be seen that the web-based awareness portal (e-Ebola awareness system), is relatively usable on smartphones. In terms of effectiveness, a considerable number of users were successful is accomplishing their goals in all tasks with the exception of task 4. Also, there were varying degree of error rates, however, tasks 3 was achieved without errors. With regard to efficiency, all tasks were relatively achieved with minimal time resources. All completed tasks took less than 2 minutes on the average to be achieved. In addition, the perceived satisfaction score for the portal was relatively high, implying that users were satisfied with the usability of the e-health awareness portal for smartphones. Future works will examine, the usability of the e-health awareness system for laptops and compare it with results produced from the smartphone context of use.   References  [1] Agarwal, R. and Venkatesh, V. 2002. Assessing A Firm’s Web Presence: A Heuristic Evaluation Procedure For The Measurement Of Usability. Information Systems Research, 13(2): 168-186. [2] Bevan, N. 2009. International Standards For Usability Should Be More Widely Used. Journal of Usabil-ity Studies, 4(3): 106-113. [3] Brooke, J. 1996. SUS: A “Quick And Dirty” Usability Scale. In P. Jordan, B. Thomas, B. Weerdmeester (Eds.), Usability Evaluation in Industry. 189-194. London, UK: Taylor & Francis. [4] Brooke, J. 2013. SUS: A Retrospect. Journal of Usability Studies, 8(2): 29-40.  [5] Coursaris, C.K., and Kim, D.J. 2011. A Meta-Analytical Review Of Empirical Mobile Usability Studies. Journal of Usability Studies, 6(3): 117-171.  [6] Culjak. G. 2012. Access, Awareness And Use Of Internet Self-Help Websites For Depression In University Students.HICSS’12, Maui, HI, 4-7 Jan 2012. 2655-2664. [7] Fox, S. 2005. Health information online. Pew Internet and American Life Project. Retrieved from http://www.pewinternet.org/PPF/r/156/report_display.asp. [8] Freudenheim, M. 2011. Health Care Is High Among Web Searches. Pew Internet and American Life Project. Retrieved from http://www.pewinternet.org/Media-Mentions/2011/NYT-health-Care-Is-High-Among-Web-Searches.aspx. [9] Groth, A., and Haslwanter, D. 2015. Perceived Usability Attractiveness And Intuitiveness Of Responsive Mobile Tourism Websites: A User Experience Study. In L. Tussyadiah, A. Inversini (Eds.), Infor-mation and Communication Technologies in Tourism 2015. 593-606. Logano, Switzerland: Springer International Publishing.  [10] ISO 9241-11 1998. ISO 9241-11: 1998: Ergonomic Requirements For Office Work With Visual Display Terminal (VDT)S – Part 11: Guidaince On Usability. [11] Ji, X., Chun, S.A. and Geller,  J. 2013. Social Infobuttons: Integrating Open Health Data with Social Data using Semantic Technology.Proc.SWIM’13, New York, USA, June 23. [12] Nah, F.F., Siau, K., and Sheng, H. 2005. The Value Of Mobile Applications: A Utility Company Study. Communication of the ACM. 48(2): 85-90. [13] Samuel, H.W. and Zaiane, O.R. 2011. HCMS: Conceptual Description Of A Health Concept Management System, SEHC’11,  Waikiki, HI, May 22-23, 2011. 17-23. [14] Sauro, J. 2011. A Practical Guide To The System Usability Scale: Background, Benchmarks, & Best Practices. Denver, CO: Measuring Usability LLC. [15] Sauro, J. 2015. SUPR-Q: A Comprehensive Measure Of The Quality Of The Website User Experience. Journal of Usability Studies, 10(2): 68-86. [16] Speicher, M. 2015. What Is Usability? A Characterization Based On ISO 9241-11 And ISO/IEC 25010. Technical Report. Retrieved from http://arxiv.org/pdf/1502.06792v1.pdf [17] Zapata, B.C., Fernandez-Aleman, J.L., Idris, A. and Toval, A. 2015. Empirical Studies On Usability Of Mhealth Apps: A Systematic Literature Review. Journal of Medical Systems, 39(1). Springer US. [18] Hussain, A., Mutalib, N. A., & Zainol, A. 2014. A Usability Testing on JFakih Learning Games for Hearing Impairment Children.Paper presented at the The 5th International Conference on Information and Communication Technology for The Muslim World (ICT4M), Kuching, Sarawak.  

Usability evaluation of a web-based health awareness portal on smartphone devices using ISO 9241-11 model

http://repo.uum.edu.my/17466/1/JT%20%2077%204%20%201%E2%80%935.pdf

Usability evaluation of a web-based health awareness portal on smartphone devices using ISO 9241-11 model

Abstract

Similar works

Full text

Available Versions

UUM Repository