An evaluation framework and selection tool for education apps usability, with a case study from health education apps

Mobile apps for health education are commonly utilised to support different users. The development of these apps is increasing rapidly. A critical evaluation framework is needed to ensure the usability and reliability of Mobile Health Education Applications (MHEAs) to save time and effort for stakeholders. This project aims to assist the evaluation of MHEAs through development of an evaluation framework, which includes suitable metrics, an efficient hybrid utilizing Heuristic Evaluation (HE) and Usability Evaluation (UE). This framework determines the usefulness and usability of MHEAs, in order to improve the software engineering to create more effective ways to evaluate such software. In this framework, the Medical Apps Selection Tool (MAST) has been developed in which performance helps to select suitable MHEAs, assisting stakeholders to choose MHEAs that meet their requirements.The thesis employs two methods to make the evaluation framework capable of performing qualitative and quantitative data analysis. The first is a qualitative method, involving interviews based on proposed selected hybrid metrics from HE and UE, with three kinds of stakeholders: Patients, Health Professionals and Software Developers, to identify specific relevant selected hybrid metrics to measure usability in MHEAs. These metrics are deployed to measure usability in different MHEAs based on ranking these apps within the evaluation framework. These metrics were converted into an evaluation questionnaire, which has been applied to several MHEAs. The second method is the translation of the outcomes form the first method to measure the usability of MHEAs and determine what stakeholders require from using MHEAs. For this purpose, it categorises stakeholders with different needs from MHEAs; this reflected in the MAST, based on matching different stakeholders with different MHEAs.The findings of the study indicate that the evaluation framework is able to evaluate MHEAs and record usability problems. Furthermore, this evaluation framework leads to selection of the most appropriate apps by developing the MAST for stakeholders. The framework is expected to be applicable to other domains and platforms

Development of a Culturally-Oriented Website Usability Evaluation

As the uni-cultural studies of website usability have matured, the paucity of cross-cultural studies of usability become increasingly apparent. Moving toward these cross-cultural studies will require the development of a new tool to assess website usability in the context of cultural dimensions. This paper introduces the preliminary results from the first phase of this project and then presents the proposed method for the research in progress that specifically is directed to the development and quantitative evaluation of a measurement scale of a culture sensitive measurement of website usability. The recognition of the need to develop this scale resulted from the identification of culture-related shortcomings of previous measurement tools that have been used widely within the Management of Information Systems (MIS) literature

Usability Evaluation in Virtual Environments: Classification and Comparison of Methods

Virtual environments (VEs) are a relatively new type of human-computer interface in which users perceive and act in a three-dimensional world. The designers of such systems cannot rely solely on design guidelines for traditional two-dimensional interfaces, so usability evaluation is crucial for VEs. We present an overview of VE usability evaluation. First, we discuss some of the issues that differentiate VE usability evaluation from evaluation of traditional user interfaces such as GUIs. We also present a review of VE evaluation methods currently in use, and discuss a simple classification space for VE usability evaluation methods. This classification space provides a structured means for comparing evaluation methods according to three key characteristics: involvement of representative users, context of evaluation, and types of results produced. To illustrate these concepts, we compare two existing evaluation approaches: testbed evaluation [Bowman, Johnson, & Hodges, 1999], and sequential evaluation [Gabbard, Hix, & Swan, 1999]. We conclude by presenting novel ways to effectively link these two approaches to VE usability evaluation

Development and Usability Evaluation of a Nursing Graduate Information Management System (GSMIS) Based on User Experience

[Research Background] Recent years, the rapid development of information technology in colleges and universities and the continuous expansion of graduate students have not only increased the pressure of graduate student management in the School of Nursing of Fudan University, but also accelerated the pace of information technology construction in the college, and the School of Nursing urgently needs to use the existing good platform to realize the information technology of nursing graduate student management in combination with the actual situation of the college, so as to realize the communication between graduate student supervisors, graduate students and management, and also to build an objective and scientific clinical competence assessment system in line with It is also important to provide material for the construction of an objective and scientific clinical competence assessment and evaluation system that is in line with the postgraduate nursing education, which is important to achieve the cultivation goals and improve the quality of nursing graduate training. [Research Purpose] Based on the theory of user experience and the concept of "User-Centered Design", this project develops the information management system for graduate students in the School of Nursing by studying the actual needs of teachers and graduate students in the School of Nursing, which can be divided into the following contents: (1) Construction of the framework content of a Nursing GSMIS (2) Development and testing of a Nursing GSMIS (3) Usability evaluation of the a Nursing GSMIS [Research Methods] 1. Demand analysis of the Nursing GSMIS (1) Preliminary framework: Using target sampling method and maximum variation method, semi- structured interviews were conducted with both faculty and student users of the School of Nursing to build the preliminary framework of the system. (2) Final framework: Expert focus group interviews were used to revise and adjust the required content such as interface design, business functions and performance, and to determine the final framework of the system. 2. Development and testing of Nursing GSMIS The “Rapid Prototype Interaction” model was used to develop the system. After the prototype was developed, the Shanghai Institute of Comprehensive Application of Network Technology was contacted for professional system testing, and then the system was modified and adjusted until it met the research requirements. 3. Usability evaluation of Nursing GSMIS A mixed evaluation method of usability test, questionnaire and qualitative interview was used to conduct a more comprehensive and objective usability evaluation of the GSMIS successfully developed in the early stage. It was used to understand the users' feelings and experiences of the system, analyze the advantages and shortcomings of the system, and explore its actual usability.[Research Results] 1. Demand analysis of Nursing GSMIS (1) Preliminary framework of the system: The faculty and student users interviewed expressed that information management is an important step in the progress of the college and the nursing discipline, and expressed strong support and need for the construction of our information system. Based on the literature review and the needs of faculty and students, the framework was initially formulated as four modules: basic information of admission, cultivation process, research results, and clinical cases. (2)The final framework of Nursing GSMIS: It is built from modules and functions, performance, interface, text, color, etc.; Modules include: basic information module for admission, training process module, scientific research achievement module, clinical case library module, employment information module, graduation tracking module; functions include password setting and retrieval, user log-in, user management, data maintenance, custom query, audit and message notification; Performance can be summarized as follows: data structure is clear, complete, with real-time, expansion, operation and stability; Interface Design is friendly and beautiful, simple and easy to use; Layout should highlight the key points; text design is easy to recognize and read, color design reflects the connotation and characteristics of the college. The performance can be summarized as: clear data structure, complete, real-time, scalability, operability and stability; friendly and beautiful interface design, simple and easy to use, layout to highlight the key points; text design is easy to identify, easy to read, color design reflects the connotation and characteristics of the college. 2. Development and testing of Nursing GSMIS Based on the requirements, the engineers used the “Rapid Prototype Interaction Model” to develop the system. After the prototype was developed, the Shanghai Institute of Comprehensive Application of Network Technology was contacted for professional system testing, and then after many rounds of software discussions and corrections between the software engineers and researchers, the GSMIS finally had the registration and log-in page, the informed consent notification page, the information notification page and six main modules. 3. Usability evaluation of Nursing GSMIS (1) Usability Test: Based on the usability test theory, the target sampling method was used, and five graduate students users were selected to participate in the test is sufficient, and the results are as follows. 1) Quantitative results: The effectiveness was mostly at 100%, and the efficiency was 56.2min/s, 72.2min/s, 53min/s, 225.2min/s, 75.8min/s, 33min/s, and 5.5min/s; The overall score of the post-scene questionnaire (ASQ) was 1.24. The evaluation indexes all indicated that the users rated the system highly. 2) Qualitative results: The test subjects all easily completed and highly praised the system during the test, indicating that the operation was not difficult and satisfied the interaction needs of users. (2) SUS Questionnaire: A total of 40 users completed the SUS using the convenience sampling method and the maximum variance method, and the Mean±SD was 70.23±7.7. Among them, the mean score of the Usability sub-scale was 71.59±8.93 , and the mean score of the ease of Learning sub-scale was 64.77±7.53, with scores ranging from 60 to 70. (3) Qualitative interviews: Using target sampling method and maximum variation method, a total of 10 people were interviewed, including 5 students and 5 staff users, and the number of interviews was 3, with no repeated interviews or secondary interviews. The maximum interview time was 46 minutes, the minimum was 25 minutes, and the average was 35.5 minutes. The results can be summarized into three aspects: general evaluation, content and function evaluation, and shortcomings and improvement. This study combines qualitative and quantitative results, reflecting the good usability of the system, but there are also shortcomings and improvements. [Conclusion] This study used various research methods such as literature review, qualitative interview, and focus group interview to construct a system framework based on user experience, completed the development of the graduate student information management system in the School of Nursing, and used mixed methods such as out-of-sound thinking method, usability testing method, questionnaire survey method, and qualitative interview to evaluate the usability of the system, and the results showed that the system has high acceptance, ease of use, and usability

Evaluating students’ experiences in self-regulated smart learning environment

The increasing development in smart and mobile technologies transforms a learning environment into a smart learning environment that can support diverse learning styles and skills development. An online learner needs to be supported for an engaging and active learning experience. Previously, this progressive research developed and implemented a self-regulated smart learning environment (mobile app) among final-year undergraduate students to support online learning experiences. To understand students' experiences, there is a need to evaluate the mobile app. However, there is a lack of a well-documented study investigating students' experiences in terms of usability, challenges, and factors influencing satisfaction to inform a decision regarding future implementation. This study attempts to fill these gaps by exploring these experiences for sustainable future implementation. The study used cyclical mixed-method evaluations to explore the experiences of 85 final-year undergraduate students. The quantitative data were collected using a survey on the constructs of the research model previously developed to evaluate factors influencing students' satisfaction, and the qualitative used focus group discussions to explore usability experiences and challenges of implementations. The quantitative data were analyzed using SPSS 25 to confirm the structural equation model's relationship. The qualitative data were analyzed using a thematic process to understand students' experiences. The findings from the first mixed-method evaluation show that students were able to follow the learning process, and the application supported their online learning experiences. However, a student expressed the need to improve user functionalities to motivate and engage them in the learning process. The suggestions were incorporated into the mobile app development for the second evaluation. The findings from the second evaluation revealed similar support. However, students suggested a web-based version to support different operating systems and improve interactions. Furthermore, the information system qualities and moderating factors investigated supported students' satisfaction. Future research could explore facilitators' experiences in the mobile app for sustainable development and implementation for engaging online learning experiences and skills development

DEVELOPMENT AND USABILITY EVALUATION OF PLATFORM INDEPENDENT MOBILE LEARNING APPLICATION (M-LA)

In today’s digital age, wireless technology and widespread use of handheld devices are going under a continuous advancement to provide information anywhere and at anytime. Furthermore, these technologies are being utilized in the field of education and called mobile learning (M-Learning). Hence, M-learning means using of mobile devices and wireless computing as a learning instrument and communication technology respectively. The limitations of M-learning include either hardware or software of mobile devices, content creation, and no standards for mobile learning system (M-LS), wireless technology, and security. The main objectives of this research are to study and design model for M-learning approach; to develop platform independent M-learning application (M-LA) for Fundamentals of Programming course; to design M-LS platform classification, and to evaluate the effectiveness and usability of the application. In addition, under the development of this application the following aspects are considered: learning theories, M-learning development principles, and some of the aforementioned M-learning limitations. To achieve the above mentioned objectives, ADDIE (Analysis, Design, Development, Implementation, and Evaluation) life cycle is adapted which is one type of instructional design model (IDM). The application has been developed using Java 2 Micro Edition (J2ME), and Extensible Markup Language (XML). It contains several sections, but the main modules are Lecture Materials, and Quiz. Quasi Experiment Design and usability attributes was used to evaluate the effectiveness and usability of the application respectively using Universiti Teknologi PETRONAS foundation students. Finally, the data was analyzed using quantitative and qualitative method. The quantitative data was analyzed using coefficient variance and independent t-Test, and Cronbach alpha used to measure the internal reliability of the data. Overall results show that M-LA is efficient to improve learners’ performance, makes learning enjoyable, support continuous learning and learning time is reduced, and fulfilled the usability needs