Automated usability evaluation during model-based interactive system development

Abstract. In this paper we describe an approach to efficiently evaluate the usability of an interactive application that has been realized to support various platforms and modalities. Therefore we combine our Multi-Access Service Platform (MASP), a model-based runtime environment to offer multimodal user interfaces with the MeMo workbench which is a tool supporting an automated usability analysis. Instead of deriving a system model by reverse-engineering or annotating screenshots for the automated usability analysis, we use the semantics of the runtime models of the MASP. This allows us to reduce the evaluation effort by automating parts of the testing process for various combinations of platforms and user groups that should be addressed by the application. Furthermore, by testing the application at runtime, the usability evaluation can also consider system dynamics and information that are unavailable at design time

An empirical approach for evaluating the usability of model-driven tools

MDD tools are very useful to draw conceptual models and to automate code generation. Even though this would bring many benefits, wide adoption of MDD tools is not yet a reality. Various research activities are being undertaken to find why and to provide the required solutions. However, insufficient research has been done on a key factor for the acceptance of MDD tools: usability. With the help of end-users, this paper presents a framework to evaluate the usability of MDD tools. The framework will be used as a basis for a family of experiments to get clear insights into the barriers to usability that prevent MDD tools from being widely adopted in industry. To illustrate the applicability of our framework, we instantiated it for performing a usability evaluation of a tool named INTEGRANOVA. Furthermore, we compared the outcome of the study with another usability evaluation technique based on ergonomic criteria.This work has been developed with the support of the Intra European Marie Curie Fellowship Grant 50911302 PIEF-2010, MICINN (TIN2008-00555, PROS-Req TIN2010-19130-C02-02), GVA (ORCA PROMETEO/2009/015), and co-financed with ERDF. We also acknowledge the support of the ITEA2 Call 3 UsiXML (20080026) and financed by the MITYC under the project TSI-020400-2011-20. Our thanks also to Ignacio Romeu for the video data gathering setup.Condori-Fernandez, N.; Panach Navarrete, JI.; Baars, AI.; Vos, TE.; Pastor López, O. (2013). An empirical approach for evaluating the usability of model-driven tools. Science of Computer Programming. 78(11):2245-2258. https://doi.org/10.1016/j.scico.2012.07.017S22452258781

Evaluation of the QVT Merge Language Proposal

-STF90 A05045This report has identified 29 weighted evaluation criteria representing desired properties of a model to model transformation language. These criteria have been used to evaluate the current QVT Merge specification. We have so far only been able to evaluate 21 of these criteria, mainly due to missing tool support. Some of the criteria are considered absolute in the sense that missing to fulfil such a criterion is considered a failure. The 21 evaluated criteria give a score of 59 out of a maximum possible score of 68 (language-based + example-based testing). We have also compared the QVT-Merge submission with the QVT-Compuware/Sun submission and at the time being the QVT-Merge seems to be the preferred one due to more support on the absolute criteria and better easy-to-use score. Eight transformation examples for solving six different transformation tasks have given a lot of insight on the ease of use criteria for both simple and complex transformations. When defining transformations using QVT Merge we believe that a lot of effort may be required in order to define the source and target  metamodels. The evaluation in this report could be improved by using the reference examples with alternative approaches published in the literature. An available QVT-Merge tool is necessary in order to provide evaluations of all the suggested criteria. In order to further investigate the usability of the graphical notation, we need to define more of the transformation examples graphically. Only one of the examples has been specified graphically in this version. The current evaluation has been done by a single evaluator who has only reviewed the transformation code that was written by somebody else. The evaluation will be further improved by incorporating input from other evaluators as well as evaluation from those who wrote the transformation code. Oppdragsgiver: EU Commissio

User Interface Evaluation with Machine Learning Methods

With the increasing complexity of user interfaces and the importance for usability evaluation, efficient methods for evaluating the usability of user interfaces are needed. Through this dissertation research, two computational models built with machine learning methods are introduced to evaluate user interface usability. This research consists of two phases. Phase I of the research implements the method of support vector machine to evaluate usability from static features of a user interface such as widget layout and dimensions. Phase II of the research implements the method of deep Q network to evaluate usability from dynamic features of a user interface such as interaction performance and task completion time. Based on the research results, a well-trained Phase I model can distinguish and classify user interfaces with common usability issues and is expected to recognize those issues when sufficient data is provided. Phase II model can simulate human-interface interaction and generate useful interaction performance data as the basis for usability analysis. The two phases of the research aim to overcome the limitations of traditional usability evaluation methods of being time-consuming and expensive, and thus have both practical and scientific values. From the practical perspective, this research aims to help evaluate and design user interfaces of computer- based information systems. For example, today’s application software development on computer based information system always integrates many functions or task components into one user interface page. This function integration needs to be carefully evaluated to avoid usability issues and the competitive field of software development requires an evaluation process with short cycles. Phase I and Phase II of the research provide an efficient but not necessarily comprehensive usability evaluation tool to meet some of the demands of the field. From the scientific perspective, this research aims to help researchers make quantifiable predictions and evaluations of user interfaces. Qualitative theories and models are important, but often insufficient for rigorous understanding and quantitative analysis. Therefore, this research work on computational model-based interface evaluation has important theoretical value in advancing the science of studying human behavior in complex human-machine-environment systems.PHDIndustrial & Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/149942/1/myx_1.pd

Measuring the Usability and Quality of Explanations of a Machine Learning Web-Based Tool for Oral Tongue Cancer Prognostication

Background: Machine learning models have been reported to assist in the proper management of cancer through accurate prognostication. Integrating such models as a web-based prognostic tool or calculator may help to improve cancer care and assist clinicians in making oral cancer management-related decisions. However, none of these models have been recommended in daily practices of oral cancer due to concerns related to machine learning methodologies and clinical implementation challenges. An instance of the concerns inherent to the science of machine learning is explainability. Objectives: This study measures the usability and explainability of a machine learning-based web prognostic tool that was designed for prediction of oral tongue cancer. We used the System Usability Scale (SUS) and System Causability Scale (SCS) to evaluate the explainability of the prognostic tool. In addition, we propose a framework for the evaluation of post hoc explainability of web-based prognostic tools. Methods: A SUS- and SCS-based questionnaire was administered amongst pathologists, radiologists, cancer and machine learning researchers and surgeons (n = 11) to evaluate the quality of explanations offered by the machine learning-based web prognostic tool to address the concern of explainability and usability of these models for cancer management. The examined web-based tool was developed by our group and is freely available online. Results: In terms of the usability of the web-based tool using the SUS, 81.9% (45.5% strongly agreed; 36.4% agreed) agreed that neither the support of a technical assistant nor a need to learn many things were required to use the web-based tool. Furthermore, 81.8% agreed that the evaluated web-based tool was not cumbersome to use (usability). The average score for the SCS (explainability) was 0.74. A total of 91.0% of the participants strongly agreed that the web-based tool can assist in clinical decision-making. These scores indicated that the examined web-based tool offers a significant level of usability and explanations about the outcome of interest. Conclusions: Integrating the trained and internally and externally validated model as a web-based tool or calculator is poised to offer an effective and easy approach towards the usage and acceptance of these models in the future daily practice. This approach has received significant attention in recent years. Thus, it is important that the usability and explainability of these models are measured to achieve such touted benefits. A usable and well-explained web-based tool further brings the use of these web-based tools closer to everyday clinical practices. Thus, the concept of more personalized and precision oncology can be achieved

Measuring the Usability and Quality of Explanations of a Machine Learning Web-Based Tool for Oral Tongue Cancer Prognostication

Background: Machine learning models have been reported to assist in the proper management of cancer through accurate prognostication. Integrating such models as a web-based prognostic tool or calculator may help to improve cancer care and assist clinicians in making oral cancer management-related decisions. However, none of these models have been recommended in daily practices of oral cancer due to concerns related to machine learning methodologies and clinical implementation challenges. An instance of the concerns inherent to the science of machine learning is explainability. Objectives: This study measures the usability and explainability of a machine learning-based web prognostic tool that was designed for prediction of oral tongue cancer. We used the System Usability Scale (SUS) and System Causability Scale (SCS) to evaluate the explainability of the prognostic tool. In addition, we propose a framework for the evaluation of post hoc explainability of web-based prognostic tools. Methods: A SUS- and SCS-based questionnaire was administered amongst pathologists, radiologists, cancer and machine learning researchers and surgeons (n = 11) to evaluate the quality of explanations offered by the machine learning-based web prognostic tool to address the concern of explainability and usability of these models for cancer management. The examined web-based tool was developed by our group and is freely available online. Results: In terms of the usability of the web-based tool using the SUS, 81.9% (45.5% strongly agreed; 36.4% agreed) agreed that neither the support of a technical assistant nor a need to learn many things were required to use the web-based tool. Furthermore, 81.8% agreed that the evaluated web-based tool was not cumbersome to use (usability). The average score for the SCS (explainability) was 0.74. A total of 91.0% of the participants strongly agreed that the web-based tool can assist in clinical decision-making. These scores indicated that the examined web-based tool offers a significant level of usability and explanations about the outcome of interest. Conclusions: Integrating the trained and internally and externally validated model as a web-based tool or calculator is poised to offer an effective and easy approach towards the usage and acceptance of these models in the future daily practice. This approach has received significant attention in recent years. Thus, it is important that the usability and explainability of these models are measured to achieve such touted benefits. A usable and well-explained web-based tool further brings the use of these web-based tools closer to everyday clinical practices. Thus, the concept of more personalized and precision oncology can be achieved

DMPonline Version 4.0: User-Led Innovation

DMPonline is a web-based tool to help researchers and research support staff produce data management and sharing plans. Between October and December 2012, we examined DMPonline in unprecedented detail. The results of this evaluation led to some major changes. We have shortened the DCC Checklist for a Data Management Plan and revised how this is used in the tool. We have also amended the data model for DMPonline, improved workflows and redesigned the user interface. This paper reports on the evaluation, outlining the methods used, the results gathered and how they have been acted upon. We conducted usability testing on v.3 of DMPonline and the v.4 beta prior to release. The results from these two rounds of usability testing are compared to validate the changes made. We also put forward future plans for a more iterative development approach and greater community input

Measuring the Usability and Quality of Explanations of a Machine Learning Web-Based Tool for Oral Tongue Cancer Prognostication

Background: Machine learning models have been reported to assist in the proper management of cancer through accurate prognostication. Integrating such models as a web-based prognostic tool or calculator may help to improve cancer care and assist clinicians in making oral cancer management-related decisions. However, none of these models have been recommended in daily practices of oral cancer due to concerns related to machine learning methodologies and clinical implementation challenges. An instance of the concerns inherent to the science of machine learning is explainability. Objectives: This study measures the usability and explainability of a machine learning-based web prognostic tool that was designed for prediction of oral tongue cancer. We used the System Usability Scale (SUS) and System Causability Scale (SCS) to evaluate the explainability of the prognostic tool. In addition, we propose a framework for the evaluation of post hoc explainability of web-based prognostic tools. Methods: A SUS- and SCS-based questionnaire was administered amongst pathologists, radiologists, cancer and machine learning researchers and surgeons (n = 11) to evaluate the quality of explanations offered by the machine learning-based web prognostic tool to address the concern of explainability and usability of these models for cancer management. The examined web-based tool was developed by our group and is freely available online. Results: In terms of the usability of the web-based tool using the SUS, 81.9% (45.5% strongly agreed; 36.4% agreed) agreed that neither the support of a technical assistant nor a need to learn many things were required to use the web-based tool. Furthermore, 81.8% agreed that the evaluated web-based tool was not cumbersome to use (usability). The average score for the SCS (explainability) was 0.74. A total of 91.0% of the participants strongly agreed that the web-based tool can assist in clinical decision-making. These scores indicated that the examined web-based tool offers a significant level of usability and explanations about the outcome of interest. Conclusions: Integrating the trained and internally and externally validated model as a web-based tool or calculator is poised to offer an effective and easy approach towards the usage and acceptance of these models in the future daily practice. This approach has received significant attention in recent years. Thus, it is important that the usability and explainability of these models are measured to achieve such touted benefits. A usable and well-explained web-based tool further brings the use of these web-based tools closer to everyday clinical practices. Thus, the concept of more personalized and precision oncology can be achieved.© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).fi=vertaisarvioitu|en=peerReviewed

An innovative user-centred support tool for Augmented Reality maintenance systems design: a preliminary study

Augmented Reality (AR) technologies in maintenance are demonstrated to positively impact on technicians’ work and performance. Current research in this area is mainly focused on solving technical challenges related with AR in industrial environments. Limited attention has been put into the user perception, ergonomics and usability aspects of AR systems design. This paper proposes an innovative user-centred design support tool for AR systems in maintenance contexts. The tool is based on the Analytic Hierarchy Process (AHP), which is a well-established multi-criteria decision-making approach. In this research, AHP is utilised for guiding designers in the evaluation of application-contexts and AR-technologies for selecting the most suitable and effective AR interaction solution. The tool’s validation has been conducted with twelve maintenance-experts in a design workshop using two case studies. The quantitative results obtained in both case studies reveal the applicability of the AHP model, as well as the effectiveness of the design support tool for complex decisions in AR for maintenance. The use of AHP methods for AR design enable experts to deal with complex and contrasting concepts and express a preference among them with a subjective judgement based on their personal understanding of the problem. Therefore, simplifying the design of AR systems for complex maintenance contexts

Evaluating system utility and conceptual fit using CASSM

There is a wealth of user-centred evaluation methods (UEMs) to support the analyst in assessing interactive systems. Many of these support detailed aspects of use – for example: Is the feedback helpful? Are labels appropriate? Is the task structure optimal? Few UEMs encourage the analyst to step back and consider how well a system supports users’ conceptual understandings and system utility. In this paper, we present CASSM, a method which focuses on the quality of ‘fit’ between users and an interactive system. We describe the methodology of conducting a CASSM analysis and illustrate the approach with three contrasting worked examples (a robotic arm, a digital library system and a drawing tool) that demonstrate different depths of analysis. We show how CASSM can help identify re-design possibilities to improve system utility. CASSM complements established evaluation methods by focusing on conceptual structures rather than procedures. Prototype tool support for completing a CASSM analysis is provided by Cassata, an open source development