Experts evaluation of usability for digital solutions directed at older adults: a scoping review of reviews

Background: it is important to standardize the evaluation and reporting procedures across usability studies to guide researchers, facilitate comparisons, and promote high-quality studies. A first step to standardizing is to have an overview of how experts-based usability evaluation studies are reported across the literature. Objectives: to describe and synthesize the procedures of usability evaluation by experts that are being reported to conduct inspection usability assessments of digital solutions relevant for older adults. Methods: a scoping review of reviews was performed using a five-stage methodology to identify and describe relevant literature published between 2009 and 2020 as follows: i) identification of the research question; ii) identification of relevant studies; iii) select studies for review; iv) charting of data from selected literature; and v) collation, summary, and report of results. The research was conducted on five electronic databases: PubMed, ACM Digital Library, IEEE, Scopus, and Web of Science. The articles that met the inclusion criteria were identified, and data extracted for further analysis, including evaluators, current usability inspection methods, and instruments to support usability inspection methods. Results: a total of 3958 articles were identified. After a detailed screening, 12 reviews matched the eligibility criteria. Conclusion: overall, we found a variety of unstandardized procedures and a lack of detail on some important aspects of the assessment, including a thorough description of the evaluators and of the instruments used to facilitate the inspection evaluation such as heuristics checklists. These findings suggest the need for a consensus framework on the experts’ assessment of usability that informs researchers and allows standardization of procedures.in publicatio

Towards Design Theory for Accessible IT Artefacts

Accessibility in the use of information technology (IT) artefacts, such as websites, applications, and user interfaces, means that they are designed in such a way that people with the broadest range of abilities can use them. However, although accessibility is a human right, IT artefacts often remain inaccessible. Aside from the available accessibility guidelines, we need sufficient design theories that explicitly state how accessibility should be addressed and designed to develop accessible IT artefacts for all users. This dissertation summarises four articles that address this problem. These studies are conducted with qualitative approaches that include a narrative literature review, a systematic literature review and a design science method comprising a participatory design and interviews. The first article develops an explaining theory of accessibility to gain an understanding of the construct of accessibility, showing possible variables of human abilities, tasks and contexts and their relationships in IT use. The second article illustrates the factors in management, development, user, and IT artefact features, including the roles and actions that these domains have and how they affect the realisation of accessibility. The other two articles contribute to accessibility guidance to improve and support content creators’ text production and writing process of accessible online text in the web context. The dissertation underscores three key determinants of the knowledge of accessibility: (1) assumptions of users’ abilities; (2) users’ actual needs; and (3) factors in the development chain. The foregoing factors contribute to the knowledge of accessibility and would help researchers, particularly design scientists, form prescriptive knowledge for practitioners to achieve accessible IT artefacts. Thus, researchers could better identify the variables, relationships and affecting factors in human abilities, management, development, content creation, tasks, and contexts that need to be addressed when designing IT artefacts for certain tasks and use contexts.Informaatioteknologia-artefaktien (IT-artefaktien), kuten verkkosivustojen, sovellusten ja käyttöliittymien saavutettavuus tarkoittaa sitä, että ihmiset erilaisine ominaisuuksineen ja kykyineen voivat käyttää niitä. Vaikka saavutettavuus on ihmisoikeus, IT-artefaktit eivät kuitenkaan ole aina saavutettavia. Käytettävissä olevista saavutettavuusohjeista huolimatta tarvitsemme suunnitteluteorioita, jotka ohjaavat IT-artefaktien suunnittelua, jotta niistä tulisi saavutettavia kaikille IT-artefaktin käyttäjille. Tämä väitöskirja on yhteenveto neljästä artikkelista, jotka käsittelevät tätä ongelmaa. Tutkimukset ovat tehty laadullisilla menetelmillä, joihin on sisältynyt narratiivinen kirjallisuuskatsaus, systemaattinen kirjallisuuskatsaus sekä suunnittelutieteellinen menetelmä sisältäen osallistavan suunnittelun ja haastattelut. Ensimmäisessä artikkelissa kehitetään kuvaileva saavutettavuuden teoria, jolla saadaan käsitys saavutettavuuden rakenteesta ja joka näyttää mahdolliset muuttujat ihmisen kyvyissä, tehtävissä ja konteksteissa, sekä niiden väliset suhteet. Toinen artikkeli kuvaa saavutettavuuteen vaikuttavia tekijöitä johtamisen, kehityksen, käyttäjän ja IT-artefaktin ominaisuuksien näkökulmista, mukaan lukien roolit ja toimenpiteet, joita näillä kohteilla on. Kaksi muuta artikkelia kehittävät ohjeistuksen sisällöntuottajien työn tueksi saavutettavan verkkotekstin tuottamiseksi. Väitöskirjassa esitetään kolme ratkaisevaa tekijää saavutettavuuden tietämyksessä: (1) olettamukset käyttäjien kyvyistä (2) käyttäjien todelliset tarpeet ja (3) tekijät kehitysketjussa. Näiden tekijöiden tuntemus auttaa erityisesti suunnittelutieteilijöitä muodostamaan ohjaavaa tietoa ammattilaisille saavutettavien IT-artefaktien saavuttamiseksi. Täten tutkijat voivat paremmin tunnistaa muuttujat, niiden väliset suhteet ja saavutettavuuteen vaikuttavat tekijät, jotka liittyvät käyttäjän kykyihin, johtamiseen, kehittämiseen, sisällöntuottamiseen, tehtäviin ja kontekstiin, kun IT-artefaktia suunnitellaan tiettyä tehtävää ja käyttökontekstia varten.fi=vertaisarvioitu|en=peerReviewed

Framework for Guiding the Development of High-Quality Conversational Agents in Healthcare

Evaluating conversational agents (CAs) that are supposed to be applied in healthcare settings and ensuring their quality is essential to avoid patient harm and ensure efficacy of the CA-delivered intervention. However, a guideline for a standardized quality assessment of health CAs is still missing. The objective of this work is to describe a framework that provides guidance for development and evaluation of health CAs. In previous work, consensus on categories for evaluating health CAs has been found. In this work, we identify concrete metrics, heuristics, and checklists for these evaluation categories to form a framework. We focus on a specific type of health CA, namely rule-based systems that are based on written input and output, have a simple personality without any kind of embodiment. First, we identified relevant metrics, heuristics, and checklists to be linked to the evaluation categories through a literature search. Second, five experts judged the metrics regarding their relevance to be considered within evaluation and development of health CAs. The final framework considers nine aspects from a general perspective, five aspects from a response understanding perspective, one aspect from a response generation perspective, and three aspects from an aesthetics perspective. Existing tools and heuristics specifically designed for evaluating CAs were linked to these evaluation aspects (e.g., Bot usability scale, design heuristics for CAs); tools related to mHealth evaluation were adapted when necessary (e.g., aspects from the ISO technical specification for mHealth Apps). The resulting framework comprises aspects to be considered not only as part of a system evaluation, but already during the development. In particular, aspects related to accessibility or security have to be addressed in the design phase (e.g., which input and output options are provided to ensure accessibility?) and have to be verified after the implementation phase. As a next step, transfer of the framework to other types of health CAs has to be studied. The framework has to be validated by applying it during health CA design and development

Establishing usability heuristics for heuristics evaluation in a specific domain: is there a consensus?

Heuristics evaluation is frequently employed to evaluate usability. While general heuristics are suitable to evaluate most user interfaces, there is still a need to establish heuristics for specific domains to ensure that their specific usability issues are identified. This paper presents a comprehensive review of 70 studies related to usability heuristics for specific domains. The aim of this paper is to review the processes that were applied to establish heuristics in specific domains and identify gaps in order to provide recommendations for future research and area of improvements. The most urgent issue found is the deficiency of validation effort following heuristics proposition and the lack of robustness and rigour of validation method adopted. Whether domain specific heuristics perform better or worse than general ones is inconclusive due to lack of validation quality and clarity on how to assess the effectiveness of heuristics for specific domains. The lack of validation quality also affects effort in improving existing heuristics for specific domain as their weaknesses are not addressed

Designing and evaluating the usability of a machine learning API for rapid prototyping music technology

To better support creative software developers and music technologists' needs, and to empower them as machine learning users and innovators, the usability of and developer experience with machine learning tools must be considered and better understood. We review background research on the design and evaluation of application programming interfaces (APIs), with a focus on the domain of machine learning for music technology software development. We present the design rationale for the RAPID-MIX API, an easy-to-use API for rapid prototyping with interactive machine learning, and a usability evaluation study with software developers of music technology. A cognitive dimensions questionnaire was designed and delivered to a group of 12 participants who used the RAPID-MIX API in their software projects, including people who developed systems for personal use and professionals developing software products for music and creative technology companies. The results from the questionnaire indicate that participants found the RAPID-MIX API a machine learning API which is easy to learn and use, fun, and good for rapid prototyping with interactive machine learning. Based on these findings, we present an analysis and characterization of the RAPID-MIX API based on the cognitive dimensions framework, and discuss its design trade-offs and usability issues. We use these insights and our design experience to provide design recommendations for ML APIs for rapid prototyping of music technology. We conclude with a summary of the main insights, a discussion of the merits and challenges of the application of the CDs framework to the evaluation of machine learning APIs, and directions to future work which our research deems valuable

Accessible charts are part of the equation of accessible papers: a heuristic evaluation of the highest impact LIS Journals

Purpose Statistical charts are an essential source of information in academic papers. Charts have an important role in conveying, clarifying and simplifying the research results provided by the authors, but they present some accessibility barriers for people with low vision. This article aims to evaluate the accessibility of the statistical charts published in the library and information science (LIS) journals with the greatest impact factor. Design/methodology/approach A list of heuristic indicators developed by the authors has been used to assess the accessibility of statistical charts for people with low vision. The heuristics have been applied to a sample of charts from 2019 issues of ten LIS journals with the highest impact factor according to the ranking of the JCR. Findings The current practices of image submission do not follow the basic recommended guidelines on accessibility like color contrast or the use of textual alternatives. On the 2 other hand, some incongruities between the technical suggestions of image submission and their application in analyzed charts also emerged. The main problems identified are: poor text alternatives, insufficient contrast ratio between adjacent colors, and the inexistence of customization options. Authoring tools do not help authors to fulfill these requirements. Research limitations The sample is not very extensive; nonetheless, it is representative of common practices and the most frequent accessibility problems in this context. Social implications The heuristics proposed are a good starting point to generate guidelines for authors when preparing their papers for publication and to guide journal publishers in creating accessible documents. Low vision users, a highly prevalent condition, will benefit from the improvements. Originality/value The results of this research provide key insights into low vision accessibility barriers, not considered in previous literature and can be a starting point for their solution.This research has been done in the framework of the PhD Programme in Engineering and Information Technology of the Universitat de Lleida (UdL). This work has been partially supported by the Spanish project PID2019-105093GB-I00 (MINECO/FEDER, UE) and CERCA Programme/Generalitat de Catalunya

Academic methods for usability evaluation of serious games: a systematic review

In the last years, there has been an increasing interest in the design of video games as a tool for education, training, health promotion, socialization, etc. Usability, which is a key factor in any video game, becomes even more important in these so-called Bserious games^, where the users’ special characteristics should be considered, and the game efficacy depends on the users’ adherence and engagement. However, evaluation of the usability of this kind of games requires a redefinition of techniques, methods and even terminology. In this paper, we elicit six research questions and conduct a systematic review of the scientific literature, which resulted in the selection of 187 papers that contained the most relevant responses. The conclusions of this systematic review illustrate the general status of current academic usability evaluations of these games and the main trends in the selection of methodologies and how are they applied. This view may be a very valuable foundation for future researchMinisterio de Ciencia e Innovación PROCUR@-IPT-2011-1038-90000

Youth and Digital Media: From Credibility to Information Quality

Building upon a process-and context-oriented information quality framework, this paper seeks to map and explore what we know about the ways in which young users of age 18 and under search for information online, how they evaluate information, and how their related practices of content creation, levels of new literacies, general digital media usage, and social patterns affect these activities. A review of selected literature at the intersection of digital media, youth, and information quality -- primarily works from library and information science, sociology, education, and selected ethnographic studies -- reveals patterns in youth's information-seeking behavior, but also highlights the importance of contextual and demographic factors both for search and evaluation. Looking at the phenomenon from an information-learning and educational perspective, the literature shows that youth develop competencies for personal goals that sometimes do not transfer to school, and are sometimes not appropriate for school. Thus far, educational initiatives to educate youth about search, evaluation, or creation have depended greatly on the local circumstances for their success or failure

Designing Heuristics for Accessible Online Text Production

Governments and public organisations provide digital services and share information on websites, so web content needs to be accessible to all citizens. Text remains the main form of providing information, and reading is the primary way to interact with digital services. However, existing guidelines are not adequate for content creators in public organisations. The wide scope and technicality of these guidelines make them confusing, difficult to understand and challenging to implement. To respond to this emerging need, in this paper, we contribute improvements to the guidance of accessible text production by proposing heuristics with a design science approach. Specifically, we (1) review accessibility guidelines and determine improvement factors related to text accessibility, (2) establish a design and evaluation workshop with 38 students, and (3) verify the feasibility of the proposal with content creators. Our evidence shows that the proposed accessibility heuristics are clear and easy to understand, and they are useful for content creators.© 2022 Scandinavian Journal of Information Systems & Authors.fi=vertaisarvioitu|en=peerReviewed

Heuristic Evaluation and Usability Testing of G-MoMo Applications

Financial technology (FinTech) has swiftly revolutionized mobile money as one of the ways of accessing financial services in developing countries. Numerous mobile money applications were developed to access mobile money services but are hindered by severe authentication security challenges, thus, forcing the researchers to design a secure multi-factor authentication (MFA) algorithm for mobile money applications. Three prototypes of native mobile money applications (G-MoMo applications) were developed to confirm that the algorithm provides high security and is feasible. This study, therefore, aimed to evaluate the usability of the G-MoMo applications using heuristic evaluation and usability testing to identify potential usability issues and provide recommendations for improvement. Heuristic evaluation and usability testing methods were used to evaluate the G-MoMo applications. The heuristic evaluation was carried out by five experts that used the 10 principles proposed by Jakob Nielsen with a five-point severity rating scale to identify the usability problems. While the usability testing was conducted with forty participants selected using a purposive sampling method to validate the usability of the G-MoMo applications by performing tasks and filling out the post-test questionnaire. Data collected were analyzed in RStudio software. Sixty-three usability issues were identified during heuristic evaluation, where 33 were minor and 30 were major. The most violated heuristic items were “help and documentation”, and “user control and freedom”, while the least violated heuristic items were “aesthetic and minimalist design” and “visibility of system status”. The usability testing findings revealed that the G-MoMo applications’ performance proved good in learnability, effectiveness, efficiency, memorability, and errors. It also provided user satisfaction, ease of use, aesthetics, usefulness, integration, and understandability. Therefore, it was highly recommended that the developers of G-MoMo applications fix the identified usability problems to make the applications more reliable and increase overall user satisfaction.info:eu-repo/semantics/publishedVersio