Investigating heuristic evaluation as a methodology for evaluating pedagogical software: An analysis employing three case studies

This paper looks specifically at how to develop light weight methods of evaluating pedagogically motivated software. Whilst we value traditional usability testing methods this paper will look at how Heuristic Evaluation can be used as both a driving force of Software Engineering Iterative Refinement and end of project Evaluation. We present three case studies in the area of Pedagogical Software and show how we have used this technique in a variety of ways. The paper presents results and reflections on what we have learned. We conclude with a discussion on how this technique might inform on the latest developments on delivery of distance learning. © 2014 Springer International Publishing

Evaluating the End-User Experience of Private Browsing Mode

Nowadays, all major web browsers have a private browsing mode. However, the mode's benefits and limitations are not particularly understood. Through the use of survey studies, prior work has found that most users are either unaware of private browsing or do not use it. Further, those who do use private browsing generally have misconceptions about what protection it provides. However, prior work has not investigated \emph{why} users misunderstand the benefits and limitations of private browsing. In this work, we do so by designing and conducting a three-part study: (1) an analytical approach combining cognitive walkthrough and heuristic evaluation to inspect the user interface of private mode in different browsers; (2) a qualitative, interview-based study to explore users' mental models of private browsing and its security goals; (3) a participatory design study to investigate why existing browser disclosures, the in-browser explanations of private browsing mode, do not communicate the security goals of private browsing to users. Participants critiqued the browser disclosures of three web browsers: Brave, Firefox, and Google Chrome, and then designed new ones. We find that the user interface of private mode in different web browsers violates several well-established design guidelines and heuristics. Further, most participants had incorrect mental models of private browsing, influencing their understanding and usage of private mode. Additionally, we find that existing browser disclosures are not only vague, but also misleading. None of the three studied browser disclosures communicates or explains the primary security goal of private browsing. Drawing from the results of our user study, we extract a set of design recommendations that we encourage browser designers to validate, in order to design more effective and informative browser disclosures related to private mode

A first approach to understanding and measuring naturalness in driver-car interaction

With technology changing the nature of the driving task, qualitative methods can help designers understand and measure driver-car interaction naturalness. Fifteen drivers were interviewed at length in their own parked cars using ethnographically-inspired questions probing issues of interaction salience, expectation, feelings, desires and meanings. Thematic analysis and content analysis found five distinct components relating to 'rich physical' aspects of natural feeling interaction typified by richer physical, analogue, tactile styles of interaction and control. Further components relate to humanlike, intelligent, assistive, socially-aware 'perceived behaviours' of the car. The advantages and challenges of a naturalness-based approach are discussed and ten cognitive component constructs of driver-car naturalness are proposed. These may eventually be applied as a checklist in automotive interaction design.This research was fully funded by a research grant from Jaguar Land Rover, and partially funded by project n.220050/F11 granted by Research Council of Norway

Methodological development

Book description: Human-Computer Interaction draws on the fields of computer science, psychology, cognitive science, and organisational and social sciences in order to understand how people use and experience interactive technology. Until now, researchers have been forced to return to the individual subjects to learn about research methods and how to adapt them to the particular challenges of HCI. This is the first book to provide a single resource through which a range of commonly used research methods in HCI are introduced. Chapters are authored by internationally leading HCI researchers who use examples from their own work to illustrate how the methods apply in an HCI context. Each chapter also contains key references to help researchers find out more about each method as it has been used in HCI. Topics covered include experimental design, use of eyetracking, qualitative research methods, cognitive modelling, how to develop new methodologies and writing up your research

Promoting community-based networks through digital media

Digital media can provide an essential contribution to the development of local communities by creating digital environments that facilitate the sharing of information, dissemination of knowledge, and construction of organizational learning. Digital media can also promote interaction and collaboration processes among different regional agents, fostering a continuous flow of territorial-based innovation. The study reported in this paper was carried out within the research activities of the CeNTER project, aiming to deepen the knowledge about the conditions that influence the dynamics of community-led territorial-based innovation in the Centro Region of Portugal, focusing on the areas of Tourism, Health, and Wellbeing. As part of this project, a mobile application is being designed, aiming to strengthen the relationship among agents of local communities/entities and to promote mediation mechanisms among all stakeholders in the process of territorial-based innovation. This paper presents the main concepts underlying the development of the CeNTER application and the outcomes from a usability evaluation of the mobile app prototype. The first section discusses the most relevant topics that this paper covers. In the second section, an overview of the theoretical background in digital media used to promote territory-based innovation is presented. The third section describes the concept of the CeNTER application and the main features of the prototype/mobile app. The fourth section addresses the methodology adopted for evaluating the prototype and the fifth presents the main outcomes of this process. Finally, the sixth and final section presents the main conclusions about the potential of the CeNTER application, and current and future research activities.publishe

Structured computer-based training in the interpretation of neuroradiological images

Computer-based systems may be able to address a recognised need throughout the medical profession for a more structured approach to training. We describe a combined training system for neuroradiology, the MR Tutor that differs from previous approaches to computer-assisted training in radiology in that it provides case-based tuition whereby the system and user communicate in terms of a well-founded Image Description Language. The system implements a novel method of visualisation and interaction with a library of fully described cases utilising statistical models of similarity, typicality and disease categorisation of cases. We describe the rationale, knowledge representation and design of the system, and provide a formative evaluation of its usability and effectiveness

Scoping analytical usability evaluation methods: A case study

Analytical usability evaluation methods (UEMs) can complement empirical evaluation of systems: for example, they can often be used earlier in design and can provide accounts of why users might experience difficulties, as well as what those difficulties are. However, their properties and value are only partially understood. One way to improve our understanding is by detailed comparisons using a single interface or system as a target for evaluation, but we need to look deeper than simple problem counts: we need to consider what kinds of accounts each UEM offers, and why. Here, we report on a detailed comparison of eight analytical UEMs. These eight methods were applied to it robotic arm interface, and the findings were systematically compared against video data of the arm ill use. The usability issues that were identified could be grouped into five categories: system design, user misconceptions, conceptual fit between user and system, physical issues, and contextual ones. Other possible categories such as User experience did not emerge in this particular study. With the exception of Heuristic Evaluation, which supported a range of insights, each analytical method was found to focus attention on just one or two categories of issues. Two of the three "home-grown" methods (Evaluating Multimodal Usability and Concept-based Analysis of Surface and Structural Misfits) were found to occupy particular niches in the space, whereas the third (Programmable User Modeling) did not. This approach has identified commonalities and contrasts between methods and provided accounts of why a particular method yielded the insights it did. Rather than considering measures such as problem count or thoroughness, this approach has yielded insights into the scope of each method

The effects of screen captures in manuals: A textual and two visual manuals compared.

The study examines the use of screen captures in manuals. Three designs of manuals were compared, one textual and two visual manuals. The two visual manuals differed in the type of screen capture that was used. One had screen captures that showed only the relevant part of the screen, whereas the other consisted of captures of the full screen. All manuals contained exactly the same textual information. We examined the time used on carrying out procedures (manual used as a job aid) and the results on retention tests (manual used for learning). We expected to find a trade-off between gain in time and learning effects. That is, we expected that higher scores on the retention tests involved an increase in time used and, vice versa, that gains in time would lead to lower retention test scores. We also explored the influence of manual design on user motivation. For job-aid purposes, there were no differences between manuals. For learning, the full-screen captures manual and the textual manual were significantly better than the partial-screen captures manual. There was no proof for the expected trade-off. More learning was not caused by an increase in time used. We found no effects on user motivation. This study does not yield convincing evidence to support the presence of screen captures in manuals. However, if one wants to include screen captures, this study gives clarity for the type of screen capture to choose. The use of full-screen captures is preferable to partial ones. Finally, we conclude that documentation designed to expedite the execution of tasks does not necessarily hamper the learning that may resul