We Need Numbers! - Heuristic Evaluation during Demonstrations (HED) for Measuring Usability in IT System Procurement

We introduce a new usability inspection method called HED (heuristic evaluation during demonstrations) for measuring and comparing usability of competing complex IT systems in public procurement. The method presented enhances traditional heuristic evaluation to include the use context, comprehensive view of the system, and reveals missing functionality by using user scenarios and demonstrations. HED also quantifies the results in a comparable way. We present findings from a real-life validation of the method in a large-scale procurement project of a healthcare and social welfare information system. We analyze and compare the performance of HED to other usability evaluation methods used in procurement. Based on the analysis HED can be used to evaluate the level of usability of an IT system during procurement correctly, comprehensively and efficiently.Peer reviewe

Retrospective protocols in usability testing: a comparison of Post-session RTA versus Post-task RTA reports

We present the results of a study that compared two placements of the Retrospective Think-aloud (RTA): A Post-session RTA where the think-aloud occurs after all tasks are complete, and a Post-task RTA where the think-aloud is elicited after each task. Data from task performance and verbal measures were collected from 24 participants. The results suggest that in terms of task performance, participants in the Post-session RTA condition performed tasks faster, with fewer errors and fewer clicks than in the Post-task RTA condition. In terms of utterances, participants in the Post-task RTA condition produced significantly more utterances that explained actions, expectations and procedural descriptions than in the Post-session RTA condition

Understanding Advice Sharing among Physicians: Towards Trust-Based Clinical Alerts

Safe prescribing of medications relies on drug safety alerts, but up to 96% of such warnings are ignored by physicians. Prior research has proposed improvements to the design of alerts, but with limited increase in adherence. We propose a different perspective: before re-designing alerts, we focus on improving the trust between physicians and computerized advice by examining why physicians trust their medical colleagues. To understand trusted advice among physicians, we conducted three contextual inquiries in a hospital setting (22 participants), and corroborated our findings with a survey (37 participants). Drivers that guide physicians in trusting peer advice include: timeliness of the advice, collaborative language, empathy, level of specialization and medical hierarchy. Based on these findings, we introduce seven design directions for trust-based alerts: endorsement, transparency, team sensing, collaborative, empathic, conflict mitigating and agency laden. Our work contributes to novel alert design strategies to improve the effectiveness of drug safety advice

Combinations of methods for collaborative evaluation of the usability of interactive software systems

Usability is a fundamental quality characteristic for the success of an interactive system. It is a concept that includes a set of metrics and methods in order to obtain easy-to-learn and easy-to-use systems. Usability Evaluation Methods, UEM, are quite diverse; their application depends on variables such as costs, time availability, and human resources. A large number of UEM can be employed to assess interactive software systems, but questions arise when deciding which method and/or combination of methods gives more (relevant) information. We propose Collaborative Usability Evaluation Methods, CUEM, following the principles defined by the Collaboration Engineering. This paper analyzes a set of CUEM conducted on different interactive software systems. It proposes combinations of CUEM that provide more complete and comprehensive information about the usability of interactive software systems than those evaluation methods conducted independentl

Development and Assessment of User Interface for Security-Critical Systems

In this master's thesis a user interface for a security application was investigated. The thesis work was done at a security company in Sweden. The goal was to find current problems with the user interface and to develop a new prototype with improvements. A user-centered design approach was employed to achieve this. Another goal was to look at how this method works for security-critical systems and how usability affects security. By conducting a usability test, usability errors were found that affected information security. There were also other problems that arose when having to deal with trying to use a user-centered design approach when working for a security company

Ordinary user experiences at work: a study of greenhouse growers

We investigate professional greenhouse growers’ user experience (UX) when using climate-management systems in their daily work. We build on the literature on UX, in particular UX at work, and extend it to ordinary UX at work. In a ten-day diary study, we collected data with a general UX instrument (AttrakDiff), a domain-specific instrument, and interviews. We find that AttrakDiff is valid at work; its three-factor structure of pragmatic quality, hedonic identification quality, and hedonic stimulation quality is recognizable in the growers’ responses. In this paper, UX at work is understood as interactions among technology, tasks, structure, and actors. Our data support the recent proposal for the ordinariness of UX at work. We find that during continued use UX at work is middle-of-the-scale, remains largely constant over time, and varies little across use situations. For example, the largest slope of the four AttrakDiff constructs when regressed over the ten days was as small as 0.04. The findings contrast existing assumptions and findings in UX research, which is mainly about extraordinary and positive experiences. In this way, the present study contributes to UX research by calling attention to the mundane, unremarkable, and ordinary user experiences at work

Attractiveness and Learnability to Support Operability in Web Applications

One of software quality criteria that is vital to determine the success of a software system is usability (ISO/IEC 9126-1:2001), also known as operability (ISO/IEC 25010:2011). There are a few sub-criteria that support operability and two of them are attractiveness and learnability. There is still lack of systematic review with regard to usability or operability with the focus on attractiveness and learnability mainly in Web applications. As more software systems nowadays are webbased, studying these quality factors are indeed essential. This study adopts a systematic literature review method to investigate existing works on the two sub-criteria besides exploring the works in both usability and operability in Web applications in general. The results specifically examine the issues, strengths and weaknesses that also conclude the gaps in existing works on attractiveness and learnability in Web applications besides the focus on existing frameworks

Identifying and explicating knowledge on method transfer: a sectoral system of innovation approach

Post-print (lokagerð höfundar)With the advances in information technology and its increasing impact on humans and society, there has been an expanding need to spread knowledge from domain to domain. This need is not least in the area of human–computer interaction, which includes a rich culture of carrying out usability evaluations in many different domains and technology platforms. This paper aims to show how transfer of methods takes place, by explicating and formalizing the process. It will contribute to the quest for knowledge on the constituents of the process of transferring methodological knowledge and their relationships. A sectoral system of innovation approach is used to analyse the constituents of a selected sector, crisis management, where training is essential and which is rapidly adopting technology for operations and training. Two case studies are described where heuristics evaluation and user testing were applied on simulation software that allows training for crisis management. The analysis results in a process model, describing the transfer of methodological knowledge within the sectoral system of innovation framework.The research leading to these results has received funding from COST Action IC0904, Twintide, and the European Union Seventh Framework Programme (FP7/2007-2013) under Grant Agreement No. [FP7-242474], CRISIPeer Reviewe