Measuring and comparing the reliability of the structured walkthrough evaluation method with novices and experts

Effective evaluation of websites for accessibility remains problematic. Automated evaluation tools still require a significant manual element. There is also a significant expertise and evaluator effect. The Structured Walkthrough method is the translation of a manual, expert accessibility evaluation process adapted for use by novices. The method is embedded in the Accessibility Evaluation Assistant (AEA), a web accessibility knowledge management tool. Previous trials examined the pedagogical potential of the tool when incorporated into an undergraduate computing curriculum. The results of the evaluations carried out by novices yielded promising, consistent levels of validity and reliability. This paper presents the results of an empirical study that compares the reliability of accessibility evaluations produced by two groups (novices and experts). The main results of this study indicate that overall reliability of expert evaluations was 76% compared to 65% for evaluations produced by novices. The potential of the Structured Walkthrough method as a useful and viable tool for expert evaluators is also examined. Copyright 2014 ACM

Use of head camera-cued recall and debrief to externalise expertise: a systematic review of literature from multiple fields of practice

Background: The study of decision making in complex naturalistic environments poses several challenges. In response to these, video-stimulated cued-recall-debrief was developed. It involves an individual wearing a head-mounted camera which records a task from their point of view. Afterwards, footage captured is reviewed along with a facilitated debrief to help externalise cognitive processes. In theory, motion, audio and visual cues generate a high level of experiential immersion which helps the expert to articulate previously hidden thoughts and actions. / Objective: To examine the current evidence for video-stimulated cued-recall-debrief as a means of explicating expert thoughts and feelings in complex tasks in a range of environments. / Study selection: MEDLINE, EMBASE, Education Resources Information Center, SPORTDiscus, PsycINFO and Google Scholar were searched for articles containing the key terms ‘cued-recall (debrief)’, ‘decision making’, ‘skills’ and ‘video recording’. Studies were included if they examined the following outcomes: (1) feasibility, (2) extent of experiential immersion, (3) ability to generate unique insight into decision-making processes and (4) current applications. 1831 articles were identified initially, and 9 studies were included in the final review. / Findings: Video-stimulated cued-recall-debrief is associated with a high level of experiential immersion and generates between two and four times the number of recollections compared with free recall. It can be used to build models of cognitive activity and to characterise the way in which more and less skilled individuals tend to think and feel. / Conclusions: The technique could be used to explicate expertise within medicine: these insights into performance could be used as a training tool for other practitioners

Scoping analytical usability evaluation methods: A case study

Analytical usability evaluation methods (UEMs) can complement empirical evaluation of systems: for example, they can often be used earlier in design and can provide accounts of why users might experience difficulties, as well as what those difficulties are. However, their properties and value are only partially understood. One way to improve our understanding is by detailed comparisons using a single interface or system as a target for evaluation, but we need to look deeper than simple problem counts: we need to consider what kinds of accounts each UEM offers, and why. Here, we report on a detailed comparison of eight analytical UEMs. These eight methods were applied to it robotic arm interface, and the findings were systematically compared against video data of the arm ill use. The usability issues that were identified could be grouped into five categories: system design, user misconceptions, conceptual fit between user and system, physical issues, and contextual ones. Other possible categories such as User experience did not emerge in this particular study. With the exception of Heuristic Evaluation, which supported a range of insights, each analytical method was found to focus attention on just one or two categories of issues. Two of the three "home-grown" methods (Evaluating Multimodal Usability and Concept-based Analysis of Surface and Structural Misfits) were found to occupy particular niches in the space, whereas the third (Programmable User Modeling) did not. This approach has identified commonalities and contrasts between methods and provided accounts of why a particular method yielded the insights it did. Rather than considering measures such as problem count or thoroughness, this approach has yielded insights into the scope of each method

Group versus Individual Web Accessibility Evaluations: Effects with Novice Evaluators

We present an experiment comparing performance of 20 novice evaluators of accessibility carrying out Web Content Accessibility Guidelines 2.0 conformance reviews working individually to performance obtained when they work in teams of two. They were asked to first carry out an individual assessment of a web page. Later on, they were matched randomly to constitute a group of two and they were asked to revise their initial assessment and to produce a group assessment of the same page. Results indicate that significant differences were found for sensitivity (inversely related to false negatives: +8%) and agreement (when measured in terms of the majority view: +10%). Members of groups exhibited strong agreement on the evaluation results among them and with the group outcome. Other measures of validity and reliability are not significantly affected by group work. Practical implications of these findings are that, for example, when it is important to reduce the false-negative rate, then employing a group of two people is more useful than having individuals carrying out the assessment. Openings for future research include further explorations of whether similar results hold for groups larger than two or what is the effect of mixing people with different accessibility background. RESEARCH HIGHLIGHTS When novice accessibility evaluators work in groups, their ability to identify all the true problems increases (by 8%). Likewise, reliability of group evaluations increases (by 10%). Individual or group evaluations can be considered as equivalent methods with respect to false positives (if differences up to 8% in correctness are tolerated). Individual or group evaluations can be considered as equivalent methods with respect to overall effectiveness (if differences up to 11% in F-measure are tolerated)

Measuring Learnability in Human-Computer Interaction

It is well accepted that learnability is a crucial attribute of usability that should be considered in almost every software system. A good learnability leads within a short time and with minimal effort to a high level of proficiency of the user. Therefore, expensive training time of complex systems is reduced. However, there is only few consensus on how to define and evaluate learnability. In addition, gathering detailed information on learnability is quite difficult. In todays books on usability evaluation, learnability gets only few attention, research publications are spread to several other fields and the term learnability is also used in other context. The objective of this thesis is to give an structured overview of learnability and methods for evaluation and additionally assist in the evaluator’s individual choice of an appropriate method. First of all, several definitions of learnability are discussed. For a deeper understanding psychological background knowledge is provided. Afterwards, methods to asses learnability are presented. This comprises nine methods that seem particularly appropriate to measure learnability. As this methods are very diverse, a framework based on analytical hierarchy process is provided. This framework aims to classify presented methods with respect to certain criteria and assess practitioners in selecting an appropriate method to measure learnability

Heuristic evaluation: Comparing ways of finding and reporting usability problems

Research on heuristic evaluation in recent years has focused on improving its effectiveness and efficiency with respect to user testing. The aim of this paper is to refine a research agenda for comparing and contrasting evaluation methods. To reach this goal, a framework is presented to evaluate the effectiveness of different types of support for structured usability problem reporting. This paper reports on an empirical study of this framework that compares two sets of heuristics, Nielsen's heuristics and the cognitive principles of Gerhardt-Powals, and two media of reporting a usability problem, i.e. either using a web tool or paper. The study found that there were no significant differences between any of the four groups in effectiveness, efficiency and inter-evaluator reliability. A more significant contribution of this research is that the framework used for the experiments proved successful and should be reusable by other researchers because of its thorough structur

The playthrough evaluation framework reliable usability evaluation for video games

This thesis presents the playthrough evaluation framework, a novel framework for the reliable usability evaluation of first-person shooter console video games. The framework includes playthrough evaluation, a structured usability evaluation method adapted from heuristic evaluation. Usability evaluation can help guide developers by pointing out design issues that cause users problems. However, usability evaluation methods suffer from the evaluator effect, where separate evaluations of the same data do not produce reliably consistent results. This can result in a number of undesirable consequences affecting issues such as: • Unreliable evaluation: Without reliable results, evaluation reports risk giving incorrect or misleading advice. • Weak methodological validation: Typically new methods (e.g., new heuristics) are validated against user tests. However, without a reliable means to describe observations, attempts to validate novel methods against user test data will also be affected by weak reliability. The playthrough evaluation framework addresses these points through a series of studies presenting the need for, and showing the development of the framework, including the following stages, 1. Explication of poor reliability in heuristic evaluation. 2. Development and validation of a reliable user test coding scheme. 3. Derivation of a novel usability evaluation method, playthrough evaluation. 4. Testing the method, quantifying results. Evaluations were conducted with 22 participants, on 3 first-person shooter action console video games, using two methodologies, heuristic evaluation and the novel playthrough evaluation developed in this thesis. Both methods proved effective, with playthrough evaluation providing more detailed analysis but requiring more time to conduct

Designing Decision Aids for Digital Service Design Technique Selection

Today, people from a multi-disciplinary background are becoming involved in digital service design processes. With the increasing number of digital service design processes in organizations, it is becoming critical to quickly onboard design novices. A huge amount of design techniques is available in digital service design processes. However, too-much-choice makes the selecting process difficult. Thus, selecting appropriate techniques is a challenge, especially for design novices. This dissertation project focuses on providing decision support for design novices to select design techniques in design processes. Several artifacts in the form of different types of classifications and web-based platforms were developed as decision aids seeking to guide novices to select design techniques in digital service design processes. A design science research paradigm was followed, and three design cycles were conducted in the entire dissertation project. Artifacts were developed and evaluated in each design cycle. In design cycle 1, an expert-based taxonomy and a set of novice-based tags were derived. In design cycle 2, the taxonomy and tags were instantiated as decision aids in a web-based platform and evaluated in a lab experiment. The evaluation demonstrated that the expert-based taxonomy outperformed the novice-based tags. In design cycle 3, an extended version of the web-based platform was developed, including a natural language user interface (UI) in combination with the taxonomy to provide selection support for design techniques. The results of the experimental evaluation of cycle 3 demonstrated that novices’ performance using graphical and natural language UI was dependent on decision-making style and duration of use. The work presented in this thesis contributes to the body of knowledge in the field of digital service design. In design cycle 1, the created taxonomy represents a theory for analysis (type I theory). In design cycle 2 and 3, the proposed design principles represent the core of a theory for design and action (type V theory) to guide the design of decision aids for supporting design novices’ selection of design techniques. Besides the theoretical contribution, the developed platform for the selection of design techniques contributes practically to help design novices select design techniques under different design situations