Supporting Worth Mapping with Sentence Completion

Expectations for design and evaluation approaches are set by the development practices within which they are used. Worth Centred Development (WCD) seeks to both shape and fit such practices. We report a study that combined two WCD approaches. Sentence completion gathered credible quantitative data on user values, which were used to identify relevant values and aversions of two player groups for an online gambling site. These values provided human value elements for a complementary WCD approach of worth mapping. Initial worth maps were extended in three workshops, which focused on outcomes and user experiences that could be better addressed in the current product and associated marketing materials. We describe how worth maps were prepared for, and presented in, workshops, and how product owners and associated business roles evaluated the combination of WCD approaches. Based on our experiences, we offer practical advice on this combinination

Developing a coding scheme for detecting usability and fun problems in computer games for young children

This article describes the development and assessment of a coding scheme for finding both usability and fun problems through observations of young children playing computer games during user tests. The proposed coding scheme is based on an existing list of breakdown indication types of the detailed video analysis method (DEVAN). This method was developed to detect usability problems in task-based products for adults. However, the new coding scheme for children's computer games takes into account that in games, fun, in addition to usability, is an important factor and that children behave differently from adults. Therefore, the proposed coding scheme uses 8 of the 14 original breakdown indications and has 7 new indications. The article first discusses the development of the new coding scheme. Subsequently, the article describes the reliability assessment of the coding scheme. The any-two agreement measure of 38.5% shows that thresholds for when certain user behavior is worth coding will be different for different evaluators. However, the any-two agreement of .92 for a fixed list of observation points shows that the distinction between the available codes is clear to most evaluators. Finally, a pilot study shows that training can increase any-two agreement considerably by decreasing the number of unique observations, in comparison with the number of agreed upon observations

Users’ design feedback in usability evaluation: a literature review

Abstract As part of usability evaluation, users may be invited to offer their reflections on the system being evaluated. Such reflections may concern the system’s suitability for its context of use, usability problem predictions, and design suggestions. We term the data resulting from such reflections users’ design feedback. Gathering users’ design feedback as part of usability evaluation may be seen as controversial, and the current knowledge on users’ design feedback is fragmented. To mitigate this, we have conducted a literature review. The review provides an overview of the benefits and limitations of users’ design feedback in usability evaluations. Following an extensive search process, 31 research papers were identified as relevant and analysed. Users’ design feedback is gathered for a number of distinct purposes: to support budget approaches to usability testing, to expand on interaction data from usability testing, to provide insight into usability problems in users’ everyday context, and to benefit from users’ knowledge and creativity. Evaluation findings based on users’ design feedback can be qualitatively different from, and hence complement, findings based on other types of evaluation data. Furthermore, findings based on users’ design feedback can hold acceptable validity, though the thoroughness of such findings may be questioned. Finally, findings from users’ design feedback may have substantial impact in the downstream development process. Four practical implications are highlighted, and three directions for future research are suggested

Putting Value into E-valu-ation

Usability Evaluation measures remain too close to what were originally dependent variables in factorial experiments. The basis for genuine ‘usability problems’ in such variables is not guaranteed, but there has been little progress on finding replacements since HCI’s shift from the laboratory to field studies. As a result, the worth of much usability evaluation is questionable. Such doubts will persist until we can fully align the purpose of evaluation with the purpose of design, which is to create value in the world through innovative products and services, whether sold in markets, or provided free by either individuals or public and voluntary agencies. This chapter reviews issues with common usability measures and introduces a framework that can plausibly re-align evaluation criteria with design purpose by adapting an approach from consumer psychology. This provides opportunities to deploy evaluation measures and instruments that meet the needs of design, rather than reflect skill sets from psychology and human factors. The current gap between design and usability evaluation narrows, but an exclusive usability focus in evaluation becomes impossible. Instead, the role of usability in delivering or degrading intended worth is placed in a wider worth systems context. The maturity of usability will thus be evidenced by its effective integration with a range of design and evaluation concerns. It can longer assume intrinsic importance, but has to demonstrate it in the context of achieved product value