Novice evaluators' behavior when consolidating usability problems individually or collaboratively

Publisher's version (útgefin grein)An important, but resource demanding step in analyzing observations from usability evaluations is to consolidate usability problems (UPs) that were identified by several evaluators into one master list. An open question is whether consolidating UPs in pairs is cost-effective. A within-subject study examined if evaluators merge UPs differently when working in pairs than individually and what motivates their decisions. Eight novice evaluators took part. The number of discarded, retained and merged UPs, evaluators' confidence and severity of UPs in the two settings were measured. The results showed that UPs merged or discarded in the collaborative setting would rather be retained in the individual setting. Participants increased confidence and UP severity in the collaborative setting but decreased UP severity and confidence in the individual setting.Peer Reviewe

The playthrough evaluation framework reliable usability evaluation for video games

This thesis presents the playthrough evaluation framework, a novel framework for the reliable usability evaluation of first-person shooter console video games. The framework includes playthrough evaluation, a structured usability evaluation method adapted from heuristic evaluation. Usability evaluation can help guide developers by pointing out design issues that cause users problems. However, usability evaluation methods suffer from the evaluator effect, where separate evaluations of the same data do not produce reliably consistent results. This can result in a number of undesirable consequences affecting issues such as: • Unreliable evaluation: Without reliable results, evaluation reports risk giving incorrect or misleading advice. • Weak methodological validation: Typically new methods (e.g., new heuristics) are validated against user tests. However, without a reliable means to describe observations, attempts to validate novel methods against user test data will also be affected by weak reliability. The playthrough evaluation framework addresses these points through a series of studies presenting the need for, and showing the development of the framework, including the following stages, 1. Explication of poor reliability in heuristic evaluation. 2. Development and validation of a reliable user test coding scheme. 3. Derivation of a novel usability evaluation method, playthrough evaluation. 4. Testing the method, quantifying results. Evaluations were conducted with 22 participants, on 3 first-person shooter action console video games, using two methodologies, heuristic evaluation and the novel playthrough evaluation developed in this thesis. Both methods proved effective, with playthrough evaluation providing more detailed analysis but requiring more time to conduct

Design and evaluation of a list gathering tool in a web-based collaborative environment

This research focuses on how to build a list structure to combine individual items of information into some sort of structure that converts the individual items of information into a structure of knowledge relative to the problem. Software was designed to provide relationships among and comparisons of the contributions in a list structure, so that individual members of a group process will be able to understand the contributions of information made by the group as a whole. A List Gathering Tool was designed and implemented, which is one component in a Web-based Social Decision Support System (SDSS) Toolkit. Then, a two-by-two factorial design (list tool support vs. no list tool support, and voting tool support vs. no voting tool support, respectively) controlled experiment and several field studies were carried out to assess the effects of this List Gathering Tool in a group problem solving process. Overall, the evaluation results are encouraging. The utilization of the List Gathering Tool or the SDSS Toolkit does tend to improve the ability to discover valid alternatives. An additional set of field trials illustrated how the SDSS Toolkit can be utilized in a collaborative learning environment to improve teaching and students\u27 learning experience. This system will also work for very practical applications in large group settings

Thinking about thinking aloud : an investigation of think-aloud methods in usability testing

In website design and engineering, the term “usability” describes how easy a website or interface is to use. As the Internet continues to grow exponentially, with millions of websites vying for users’ attention, usability has become a critical factor determining whether a website will survive or fail. If websites are not sufficiently usable, users will simply abandon them in favour of alternatives that better cater to their needs. It is therefore crucial that designers employ effective evaluation methods in order to assess usability and improve user interface design. One of the most widely used methods of evaluating the usability of websites is the Thinking Aloud protocol, wherein users are encouraged to verbalise their experiences, thoughts, actions, and feelings whilst interacting with the design. This provides direct insight into the cognitive processes employed by users—knowledge which can then inform strategies to improve usability. However, despite the common usage of Thinking Aloud protocol in the field, the specific think-aloud procedures employed vary widely among usability professionals. The aim of this thesis is to investigate the utility and validity of the different variations of think-aloud usability testing methods. To this end, three empirical studies were conducted, using library websites, to compare the practical benefits of the various methods. The studies measured five points of comparison: overall task performance, the experiences of the test participants, the quantity and quality of usability problems discovered, the costs of employing the method in question, and the relationship between sample size and the number of problems detected. Study One examined three classic think-aloud methods: concurrent think-aloud, retrospective think-aloud, and a hybrid method. The results revealed that the concurrent method outperformed both the retrospective method and the hybrid method in facilitating successful usability testing. It detected higher numbers of usability problems than the retrospective method, and produced output comparable to that of the hybrid method. The method received average to positive ratings from its users, and no reactivity (a potential issue wherein the act of verbalising the cognitive process alters that process) was observed. In addition, this method required much less time on the evaluator’s part than did the other two methods, which involved double the testing and analysis time. Lastly, in terms of the relationship between the sample size and the number of problems discovered, the concurrent and the hybrid methods showed similar patterns, and both outperformed the retrospective method in this regard. Study Two compared the performance of the classic concurrent think-aloud method with two variations on this method in which the evaluator plays a more active role—namely, the active intervention method and the speech-communication method. The results showed that these three methods enabled the identification of a similar number of usability problems and types, and showed similar patterns with regard to the relationship between the sample size and the number of problems discovered. However, the active intervention method was found to cause some reactivity, modifying participants’ interactions with the interface, and negatively affecting their feelings towards the evaluator. The active intervention method also required much greater investment than did the other two methods, both in terms of evaluators' time, and, it was estimated, in financial terms. Study Three compared the classic concurrent think-aloud method with the co-participation method, wherein a pair of participants work together to perform their tasks, and verbalise their processes as they interact with the interface and with one another. This study found no difference between the methods in terms of task performance. However, the co-participation method was evaluated more positively by users in comparison with the classic method. It led to the detection of more minor usability problems, and performed better in terms of the relationship between the sample size and the number of problems detected. The co-participation method was, however, found to require a greater investment of time on the part of the evaluato

Design da interação na Web pragmática : reduzindo barreiras semióticas na colaboração mediada pela Web

Orientador: Maria Cecília Calani BaranauskasTese (doutorado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Web e suas tecnologias de base facilitam interações entre pessoas que alguns anos atrás não eram imagináveis. A colaboração é um tipo importante de interação que tem um propósito. Pessoas de diferentes contextos sociais e culturais, e com diferentes preferências e habilidades, podem colaborar mediadas pela Web. A colaboração muitas vezes acontece em contextos heterogêneos, que são definidos tanto pelas situações atuais dos parceiros na colaboração, quanto pelas experiências passadas, sejam elas individuais ou coletivas. A Web como um meio/uma mídia tem um impacto na colaboração e facilita certos aspectos da colaboração enquanto dificulta outros. Adotando uma perspectiva informada pela Web Pragmática, nesta tese investigamos questões da colaboração mediada pela Web, relacionadas com o Design da Interação. Nosso objetivo principal é entender barreiras semióticas da colaboração mediada pela Web e propor uma abordagem ao Design da Interação que reduza tais barreiras. Barreiras semióticas são barreiras relacionadas à comunicação, mediação e representação. Estas barreiras surgem na colaboração mediada pela Web pois muitos mecanismos da comunicação interpessoal face-a-face não estão disponíveis. Dependendo do contexto, barreiras semióticas frequentemente exercem um impacto negativo à colaboração; entretanto, em alguns casos o impacto pode ser positivo também. A abordagem ao Design da Interação aqui proposta tem suas bases na Web Pragmática e utiliza a Semiótica Organizacional e a Teoria da Atividade como referenciais teórico-metodológicos. As investigações teóricas contaram com uma contrapartida em termos de um embasamento em práticas reais através da participação em um projeto de pesquisa no domínio da educação inclusiva. Materializamos a abordagem proposta no design de um protótipo e na implementação de uma ferramenta correspondente ao protótipo, que apoia uma prática de profissionais no domínio da educação inclusiva. Além disso, propusemos e conduzimos um método de avaliação guiada pela pragmática dentro do contexto de um estudo de caso longitudinal. O design do protótipo, a implementação da ferramenta e a avaliação conduzida fornecem evidências de que a abordagem proposta ao Design da Interação guiada pela pragmática contribui para a redução de barreiras semióticas e para a promoção da colaboração mediada pela WebAbstract: The Web and its underlying technologies enable interactions among people that were unimaginable a few years ago. An important type of purposeful interaction is collaboration. Mediated by the Web, people from different social and cultural backgrounds, with different needs, preferences and capabilities can collaborate with each other. Collaboration often takes place in heterogeneous contexts that are not only defined by the actual situations of the collaboration partners, but also by individual and collective past experiences. The Web as a medium has an impact on collaboration and facilitates or enables certain aspects of collaboration while making others more difficult. In this PhD thesis we investigate Interaction Design related questions about web-mediated collaboration under a Pragmatic Web perspective. Our prime objective is to understand semiotic barriers to web-mediated collaboration and propose an approach to Interaction Design that reduces these barriers. Semiotic barriers are barriers related to communication, mediation and representation. These barriers emerge during web-based collaboration since many mechanisms of interpersonal face-to-face communication are not available. Depending on the context, semiotic barriers often have a negative impact on collaboration, but in some cases they might also have positive effects. The approach to Interaction Design proposed in this PhD thesis is rooted in the Pragmatic Web and uses Organizational Semiotics and Activity Theory as its theoretical and methodological frames of reference. The theoretic investigations were practically grounded in real world practices by participating in a research project in the domain of inclusive education. We materialized the proposed approach in the design of a prototype and the implementation of the corresponding tool that supports a practice of inclusive education professionals. Furthermore we proposed and applied a pragmatics-driven evaluation method in a longitudinal case study. Prototype design, tool implementation, and the conducted evaluation provided evidence that the proposed approach to pragmatics-driven Interaction Design can reduce semiotic barriers and thus promote web-mediated collaborationDoutoradoCiência da ComputaçãoDoutor em Ciência da Computaçã

Design revolutions: IASDR 2019 Conference Proceedings. Volume 3: People

In September 2019 Manchester School of Art at Manchester Metropolitan University was honoured to host the bi-annual conference of the International Association of Societies of Design Research (IASDR) under the unifying theme of DESIGN REVOLUTIONS. This was the first time the conference had been held in the UK. Through key research themes across nine conference tracks – Change, Learning, Living, Making, People, Technology, Thinking, Value and Voices – the conference opened up compelling, meaningful and radical dialogue of the role of design in addressing societal and organisational challenges. This Volume 3 includes papers from People track of the conference

Validation of an Observation and Evaluation Instrument for the Supervision of Middle and Secondary Pre-Service Teachers

The purpose of the study was to determine the validity and reliability of a revised observation and evaluation instrument of middle and secondary pre-service clinical teaching to be used as part of the clinical supervision cycle and for formative purposes. The North Texas Appraisal of Classroom Teaching (NTACT) serves as a performance assessment tool utilized by a south-central university-based educator preparation program for the evaluation and supervision of pre-service teachers during their last semester of their program. The researcher piloted and field-tested a redesigned observation and evaluation instrument (NTACT-V2) on observer participants with varying educational experiences in the south-central region. To accumulate evidence of validity and reliability, this study employed methods of factor analysis and generalizability study for developing a valid and reliable instrument to guide the refinement process of the NTACT observation and evaluation instrument. Some of the significant conclusions reached in this study were (a) the NTACT-V2 is a practical, user-friendly classroom observation and evaluation instrument; (b) the instrument refined and developed in this study exhibits appropriate content, face, and criterion validity as determined by a panel of experts and an extensive review of the literature; and, (c) a variety of observers can use the evaluation instrument with relative ease while achieving a high degree of reliability