Exploring two methods of usability testing: concurrent versus retrospective think-aloud protocols

Think-aloud protocols are commonly used for the usability testing of instructional documents, Web sites and interfaces. This paper addresses the benefits and drawbacks of two think-aloud variations: the traditional concurrent think-aloud method and the less familiar retrospective think-aloud protocols. It also offers an outline of a long-term research project designed to empirically investigate the value of both variants. The results of a first comparative study indicate that, although the two methods have distinct differences, they do seem to produce a similar outcome. A more detailed description of the results will be offered during the presentation

Analyzing the interaction between facilitator and participants in two variants of the think-aloud method.

This paper focuses on the interaction between test participants and test facilitator in two variants of the think-aloud method. In a first, explorative study, we analyzed think-aloud transcripts from two usability tests: a concurrent think-aloud test and a constructive interaction test. The results of our analysis show that while the participants in both studies never explicitly addressed the facilitator, the think-aloud participants showed more signs of awareness of the facilitator than the participants in the constructive interaction test. This finding may have practical implications for the validity of the two methods

Hardopdenkprotocollen als pretestmethode : synchroon en retrospectief hardopdenken vergeleken

Hardopdenkprotocollen zijn een veelgebruikte methode voor de formatieve evaluatie van software, interfaces, websites en instructieve documenten. De methode houdt in dat gebruikers uit de doelgroep een aantal taken met behulp van het te evalueren communicatiemiddel uitvoeren en daarbij voortdurend hun gedachten verbaliseren. De methode heeft face validity, omdat de data die ermee verkregen worden het feitelijke gebruik weerspiegelen, en dus verder gaan dan het oordeel van proefpersonen over de gebruikersvriendelijkheid. Hardopdenkonderzoek is ingebed in een lange en gerespecteerde onderzoekstraditie, gericht op de cognitieve processen van proefpersonen tijdens de uitvoering van een breed scala aan taken – zoals schaken, probleemoplossen, schrijven, lezen en besluitvorming – met de monografie van Ericsson & Simon (1993) als belangrijke mijlpaal. Wanneer hardopdenken wordt gebruikt als pretestmethode, gaat het echter niet primair om inzicht in de cognitieve processen, maar om zicht op de kwaliteit van communicatiemiddelen of artefacten

Hardopdenkprotocollen en gebruikersonderzoek ; volledigheid en reactiviteit van de synchrone hardopdenkmethode

De hardopdenkmethode heeft zich ontwikkeld tot een gangbare onderzoeksmethode voor uiteenlopend lees- en schrijfonderzoek. In dit artikel staat de validiteit van de hardopdenkmethode als pretestinstrument centraal. Discussies over de validiteit van hardopdenkprotocollen betreffen de reactiviteit van de methode en de volledigheid van de verbalisaties. De vraag wordt beantwoord in hoeverre de aard van het testobject van invloed is op de reactiviteit van de methode en de volledigheid van de protocollen. Daartoe zijn synchrone en retrospectieve hardopdenkprotocollen vergeleken voor twee typen internet-applicaties: een online bibliotheekcatalogus en een gemeentelijke website. In beide gevallen blijken synchrone hardopdenkprotocollen minder geverbaliseerde problemen te bevatten dan retrospectieve protocollen. De reactiviteit van de methode blijkt samen te hangen met het testobject: in de bibliotheekcatalogus leidde de opdracht om synchroon hardop te denken tot meer observeerbare fouten in de taakuitvoering, op de gemeentelijke website niet. Deze verschillen kunnen worden herleid tot verschillen in taken tussen de beide testobjecten

Evaluation of an Informational Web Site: Three Variants of the Think-aloud Method Compared

To evaluate Web sites, usability experts often use methods that were originally employed for the evaluation of software applications. In doing so, they assume that these methods will work exactly the same for both types of test objects. However, there is a major difference between transactional software applications and informational Web sites, a difference that could have an effect on the workings of various usability methods. As such, we felt that it was valuable to repeat one of our previous studies in which we compared concurrent think-aloud protocols, retrospective think-aloud protocols, and constructive interaction to evaluate a Web application, this time using a Web site. The results of our study showed that in some respects, the methods did work differently depending on the test object they were applied to. However, we conclude that the three methods are largely interchangeable and that the decision to choose one variant of the think-aloud method over the other should be based on practical considerations

Evaluation of an informational web site: Three variants of the think-aloud method compared

Questions the assumption that usability methods are equally suitable for evaluating Web sites and Web applications Concludes that the decision to choose one variant of the think-aloud method over another should be based on practical considerations

Constructive interaction: An analysis of verbal interaction in a usability setting

This paper focuses on the interaction between teams of participants in the constructive interaction (CI) method. We analyzed transcripts and video recordings from five CI sessions in order to determine the types and frequencies of communicative acts performed as well as their usefulness to usability testers. In addition, we examined the contribution of the individual team members to the interaction and investigated whether the interaction took place according to a pattern of cooperation. Our analysis showed that team interaction in the CI test is highly task-oriented and hardly descriptive. For the most part, the five CI sessions contained communicative acts that are (potentially) useful to usability testers. The contribution of the team members to the interaction was largely similar in terms of types/frequencies of communicative acts, but the teams' cooperation pattern revealed that this equal contribution was no immediate guarantee for successful cooperation. To address this issue, we suggest several ways to improve the CI method

Does think aloud work? How do we know?

The think aloud method is widely used in usability research to collect user's reports of the experience of interacting with a design so that usability evaluators can find the underlying usability problems. However, concerns remain about the validity and usefulness of think aloud in usability studies. In this panel we will present current studies of the think aloud method, examine and question its usage in the field, discuss the possible pitfalls that may threaten the validity of the method, and provide comments/suggestions on the application of the method. Panel participants will discuss results drawn from both applied research and basic research. We believe that this panel discussion will be useful for HCI designers and usability practitioners in that it will acquaint them with concerns that people have about the think aloud method and provide them with suggestions for improved use of the method. For HCI or usability researchers, this panel discussion will address the importance of formally investigating currently used or newly designed usability methods