Toward a document evaluation methodology: What does research tell us about the validity and reliability of evaluation methods?

Although the usefulness of evaluating documents has become generally accepted among communication professionals, the supporting research that puts evaluation practices empirically to the test is only beginning to emerge. This article presents an overview of the available research on troubleshooting evaluation methods. Four lines of research are distinguished concerning the validity of evaluation methods, sample composition, sample size, and the implementation of evaluation results during revisio

User Research of a Voting Machine: Preliminary Findings and Experiences

This paper describes a usability study of the Nedap voting machine in the Netherlands. On the day of the national elections, 566 voters participated in our study immediately after having cast their real vote. The research focused on the correspondence between voter intents and voting results, distinguishing between usability (correspondence between voter intents and voter input) and machine reliability (correspondence between voter input and machine output). For the sake of comparison, participants also cast their votes using a paper ballot.\ud The machine reliability appeared to be 100%, indicating that, within our sample, all votes that had been cast were correctly represented in the output of the voting machine. Regarding usability, 1.4% of the participants had cast the wrong vote using the voting machine. This percentage was similar to that of the paper ballot. \ud Practical implications as well as experiences with this type of usability testing are discussed

Exploring two methods of usability testing: concurrent versus retrospective think-aloud protocols

Think-aloud protocols are commonly used for the usability testing of instructional documents, Web sites and interfaces. This paper addresses the benefits and drawbacks of two think-aloud variations: the traditional concurrent think-aloud method and the less familiar retrospective think-aloud protocols. It also offers an outline of a long-term research project designed to empirically investigate the value of both variants. The results of a first comparative study indicate that, although the two methods have distinct differences, they do seem to produce a similar outcome. A more detailed description of the results will be offered during the presentation

Analyzing the interaction between facilitator and participants in two variants of the think-aloud method.

This paper focuses on the interaction between test participants and test facilitator in two variants of the think-aloud method. In a first, explorative study, we analyzed think-aloud transcripts from two usability tests: a concurrent think-aloud test and a constructive interaction test. The results of our analysis show that while the participants in both studies never explicitly addressed the facilitator, the think-aloud participants showed more signs of awareness of the facilitator than the participants in the constructive interaction test. This finding may have practical implications for the validity of the two methods