Remote Usability Testing Methods a la Carte

Although existing lab-based formative usability testing is frequently and effectively applied to improving usability of software user interfaces, it has limitations that have led developers to turn to remote usability evaluation methods (RUEMs) to collect formative usability data from daily usage by real users in their own real-world task environments. The enormous increase in Web usage, where users can be isolated and the network and remote work settingbecome intrinsic parts of usage patterns, is strong motivation for supplementing lab-based testing with remote usability evaluation methods. Another significant impetus for remote evaluation is the fact that the iterative development cycle for any software, Web application or not, does not end with initial deployment. We review and informally compare several approaches to remote usability evaluation with respect to quantity and quality of data collected and the effort to collect the data

Improving Deaf Accessibility in Remote Usability Testing

For studies involving Deaf participants in United States, remote usability testing has several potential advantages over face-to-face testing, including convenience, lower cost and the ability to recruit participants from diverse geographic regions. However, current technologies force Deaf participants to use English instead of their preferred language, which is American Sign Language (ASL). A new remote testing technology allows researchers to conduct studies exclusively in ASL at a lower cost than face-toface testing. The technology design facilitates open-ended questions and is reconfigurable for use in a variety of studies. Results from usability tests of the tool are encouraging and a fullscale study is underway to compare this approach to face-to-face testing

Enhancing the Informatics Evaluation Toolkit with Remote Usability Testing

Developing functional clinical informatics products that are also usable remains a challenge. Despite evidence that usability testing should be incorporated into the lifecycle of health information technologies, rarely does this occur. Challenges include poor standards, a lack of knowledge around usability practices, and the expense involved in rigorous testing with a large number of users. Remote usability testing may be a solution for many of these challenges. Remotely testing an application can greatly enhance the number of users who can iteratively interact with a product, and it can reduce the costs associated with usability testing. A case study presents the experiences with remote usability testing when evaluating a Web site designed for health informatics knowledge dissemination. The lessons can inform others seeking to enhance their evaluation toolkits for clinical informatics products

Remote Usability Testing Using Eyetracking

Abstract. In the paper we present a low cost method of using eyetracking to perform remote usability tests on users. Remote usability testing enables to test users in their natural environment. Eyetracking is one of the most popular techniques for usability testing in the laboratory environment. We decided to try to use this technique in remote tests. We used standard web camera with freeware software. Our experiment showed that such method is not perfect, but it could be a good addition to the standard remote tests, and a foundation for further development

The aud one out in the final battle : an Anya-centered feminist analysis of Buffy the vampire slayer

Current usability testing is often conducted via face-to-face interactions. This method can be costly, both in terms of timelines and budget. However, remote usability testing has been shown to be a viable alternative, in that performance scores have been shown to be quite similar to face-to-face methods. Although performance appears similar, remote usability testing may present challenges that threaten the validity and reliability of usability testing results. Rather than focusing on the performance of users in remote versus co-located conditions, the proposed study investigates the emotional and attitudinal responses of users engaged in software usability tests. The purpose of this study was to compare users’ anxiety and satisfaction with communication in remote and face-to-face usability tests. It was hypothesized that participants in the remote condition would exhibit a lower level of anxiety and be less satisfied with the communication method. Multiple usability tasks were administered and measures were recorded at three time intervals. Responses on the Social Anxiety Thoughts (SAT) questionnaire and the Communication Satisfaction Inventory (CSI) were collected. Although there were no significant differences between the groups in terms of anxiety and communication satisfaction, methodological limitations may have prevented the detection of differences and additional research is required to explore the strengths and weaknesses of remote usability testing

Emotional and attitudinal responses to remote versus co-located usability testing

Current usability testing is often conducted via face-to-face interactions. This method can be costly, both in terms of timelines and budget. However, remote usability testing has been shown to be a viable alternative, in that performance scores have been shown to be quite similar to face-to-face methods. Although performance appears similar, remote usability testing may present challenges that threaten the validity and reliability of usability testing results. Rather than focusing on the performance of users in remote versus co-located conditions, the proposed study investigates the emotional and attitudinal responses of users engaged in software usability tests. The purpose of this study was to compare users’ anxiety and satisfaction with communication in remote and face-to-face usability tests. It was hypothesized that participants in the remote condition would exhibit a lower level of anxiety and be less satisfied with the communication method. Multiple usability tasks were administered and measures were recorded at three time intervals. Responses on the Social Anxiety Thoughts (SAT) questionnaire and the Communication Satisfaction Inventory (CSI) were collected. Although there were no significant differences between the groups in terms of anxiety and communication satisfaction, methodological limitations may have prevented the detection of differences and additional research is required to explore the strengths and weaknesses of remote usability testing

Search engine UIs: remote usability testing with blind persons

This paper describes a remote usability testing which was the final phase of a research project aimed at improving usability of web search tools for blind users who interact via screen reader and voice synthesizer. The testing aimed to evaluate a new implementation of Google user interfaces - according to a set of criteria previously proposed specifically for search engine user interfaces - for the simple search and the result exploration. To prepare the environment for the remote testing we needed to re-implement the original Google interfaces, using Google APIs, PERL programming and XSLT transformations. The results of the testing highlight how Google interfaces, although accessible, may be further improved in order to simplify the interaction for the sightless. In this article, first an overview of the project is introduced; then we discuss the design and implementation of the UIs. Finally, we describe in detail the usability testing which involved 12 totally blind persons

An Experiential comparative analysis of two remote usability testing methods

Remote usability testing is a key tool for usability professionals. Several remote methods exist and it is often difficult to choose the appropriate method. Testing lower-fidelity prototypes often present unique problems because they provide minimally aesthetic and minimally interactive partial representations of a final product. This qualitative and experiential pilot study is an attempt to compare a remote synchronous (RS) usability testing method where the moderator and participant are displaced by space, and a remote asynchronous (RA) usability testing method where the moderator and participant are displaced by both time and space. An important byproduct of the comparison is the creation of a low-cost, online asynchronous testing mechanism. The results show that the usability issues and participant experience vary between remote synchronous and asynchronous usability testing. While the remote asynchronous method does not require a test moderator and participants find it more convenient to complete the test whenever he or she chooses, participants may require clarification on tasks and usability issues discovered during the test. Participants are also critical of the remote asynchronous method and sometimes find it difficult to complete an entire session on his or her own time without anyone to guide them. Further research is needed to validate the results using a more controlled methodology

Remote Usability Testing - A New Approach Facilitated By Virtual Worlds

Synchronous remote usability testing, involves a facilitator conducting a usability test in real time, interacting with a participant who is remote. This study proposes a new methodology for conducting these studies using a three-dimensional virtual world, Wonderland, and compares it with two other commonly used synchronous usability test methods: the traditional lab approach and WebEx, a web-based conferencing and screen sharing approach. The study involved 48 participants in total, 36 test subjects and 12 test facilitators. These 36 were equally divided among the three environments with the 12 test facilitators being paired with one participant in each of the environments. The participants completed 5 tasks on an e-commerce website. The three methodologies were compared with respect to the dependent variables, the time taken to complete the tasks; the usability defects identified; the severity of these usability issues; and the subjective ratings from the NASA-TLX, the presence and post-test subjective questionnaires. Most importantly, the three methodologies agreed closely in terms of the total number defects identified, number of high severity defects identified and the time taken to complete the tasks. However, there was a significant difference in the workload experienced by the test participants and facilitators, with the traditional lab condition being the least and the Wonderland and the WebEx conditions being almost the same. It was also found that both test participants and test facilitators experienced better involvement and immersive experiences in the Wonderland condition, than the WebEx condition and almost the same for traditional lab condition. The results of this study suggest that participants were productive and enjoyed the Wonderland condition, indicating the potential of a virtual world based approach as an alternative to the conventional approaches