Reviewing and extending the five-user assumption: A grounded procedure for interaction evaluation

" © ACM, 2013. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Transactions on Computer-Human Interaction (TOCHI), {VOL 20, ISS 5, (November 2013)} http://doi.acm.org/10.1145/2506210 "The debate concerning how many participants represents a sufficient number for interaction testing is well-established and long-running, with prominent contributions arguing that five users provide a good benchmark when seeking to discover interaction problems. We argue that adoption of five users in this context is often done with little understanding of the basis for, or implications of, the decision. We present an analysis of relevant research to clarify the meaning of the five-user assumption and to examine the way in which the original research that suggested it has been applied. This includes its blind adoption and application in some studies, and complaints about its inadequacies in others. We argue that the five-user assumption is often misunderstood, not only in the field of Human-Computer Interaction, but also in fields such as medical device design, or in business and information applications. The analysis that we present allows us to define a systematic approach for monitoring the sample discovery likelihood, in formative and summative evaluations, and for gathering information in order to make critical decisions during the interaction testing, while respecting the aim of the evaluation and allotted budget. This approach – which we call the ‘Grounded Procedure’ – is introduced and its value argued.The MATCH programme (EPSRC Grants: EP/F063822/1 EP/G012393/1

Estimating Sample Size for Usability Testing

One strategy used to assure that an interface meets user requirements is to conduct usability testing. When conducting such testing one of the unknowns is sample size. Since extensive testing is costly, minimizing the number of participants can contribute greatly to successful resource management of a project. Even though a significant number of models have been proposed to estimate sample size in usability testing, there is still not consensus on the optimal size. Several studies claim that 3 to 5 users suffice to uncover 80% of problems in a software interface. However, many other studies challenge this assertion. This study analyzed data collected from the user testing of a web application to verify the rule of thumb, commonly known as the “magic number 5”. The outcomes of the analysis showed that the 5-user rule significantly underestimates the required sample size to achieve reasonable levels of problem detection

The Effectiveness, Usability, and Motivational Characteristics of Using Animated Role- Playing Situational Simulation Programs for Air War College Distance Learning Curriculum

This study examined perceived effectiveness, usability, and motivational characteristics of using animated role playing situational simulations for Air War College distance learning (AWC/DL). The AWC/DL curriculum provides education to senior military officers who are geographically dispersed around the world with varying degrees of internet connectivity, creating specific limitations as to what methods of instruction are viable. An additional challenge is the very high student-to-teacher ratio (620:1). Traditionally, the AWC/DL program relied on text-based readings and tests to teach and evaluate students. Simulations might provide a realistic and valuable augmentation to the curriculum. The key potential advantages of adding simulations are increased motivation and better transfer of learning. However, the key disadvantages are the large expenditure of both time and money to develop simulations. The AWC/DL incorporated their original cultural simulation (OS) into the curriculum in January 2008. A second simulation entitled Visual Expeditionary Skills Training (VEST) was added in 2011 as an alternative to the original simulation. Most research on games and simulations analyze younger groups of students, whereas this research focused on AWC/DL students who are typically in their mid-30s to mid-40s. The geographical diversity, age range of the students, potential benefits from simulations, and high costs for creating simulations all justify research in this area. This study surveyed students who completed either OS or VEST. Students rated perceived effectiveness, usability, and motivation using Likert-scale questions. Motivation questions utilized Keller\u27s (2010) 36-item Instructional Materials Motivation survey. Additionally, completion codes, reflecting choices students made in completing the OS, were analyzed. A total of 1,192 surveys and 2,671 simulation completion codes were analyzed using factor analysis, MANOVAs, stepwise discriminant analysis, and chi squared association analysis. Females generally reported lower levels of video game experience than did their male counterparts. The study found statistical significance between usability and gender as well as between usability and video game experience. Males and experienced video game users seemed to find the simulation more usable. However, the estimated effect size was small (2%). The analysis found no evidence of an interaction between gender and video game experience. The study did find significant associations between gender and the choices made during the simulation. Additionally, 1,871 comments from open-ended questions were analyzed and although there were issues with both simulations, students tended to view simulations as good learning tools. The OS required users to load the software on their computer, resulting in numerous technical issues. Furthermore, the structure of the OS led some students to be caught in an endless loop, resulting in frustration that was specifically cited in 10.1% of surveys. The OS comments provided insight on the importance of how instructions are conveyed, how simulation progress is displayed, and the importance of making controls intuitive or automated. The VEST simulation was internet dependent and received low ratings from students in bandwidth restricted locations. However, those with robust internet connections generally found the simulation engaging and valuable. Both simulations clarified the challenges of using computer simulation in academically isolated and technologically diverse environments

Design And Development of Online Admission System For Alquds Open University in Palestine

Student admissions are playing very important role in major activities of the any university as the basic requirement of the university is students and without students university cannot survive. An inefficient admission application system may reduce the number of admitted student in the esteemed university because if the admission system is slow and having many delays in the process. When considering Palestine students this is unfortunate, but when considering Palestinian international students it can mean the difference between success and failure because of the large sums of money each brings to the university’s economy. This project is to design and develop the under-graduate Palestinian international and local admission process at the University of Alquds Open University to develop an easy to use system that will significantly quicken and simplify this process. The interviews were conducted for the data collection, after that the design was made in UML for meeting the requirement of admission system. This admission system was developed by using JSP and MySQL. The system usefulness, quality of information and quality of interface were evaluated by distributing questionnaire to users and analysis on the data collected from questionnaire was analyzed by using SPSS software

Estimating Sample Size for Usability Testing

One strategy used to assure that an interface meets user requirements is to conduct usability testing. When conducting such testing one of the unknowns is sample size. Since extensive testing is costly, minimizing the number of participants can contribute greatly to successful resource management of a project. Even though a significant number of models have been proposed to estimate sample size in usability testing, there is still not consensus on the optimal size. Several studies claim that 3 to 5 users suffice to uncover 80% of problems in a software interface. However, many other studies challenge this assertion. This study analyzed data collected from the user testing of a web application to verify the rule of thumb, commonly known as the “magic number 5”. The outcomes of the analysis showed that the 5-user rule significantly underestimates the required sample size to achieve reasonable levels of problem detection

Master of Science

thesisThe Residency Review Committee (RRC) requires that general surgery residents document their Surgical Intensive Care Unit (SICU) experiences. To satisfy these requirements we created a web based intranet log to make it easier for residents to track their patients and determine when these requirements were complete. A premium was put on usability to promote acceptance by surgical residents. A prototype web site was designed with input from an attending general surgeon. Three general surgery residents were selected to participate in the iterative design phase. They went through three iterations using a "think-aloud" method while performing tasks on the prototype web site. Each iteration led to improvements to the web site. In a comparison test, a group of seven medical students performed 14 typical web site tasks using both the prototype and the final versions. They were asked to complete a Questionnaire for User Interaction Satisfaction (QUIS) for each version. The time for completion of these tasks was also recorded. The user interaction satisfaction did not show any improvement (F(1,6)=0.13, p=0.912). Similarly, there was no improvement in times for delete and add tasks ( Delete F(1,5) = 0.949, p=0.375, Add F(1,5)=0.267, p=0.628 ); however, the time to complete edit tasks was faster for the final version of the web site (F (1,5)= 14.3, p=0.013). The primary reason for not detecting other differences between the two web sites is likely that the comparison study did not have sufficient power. This was suggested by the participants whose comments favored the final version over the prototype as well as a trend of consistently higher mean subset scores in the final version. The results indicate that differences may be seen when more complex tasks are completed (editing information) versus the two simpler tasks (adding or deleting a patient record in a web site). Future studies should focus on the impact of navigation strategies on speed and data warehouse approaches to creating the application. This study shows the benefits of using an iterative design approach to create a usable web site and demonstrates the importance of further research in the field of usability