Usability problem reports for comparative studies: consistency and inspectability

The present study explores issues of consistency and inspectability in usability test data analysis processes and reports. Problem reports resulting from usability tests performed by three professional usability labs in three different countries are compared. Each of the labs conducted a usability test on the same product, applying an agreed test protocol that was collaboratively developed by the labs. Each lab first analyzed their own findings as they always do in their regular professional practice. A few weeks later, they again analyzed their findings but then everyone applied the same method (SlimDEVAN: a simplified version of DEVAN, a method developed for facilitating comparison of findings from usability tests in an academic setting). It was found that levels of agreement between labs did not improve when they all used SlimDEVAN there was inherent subjectivity in their analyses. It was found that consistency of single analysts in analyzing their data can be improved by using a method like SlimDEVAN. Such methods can also help in making the analysis process and findings more inspectable. Inspectability is helpful in comparative studies based on identified usability problems because it allows for the traceability of findings to original observations, as well as for laying bare the subjective parts of the data analysis

Task Prioritization in Dual-Tasking: Instructions versus Preferences

<div><p>The role of task prioritization in performance tradeoffs during multi-tasking has received widespread attention. However, little is known on whether people have preferences regarding tasks, and if so, whether these preferences conflict with priority instructions. Three experiments were conducted with a high-speed driving game and an auditory memory task. In Experiment 1, participants did not receive priority instructions. Participants performed different sequences of single-task and dual-task conditions. Task performance was evaluated according to participants’ retrospective accounts on preferences. These preferences were reformulated as priority instructions in Experiments 2 and 3. The results showed that people differ in their preferences regarding task prioritization in an experimental setting, which can be overruled by priority instructions, but only after increased dual-task exposure. Additional measures of mental effort showed that performance tradeoffs had an impact on mental effort. The interpretation of these findings was used to explore an extension of Threaded Cognition Theory with Hockey’s Compensatory Control Model.</p></div

Experimental design.

<p>Experimental design.</p

Memory task performance (A) and driving task performance (B,C) as function of sequence and preference.

<p>Lines are added for interpretation only. Error bars represent +/- 1 standard error of the mean, corrected for within-subjects variability. Note: participants did not receive priority instructions.</p

Memory task performance (A), driving task performance (B) and subjective mental effort (C,D) as function of instruction.

<p>Lines are added for interpretation only. Error bars represent +/- 1 standard error of the mean, corrected for within-subject variability.</p

Results of the ‘driving’ and ‘equal’ priority instruction groups as function of preference.

<p>Lines are added for interpretation only. Error bars represent +/- 1 standard error of the mean, corrected for within-subject variability. The participant with a preference for the memory task in the 'free choice' group was omitted. Note: no instruction was provided in the DUAL_{<b>baseline</b>} condition.</p

Control flowchart interpretation of Threaded Cognition Theory [9], with goal-related instructions fired by the procedural resource.

<p>Control flowchart interpretation of Threaded Cognition Theory [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0158511#pone.0158511.ref009" target="_blank">9</a>], with goal-related instructions fired by the procedural resource.</p

Performance by the control group (Experiment 2) versus two preferences (Experiment 1).

<p>Lines are added for interpretation only. Error bars represent +/- 1 standard error of the mean, corrected for within-subjects variability. Note: participants did not receive priority instructions.</p

Model of task prioritization in the context of task interference.

<p>Model of task prioritization in the context of task interference.</p

Summary of ANOVA results on performance and mental effort as function of instruction.

<p>Summary of ANOVA results on performance and mental effort as function of instruction.</p