Assessing the Viability of Online Interruption Studies

Researchers have been collecting data online since the early days of the Internet and as technology improves, increasing numbers of traditional experiments are being run online. However, there are still questions about the kinds of experiments that work online, particularly over experiments with time-sensitive performance measures. We are interested in one time-sensitive measure specifically, the time taken to resume a task following an interruption. We ran participants through an archetypal interruption study online and in the lab. Statistical comparisons showed no significant differences in the time it took to resume following an interruption. However, there were issues with data quality that stemmed from participant confusion about the task. Our findings have implications for experiments that assess time-sensitive performance measures in tasks that require continuous attention

Naturalistic Distributed Experimentation as a Source of New Insight

Human performance experiments are often conducted online with the help of paid crowdworkers and citizen scientists. This approach produces reliable data, but there are concerns that the inevitable loss of control that accompanies online experimentation might confound results. Researchers have therefore spent time considering how to regain control and mitigate the effects of confounds. In this abstract we argue that confounding factors in online work can be put to novel use, giving us insight into research questions we might otherwise be unable to answer

Validation of the International Engineering Motivation Construct with Multigroup Analysis

The purpose of this study was twofold. First, the study aimed to validate multicultural workforce motivation in engineering. Second, it aimed to perform a subgroup analyses of the identified model to determine if the theoretical model is valid across demographics. Data were collected from a total of 268 employees working in the United States either as a natural citizen or with an Employment Authorization Document (EAD). The theoretical framework used in the study was Herzberg’s Motivation-Hygiene Theory, which is suggested by many (Ben Slimane, 2017; Shuck and Herd, 2012; Singh & Bhattacharjee, 2019). The instruments used were Minnesota Satisfaction Questionnaire (MSQ) suggested by Weiss et al. (1967) and the Vancouver Index of Acculturation (VIA) suggested by Ryder et al., (2000). Data were collected using crowdsourcing agency.  Both exploratory factor analysis (EFA) and confirmatory factor analyses (CFA) were used in the study. The model was tested with a specific bias test and for model invariance at both the global and local levels. A multi-group difference test was conducted on the participants’ industry, position, birth country, ethnicity, gender, age group, and educational background to confirm or deny configural, metric and scalar invariance. The study validated the results from prior research that reported different number of factors for the motivation construct.  The results of the study suggest that even old theories from a western country, such as Herzberg’s Motivation-Hygiene Theory, still could be valid in multicultural organizations.  Recommendations for future research are provided

Crowdsourcing interface feature design with Bayesian optimization

Designing novel interfaces is challenging. Designers typically rely on experience or subjective judgment in the absence of analytical or objective means for selecting interface parameters. We demonstrate Bayesian optimization as an efficient tool for objective interface feature refinement. Specifically, we show that crowdsourcing paired with Bayesian optimization can rapidly and effectively assist interface design across diverse deployment environments. Experiment 1 evaluates the approach on a familiar 2D interface design problem: a map search and review use case. Adding a degree of complexity, Experiment 2 extends Experiment 1 by switching the deployment environment to mobile-based virtual reality. The approach is then demonstrated as a case study for a fundamentally new and unfamiliar interaction design problem: web-based augmented reality. Finally, we show how the model generated as an outcome of the refinement process can be used for user simulation and queried to deliver various design insights

Home is Where the Lab is: A Comparison of Online and Lab Data From a Time-sensitive Study of Interruption

While experiments have been run online for some time with positive results, there are still outstanding questions about the kinds of tasks that can be successfully deployed to remotely situated online participants. Some tasks, such as menu selection, have worked well but these do not represent the gamut of tasks that interest HCI researchers. In particular, we wondered whether long-lasting, time-sensitive tasks that require continuous concentration could work successfully online, given the confounding effects that might accompany the online deployment of such a task. We ran an archetypal interruption experiment both online and in the lab to investigate whether studies demonstrating such characteristics might be more vulnerable to a loss of control than the short, time-insensitive studies that are representative of the majority of previous online studies. Statistical comparisons showed no significant differences in performance on a number of dimensions. However, there were issues with data quality that stemmed from participants misunderstanding the task. Our findings suggest that long-lasting experiments using time-sensitive performance measures can be run online but that care must be taken when introducing participants to experimental procedures