Naturalistic Distributed Experimentation as a Source of New Insight

Human performance experiments are often conducted online with the help of paid crowdworkers and citizen scientists. This approach produces reliable data, but there are concerns that the inevitable loss of control that accompanies online experimentation might confound results. Researchers have therefore spent time considering how to regain control and mitigate the effects of confounds. In this abstract we argue that confounding factors in online work can be put to novel use, giving us insight into research questions we might otherwise be unable to answer

Home is Where the Lab is: A Comparison of Online and Lab Data From a Time-sensitive Study of Interruption

While experiments have been run online for some time with positive results, there are still outstanding questions about the kinds of tasks that can be successfully deployed to remotely situated online participants. Some tasks, such as menu selection, have worked well but these do not represent the gamut of tasks that interest HCI researchers. In particular, we wondered whether long-lasting, time-sensitive tasks that require continuous concentration could work successfully online, given the confounding effects that might accompany the online deployment of such a task. We ran an archetypal interruption experiment both online and in the lab to investigate whether studies demonstrating such characteristics might be more vulnerable to a loss of control than the short, time-insensitive studies that are representative of the majority of previous online studies. Statistical comparisons showed no significant differences in performance on a number of dimensions. However, there were issues with data quality that stemmed from participants misunderstanding the task. Our findings suggest that long-lasting experiments using time-sensitive performance measures can be run online but that care must be taken when introducing participants to experimental procedures

Effects of External and Internal Task-switching upon Idea Generation Performance

학위논문 (석사)-- 서울대학교 대학원 : 심리학과 인지심리학, 2016. 2. 박주용.Multi-tasking pervades in our daily lives. We are handling more than two tasks at the same time. In a situation where more than two tasks are to be carried out within a set amount of time, there are mainly two reasons why a person switches from one task to another. In one instance, switching occurs because of external circumstances regardless of the intent of performers, such as the close of the set time. In the other, switching occurs because the performer gets bored or can no longer concentrate on the task. This study was carried out to examine how these two types of task switching affect idea generation. Two idea generation tasks were presented to different groups under various conditions. For each task participants were allowed to use for 12 min. In the control group (n=25), participants were instructed to tackle the two tasks one at a time (Task A then Task B). There were two experimental conditions: in the external task switching condition, participants were obliged to alternate the two tasks sequentially for 12 min each (Task A, Task B, Task A, Task B)In the internal switch condition, participants were allowed to switch the two tasks whenever they want to within 24 minutes for each task. The number and the quality of the ideas were rated by two graders who were not informed about the purpose of the study. The results showed that the quality of the ideas was lower for the external task switching condition than that for the control condition. However, the performance of the internal task switching group was superior to that of the control condition. Subsequent analysis showed that the increase in the number of switching resulted in lower performance even in case of the group who were given the choice when to switch. In conclusion, the results showed that performance improves when task switching choice is allowed but that too many switches have a negative influence on performance. Possible applications of these results were discussed.INTRODCTION 1 Interruption and Switch Cost 5 Task Switch and Incubation Effect 8 Switch Cost and Problem State 12 Switch Frequency 15 Current Study 16 METHOD 19 Participants 19 Materials 19 Procedure 20 Measures 23 RESULTS 24 Task Performance 24 Switch Frequency and Task Performance 26 DISCUSSION 28 Limitations and Future Direction 31 Implications 32 References 35 Appendices 46 Appendix A 실험에서 사용하는 자극 46 Appendix B 설문지 47 Abstract in Korean 50Maste

Diminished Control in Crowdsourcing: An Investigation of Crowdworker Multitasking Behavior

Obtaining high-quality data from crowds can be difficult if contributors do not give tasks sufficient attention. Attention checks are often used to mitigate this problem, but, because the roots of inattention are poorly understood, checks often compel attentive contributors to complete unnecessary work. We investigated a potential source of inattentiveness during crowdwork: multitasking. We found that workers switched to other tasks every five minutes, on average. There were indications that increasing switch frequency negatively affected performance. To address this, we tested an intervention that encouraged workers to stay focused on our task after multitasking was detected. We found that our intervention reduced the frequency of task-switching. It also improves on existing attention checks because it does not place additional demands on workers who are already focused. Our approach shows that crowds can help to overcome some of the limitations of laboratory studies by affording access to naturalistic multitasking behavior

Frequency and Duration of Self-Initiated Task-Switching in an Online Investigation of Interrupted Performance

Online experimentation has been shown to be a reliable method for collecting data quickly and easily. Nevertheless, this method of data collection can affect some researchers’ confidence in results because of the loss of experimental control that comes with moving experiments online. To relieve these concerns, researchers have developed a number of tools and techniques for evaluating the performance of online participants and crowdworkers. In this paper we investigate a measure of participants’ attentiveness during an online data-entry experiment. We explored online participants’ propensity to switch away from the experiment to other tasks, and the duration of time that they spent away from the experiment during each switch. Our results show that switching is prevalent, even when participants are asked not to switch to other tasks

Memory reconsolidation in trauma-like memory

Individuals might meet difficulties in updating and integrating traumatic memories with existing autobiographical memories, which could lead to posttraumatic stress disorder. Eye Movement Desensitisation and Reprocessing (EMDR) therapy putatively facilitates adaptive memory updating and integration. While eye movement is critical to the therapeutic benefit of EMDR, their mechanism of action remains unclear. We applied video trauma memory with counterconditioning to test the effect of eye movement in EMDR directly. We also explored the importance of internet-based interventions and culture differences in updating trauma-like memories. In experiment 1, we recruited 69 undergraduates from the University of Birmingham. Participants were exposed to distressing film clips and counterconditioning took place a day later. Subjective distress was recorded daily for one week, and declarative memory for the trauma video was also tested on the final day. In experiment 2, we recruited 35 healthy participants from the Southwest University in China and 24 healthy participants from the University of Birmingham; 26 participants completed their experiments in person, and 33 participants completed their experiments online. The experimental procedures were the same as the ones in experiment 1. The results in experiment 1 indicated that eye movement combined with counterconditioning had the lowest IES-R scores among all the groups, but these results were not repeated in declarative memory tasks. In experiment 2, we replicated the effect (eye movement + war + humour) in different cultures and types. However, we only compared the difference between fully memory reconsolidation condition (eye movement + counterconditioning) and control group. Therefore, we were unable to conclude that eye movement can enhance memory reconsolidation, but counterconditioning might play an important role in memory reconsolidation which is a universal phenomenon

What makes an interruption disruptive? Understanding the effects of interruption relevance and timing on performance

Interruptions disrupt activity, hindering performance and provoking errors. They present an obvious challenge in safety-critical environments where momentary slips can have fatal consequences. Interruptions are also a problem in more workaday settings, like offices, where they can reduce productivity and increase stress levels. To be able to systematically manage the negative effects of interruptions, we first need to understand the factors that influence their disruptiveness. This thesis explores how the disruptiveness of interruptions is influenced by their relevance and timing. Seven experimental studies investigate these properties in the context of a routine data-entry task. The first three experiments explore how relevance and timing interact. They demonstrate that the relevance of interruptions depends on the contents of working memory at the moment of interruption. Next, a pair of experiments distinguish the oft-conflated concepts of interruption relevance and relatedness. They show that interruptions with similar content to the task at hand can negatively affect performance if they do not contribute toward the rehearsal of goals in working memory. By causing active interference, seemingly useful interruptions that are related to the task at hand have the potential to be more disruptive than entirely unrelated, irrelevant interruptions. The final two experiments in this thesis test the reliability of the effects observed in the first five experiments through alternative experimental paradigms. They show that relevance and timing effects are consistent even when participants are given control over interruptions and that these effects are robust even in an online setting where experimental control is compromised. The work presented in this thesis enhances our understanding of the factors influencing the disruptiveness of interruptions. Its primary contribution is to show that when we talk about interruptions, ‘relevance’, ‘irrelevance’ and ‘relatedness’ must be considered in the context of the contents of working memory at the moment of interruption. This finding has implications for experimental investigations of interrupted performance, efforts to under- stand the effects of interruptions in the workplace, and the development of systems that help users manage interruptions

Applications of the Flexilevel Test to Assessment in Higher Education

The work reported in this dissertation investigates the potential for embedding Computerised Adaptive Testing (CAT) in students’ and tutors’ educational experiences. It seems that the tailored assessments that CAT can provide would be of real educational value in a range of contemporary Higher Education settings, however the resource requirements of some forms of CAT are prohibitive for making CAT assessments available to students across their studies. A form of CAT that is less resource intensive than other forms, the Flexilevel test, was selected for this programme of research to investigate its effectiveness in real educational contexts and explore possible applications for the approach. Ten empirical studies and a real data simulation study were conducted to test the effectiveness of the approach. It was found to show statistically significant correlations with other forms of assessment – in particular conventional Computer Based Testing (CBT) assessments, which is commonly used in contemporary educational settings. Another strand of work concerned the attitudes of stakeholders to the approach. Part of this work was carried out through the empirical studies, and further studies including interviews were also undertaken to explore the views of academic staff and students to the use of the Flexilevel test. Both groups were positive about the use of the Flexilevel test and this was taken to support the idea that academic staff and students would accept the use of the Flexilevel test in their educational experiences. In terms of both effectiveness and the acceptability of the approach to academic staff and students, the Flexilevel test was found to be a good candidate for embedding CAT in real educational contexts in Higher Education