12 research outputs found
The Experiment Factory: Standardizing Behavioral Experiments
The administration of behavioral and experimental paradigms for psychology research is hindered by lack of a coordinated effort to develop and deploy standardized paradigms. While several frameworks (de Leeuw (2015); McDonnell et al. (2012); Mason and Suri (2011); Lange et al. (2015)) have provided infrastructure and methods for individual research groups to develop paradigms, missing is a coordinated effort to develop paradigms linked with a system to easily deploy them. This disorganization leads to redundancy in development, divergent implementations of conceptually identical tasks, disorganized and error-prone code lacking documentation, and difficulty in replication. The ongoing reproducibility crisis in psychology and neuroscience research (Baker (2015); Open Science Collaboration (2015)) highlights the urgency of this challenge: reproducible research in behavioral psychology is conditional on deployment of equivalent experiments. A large, accessible repository of experiments for researchers to develop collaboratively is most efficiently accomplished through an open source framework. Here we present the Experiment Factory, an open source framework for the development and deployment of web-based experiments. The modular infrastructure includes experiments, virtual machines for local or cloud deployment, and an application to drive these components and provide developers with functions and tools for further extension. We release this infrastructure with a deployment (http://www.expfactory.org) that researchers are currently using to run a set of over 80 standardized web-based experiments on Amazon Mechanical Turk. By providing open source tools for both deployment and development, this novel infrastructure holds promise to bring reproducibility to the administration of experiments, and accelerate scientific progress by providing a shared community resource of psychological paradigms
WebQAmGaze: A Multilingual Webcam Eye-Tracking-While-Reading Dataset
We create WebQAmGaze, a multilingual low-cost eye-tracking-while-reading
dataset, designed to support the development of fair and transparent NLP
models. WebQAmGaze includes webcam eye-tracking data from 332 participants
naturally reading English, Spanish, and German texts. Each participant performs
two reading tasks composed of five texts, a normal reading and an
information-seeking task. After preprocessing the data, we find that fixations
on relevant spans seem to indicate correctness when answering the comprehension
questions. Additionally, we perform a comparative analysis of the data
collected to high-quality eye-tracking data. The results show a moderate
correlation between the features obtained with the webcam-ET compared to those
of a commercial ET device. We believe this data can advance webcam-based
reading studies and open a way to cheaper and more accessible data collection.
WebQAmGaze is useful to learn about the cognitive processes behind question
answering (QA) and to apply these insights to computational models of language
understanding
Value generalization in human avoidance learning.
Generalization during aversive decision-making allows us to avoid a broad range of potential threats following experience with a limited set of exemplars. However, over-generalization, resulting in excessive and inappropriate avoidance, has been implicated in a variety of psychological disorders. Here, we use reinforcement learning modelling to dissect out different contributions to the generalization of instrumental avoidance in two groups of human volunteers (N = 26, N = 482). We found that generalization of avoidance could be parsed into perceptual and value-based processes, and further, that value-based generalization could be subdivided into that relating to aversive and neutral feedback - with corresponding circuits including primary sensory cortex, anterior insula, amygdala and ventromedial prefrontal cortex. Further, generalization from aversive, but not neutral, feedback was associated with self-reported anxiety and intrusive thoughts. These results reveal a set of distinct mechanisms that mediate generalization in avoidance learning, and show how specific individual differences within them can yield anxiety.Wellcome, Arthritis Research U
Correction Without Consciousness in Complex Tasks: Evidence from Typing
Published: 07 January 2022It has been demonstrated that with practice, complex tasks can become independent
of conscious control, but even in those cases, repairing errors is thought to remain
dependent on conscious control. This paper reports two studies probing conscious
awareness over repairs in nearly 15,000 typing errors collected from 145 participants
in a single-word typing-to-dictation task. We provide evidence for subconscious repairs
by ruling out alternative accounts, and report two sets of analyses showing that a)
such repairs are not confined to a specific stage of processing and b) that they are
sensitive to the final outcome of repair. A third set of analyses provides a detailed
comparison of the timeline of trials with conscious and subconscious repairs, revealing
that the difference is confined to the repair process itself. We propose an account of
repair processing that accommodates these empirical findings.This project was supported by the Therapeutic Cognitive Neurology Fund to Johns Hopkins School of Medicine, Department of Neurology, Division of Cognitive Neurology
Experimenting with online governance
To solve the problems they face, online communities adopt comprehensive governance methods including committees, boards, juries, and even more complex institutional logics. Helping these kinds of communities succeed will require categorizing best practices and creating toolboxes that fit the needs of specific communities. Beyond such applied uses, there is also a potential for an institutional logic itself to evolve, taking advantage of feedback provided by the fast pace and large ecosystem of online communication. Here, we outline an experimental strategy aiming at guiding and facilitating such an evolution. We first review the advantages of studying collective action using recent technologies for efficiently orchestrating massive online experiments. Research in this vein includes attempts to understand how behavior spreads, how cooperation evolves, and how the wisdom of the crowd can be improved. We then present the potential usefulness of developing virtual-world experiments with governance for improving the utility of social feedback. Such experiments can be used for improving community rating systems and monitoring (dashboard) systems. Finally, we present a framework for constructing large-scale experiments entirely in virtual worlds, aimed at capturing the complexity of governance dynamics, to empirically test outcomes of manipulating institutional logic.Received: 14 November 2020; Accepted: 23 March 2021; Published: 26 April 2021
Are acoustics enough? Semantic effects on auditory salience in natural scenes
Auditory salience is a fundamental property of a sound that allows it to grab a listener's attention regardless of their attentional state or behavioral goals. While previous research has shed light on acoustic factors influencing auditory salience, the semantic dimensions of this phenomenon have remained relatively unexplored owing both to the complexity of measuring salience in audition as well as limited focus on complex natural scenes. In this study, we examine the relationship between acoustic, contextual, and semantic attributes and their impact on the auditory salience of natural audio scenes using a dichotic listening paradigm. The experiments present acoustic scenes in forward and backward directions; the latter allows to diminish semantic effects, providing a counterpoint to the effects observed in forward scenes. The behavioral data collected from a crowd-sourced platform reveal a striking convergence in temporal salience maps for certain sound events, while marked disparities emerge in others. Our main hypothesis posits that differences in the perceptual salience of events are predominantly driven by semantic and contextual cues, particularly evident in those cases displaying substantial disparities between forward and backward presentations. Conversely, events exhibiting a high degree of alignment can largely be attributed to low-level acoustic attributes. To evaluate this hypothesis, we employ analytical techniques that combine rich low-level mappings from acoustic profiles with high-level embeddings extracted from a deep neural network. This integrated approach captures both acoustic and semantic attributes of acoustic scenes along with their temporal trajectories. The results demonstrate that perceptual salience is a careful interplay between low-level and high-level attributes that shapes which moments stand out in a natural soundscape. Furthermore, our findings underscore the important role of longer-term context as a critical component of auditory salience, enabling us to discern and adapt to temporal regularities within an acoustic scene. The experimental and model-based validation of semantic factors of salience paves the way for a complete understanding of auditory salience. Ultimately, the empirical and computational analyses have implications for developing large-scale models for auditory salience and audio analytics