Evaluating Amazon\u27s Mechanical Turk as a Tool for Experimental Behavioral Research

Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter\u27s control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers

Fast and flexible: Human program induction in abstract reasoning tasks

The Abstraction and Reasoning Corpus (ARC) is a challenging program induction dataset that was recently proposed by Chollet (2019). Here, we report the first set of results collected from a behavioral study of humans solving a subset of tasks from ARC (40 out of 1000). Although this subset of tasks contains considerable variation, our results showed that humans were able to infer the underlying program and generate the correct test output for a novel test input example, with an average of 80% of tasks solved per participant, and with 65% of tasks being solved by more than 80% of participants. Additionally, we find interesting patterns of behavioral consistency and variability within the action sequences during the generation process, the natural language descriptions to describe the transformations for each task, and the errors people made. Our findings suggest that people can quickly and reliably determine the relevant features and properties of a task to compose a correct solution. Future modeling work could incorporate these findings, potentially by connecting the natural language descriptions we collected here to the underlying semantics of ARC.Comment: 7 pages, 7 figures, 1 tabl

A computational fluid dynamic investigation of inhomogeneous hydrogen flame acceleration and transition to detonation

Gas explosions in homogeneous reactive mixtures have been widely studied both experimentally and numerically. However, in practice and industrial applications, combustible mixtures are usually inhomogeneous and subject to vertical concentration gradients. Limited studies have been conducted in such context which resulted in limited understanding of the explosion characteristics in such situations. The present numerical investigation aims to study the dynamics of Deflagration to Detonation Transition (DDT) in inhomogeneous hydrogen/air mixtures and examine the effects of obstacle blockage ratio in DDT. VCEFoam, a reactive density-based solver recently assembled by the authors within the frame of OpenFOAM CFD toolbox has been used. VCEFoam uses the Harten–Lax–van Leer–Contact (HLLC) scheme fr the convective fluxes contribution and shock capturing. The solver has been verified by comparing its predictions with the analytical solutions of two classical test cases. For validation, the experimental data of Boeck et al. (1) is used. The experiments were conducted in a rectangular channel the three different blockage ratios and hydrogen concentrations. Comparison is presented between the predicted and measured flame tip velocities. The shaded contours of the predicted temperature and numerical Schlieren (magnitude of density gradient) will be analysed to examine the effects of the blockage ratio on flame acceleration and DDT

Measuring category intuitiveness in unconstrained categorization tasks

What makes a category seem natural or intuitive? In this paper, an unsupervised categorization task was employed to examine observer agreement concerning the categorization of nine different stimulus sets. The stimulus sets were designed to capture different intuitions about classification structure. The main empirical index of category intuitiveness was the frequency of the preferred classification, for different stimulus sets. With 169 participants, and a within participants design, with some stimulus sets the most frequent classification was produced over 50 times and with others not more than two or three times. The main empirical finding was that cluster tightness was more important in determining category intuitiveness, than cluster separation. The results were considered in relation to the following models of unsupervised categorization: DIVA, the rational model, the simplicity model, SUSTAIN, an Unsupervised version of the Generalized Context Model (UGCM), and a simple geometric model based on similarity. DIVA, the geometric approach, SUSTAIN, and the UGCM provided good, though not perfect, fits. Overall, the present work highlights several theoretical and practical issues regarding unsupervised categorization and reveals weaknesses in some of the corresponding formal models