The Experiment Factory: Standardizing Behavioral Experiments

The administration of behavioral and experimental paradigms for psychology research is hindered by lack of a coordinated effort to develop and deploy standardized paradigms. While several frameworks (de Leeuw (2015); McDonnell et al. (2012); Mason and Suri (2011); Lange et al. (2015)) have provided infrastructure and methods for individual research groups to develop paradigms, missing is a coordinated effort to develop paradigms linked with a system to easily deploy them. This disorganization leads to redundancy in development, divergent implementations of conceptually identical tasks, disorganized and error-prone code lacking documentation, and difficulty in replication. The ongoing reproducibility crisis in psychology and neuroscience research (Baker (2015); Open Science Collaboration (2015)) highlights the urgency of this challenge: reproducible research in behavioral psychology is conditional on deployment of equivalent experiments. A large, accessible repository of experiments for researchers to develop collaboratively is most efficiently accomplished through an open source framework. Here we present the Experiment Factory, an open source framework for the development and deployment of web-based experiments. The modular infrastructure includes experiments, virtual machines for local or cloud deployment, and an application to drive these components and provide developers with functions and tools for further extension. We release this infrastructure with a deployment (http://www.expfactory.org) that researchers are currently using to run a set of over 80 standardized web-based experiments on Amazon Mechanical Turk. By providing open source tools for both deployment and development, this novel infrastructure holds promise to bring reproducibility to the administration of experiments, and accelerate scientific progress by providing a shared community resource of psychological paradigms

Cognitive tasks, anatomical MRI, and functional MRI data evaluating the construct of self-regulation

We describe the following shared data from N = 103 healthy adults who completed a broad set of cognitive tasks, surveys, and neuroimaging measurements to examine the construct of self-regulation. The neuroimaging acquisition involved task-based fMRI, resting state fMRI, and structural MRI. Each subject completed the following ten tasks in the scanner across two 90-minute scanning sessions: attention network test (ANT), cued task switching, Columbia card task, dot pattern expectancy (DPX), delay discounting, simple and motor selective stop signal, Stroop, a towers task, and a set of survey questions. The dataset is shared openly through the OpenNeuro project, and the dataset is formatted according to the Brain Imaging Data Structure (BIDS) standard

Implications of the lacking relationship between cognitive task and self report measures for psychiatry

At the heart of science is measurement, and the quality of measurements limits the quality of the resulting conclusions. In psychiatric research, the most common measurement has traditionally been through self-report, using scales that assess the degree and frequency of psychiatric symptoms. However, self-report has largely been eschewed within biological and computational psychiatry for lacking the ability to provide mechanistic insights into the disorders in question. Instead, researchers now focus primarily on task-based measures of behavior combined with model-based analyses. This approach is thought to allow a deeper insight into the underlying neural and computational mechanisms whose dysfunction ultimately gives rise to psychiatric symptoms and illness. Indeed, the RDoC framework is explicitly built around these underlying neurocognitive dimensions. The measures proposed in the framework are meant to assess the function of mechanisms instead of or in addition to the frequency and severity of symptoms. The subjective nature of self-report, compared to the seemingly objective nature of cognitive tasks in combination with sophisticated computational models, has led many researchers to move away from the former

Implications of the lacking relationship between cognitive task and self report measures for psychiatry

At the heart of science is measurement, and the quality of measurements limits the quality of the resulting conclusions. In psychiatric research, the most common measurement has traditionally been through self-report, using scales that assess the degree and frequency of psychiatric symptoms. However, self-report has largely been eschewed within biological and computational psychiatry for lacking the ability to provide mechanistic insights into the disorders in question. Instead, researchers now focus primarily on task-based measures of behavior combined with model-based analyses. This approach is thought to allow a deeper insight into the underlying neural and computational mechanisms whose dysfunction ultimately gives rise to psychiatric symptoms and illness. Indeed, the RDoC framework is explicitly built around these underlying neurocognitive dimensions. The measures proposed in the framework are meant to assess the function of mechanisms instead of or in addition to the frequency and severity of symptoms. The subjective nature of self-report, compared to the seemingly objective nature of cognitive tasks in combination with sophisticated computational models, has led many researchers to move away from the former

Sound credit scores and financial decisions despite cognitive aging.

Age-related deterioration in cognitive ability may compromise the ability of older adults to make major financial decisions. We explore whether knowledge and expertise accumulated from past decisions can offset cognitive decline to maintain decision quality over the life span. Using a unique dataset that combines measures of cognitive ability (fluid intelligence) and of general and domain-specific knowledge (crystallized intelligence), credit report data, and other measures of decision quality, we show that domain-specific knowledge and expertise provide an alternative route for sound financial decisions. That is, cognitive aging does not spell doom for financial decision-making in domains where the decision maker has developed expertise. These results have important implications for public policy and for the design of effective interventions and decision aids

Sound credit scores and financial decisions despite cognitive aging.

Age-related deterioration in cognitive ability may compromise the ability of older adults to make major financial decisions. We explore whether knowledge and expertise accumulated from past decisions can offset cognitive decline to maintain decision quality over the life span. Using a unique dataset that combines measures of cognitive ability (fluid intelligence) and of general and domain-specific knowledge (crystallized intelligence), credit report data, and other measures of decision quality, we show that domain-specific knowledge and expertise provide an alternative route for sound financial decisions. That is, cognitive aging does not spell doom for financial decision-making in domains where the decision maker has developed expertise. These results have important implications for public policy and for the design of effective interventions and decision aids