The effect of noise-induced variance on parameter recovery from reaction times

BACKGROUND: Technical noise can compromise the precision and accuracy of the reaction times collected in psychological experiments, especially in the case of Internet-based studies. Although this noise seems to have only a small impact on traditional statistical analyses, its effects on model fit to reaction-time distributions remains unexplored. RESULTS: Across four simulations we study the impact of technical noise on parameter recovery from data generated from an ex-Gaussian distribution and from a Ratcliff Diffusion Model. Our results suggest that the impact of noise-induced variance tends to be limited to specific parameters and conditions. CONCLUSIONS: Although we encourage researchers to adopt all measures to reduce the impact of noise on reaction-time experiments, we conclude that the typical amount of noise-induced variance found in these experiments does not pose substantial problems for statistical analyses based on model fitting

Recipes for replication:Applying open science principles to research software development and data collection with cognitive tasks

During the past decade, science witnessed a replication crisis where many findings of published research could not be reproduced by other researchers. Attempts to address this "replication crisis" have identified several avenues for improvement, such as making science more open. This allows for more transparency in the scientific workflow, thus reducing a researcher's degrees of freedom and enabling researchers to check each other's work more extensively. Open science is a multifaceted concept, but in practice, there seems to be a strong focus on pre-registration of research designs, open data, and open access publications. However, between the research design and data/publication, there is the phase of data collection. So far, this phase has received relatively little attention even though it is an essential part of the scientific workflow. Hence, this dissertation focuses on open data collection in the behavioral domain, with an emphasis on cognitive tasks. In modern behavioral science, cognitive task procedures are often automated by software running on a computer. Hence, the focus is on research software development and data collection with cognitive tasks, which are evaluated from the perspective of five schools of thought on open science: the democratic, infrastructure, pragmatic, measurement, and public school. I discuss how applying open science principles to research software development and behavioral data collection with cognitive tasks may address the replication crisis and may improve the quality of science in general