Justify your alpha

Benjamin et al. proposed changing the conventional “statistical significance” threshold (i.e.,the alpha level) from p ≤ .05 to p ≤ .005 for all novel claims with relatively low prior odds. They provided two arguments for why lowering the significance threshold would “immediately improve the reproducibility of scientific research.” First, a p-value near .05provides weak evidence for the alternative hypothesis. Second, under certain assumptions, an alpha of .05 leads to high false positive report probabilities (FPRP2 ; the probability that a significant finding is a false positive

Justify your alpha

In response to recommendations to redefine statistical significance to p ≤ .005, we propose that researchers should transparently report and justify all choices they make when designing a study, including the alpha level

Complexity-Theoretic Limits on the Promises of Artificial Neural Network Reverse-Engineering

Emerging folklore in the cognitive sciences suggests that interpretability techniques to reverse-engineer artificial neural networks (ANNs) could speed up discovery and theory-building. For many researchers in psychology, linguistics, neuroscience, and artificial intelligence (AI), the full observability and perturbability of ANNs trained on complex tasks affords a shortcut to domain insights, cognitive theories, neurocognitive models, application improvement, and user safety. Folklore intuitions, however, are typically disconnected from other relevant knowledge. Here we examine these intuitions formally by drawing relevant connections to computational complexity theory. We model interpretability queries computationally and analyze their resource demands for biological/artificial high-level cognition. We prove mathematically that, contrary to folklore, basic circuit-finding queries in classic ANNs are already infeasibly demanding to answer even approximately. We discuss how interdisciplinary integration can mitigate this disconnect and situate the broader implications for the cognitive sciences, the philosophy of AI-fueled discovery, and AI ethics

From empirical problem-solving to theoretical problem-finding perspectives on the cognitive sciences

Meta-theoretical perspectives on the research problems and activities of (cognitive) scientists often emphasize empirical problems and problem-solving as the main aspects that account for scientific progress. While certainly useful to shed light on issues of theory-observation relationships, these conceptual analyses typically begin when empirical problems are already there for researchers to solve. As a result, the role of theoretical problems and problem-finding remain comparatively obscure. How do the scientific problems of Cognitive Science arise, and what do they comprise, empirically and theoretically? Here we attempt to understand the research activities that lead to adequate explanations through a broader conception of the problems researchers must attend to and how they come about. To this end, we bring theoretical problems and problem-finding out of obscurity to paint a more integrative picture of how these complement empirical problems and problem-solving to advance cognitive science

Reclaiming AI as a theoretical tool for cognitive science

The idea that human cognition is, or can be understood as, a form of computation is a useful conceptual tool for cognitive science. It was a foundational assumption during the birth of cognitive science as a multidisciplinary field, with Artificial Intelligence (AI) as one of its contributing fields. One conception of AI in this context is as a provider of computational tools (frameworks, concepts, formalisms, models, proofs, simulations, etc.) that support theory building in cognitive science. The contemporary field of AI, however, has taken the theoretical possibility of explaining human cognition as a form of computation to imply the practical feasibility of realising human(-like or -level) cognition in factual computational systems; and, the field frames this realisation as a short-term inevitability. Yet, as we formally prove herein, creating systems with human(-like or -level) cognition is intrinsically computationally intractable. This means that any factual AI systems created in the short-run are at best decoys. When we think these systems capture something deep about ourselves and our thinking, we induce distorted and impoverished images of ourselves and our cognition. In other words, AI in current practice is deteriorating our theoretical understanding of cognition rather than advancing and enhancing it. The situation could be remediated by releasing the grip of the currently dominant view on AI and by returning to the idea of AI as a theoretical tool for cognitive science. In reclaiming this older idea of AI, however, it is important not to repeat conceptual mistakes of the past (and present) that brought us to where we are today

On the importance of severely testing deep learning models of cognition

Researchers studying the correspondences between Deep Neural Networks (DNNs) and humans often give little consideration to severe testing when drawing conclusions from empirical findings, and this is impeding progress in building better models of minds. We first detail what we mean by severe testing and highlight how this is especially important when working with opaque models with many free parameters that may solve a given task in multiple different ways. Second, we provide multiple examples of researchers making strong claims regarding DNN-human similarities without engaging in severe testing of their hypotheses. Third, we consider why severe testing is undervalued. We provide evidence that part of the fault lies with the review process. There is now a widespread appreciation in many areas of science that a bias for publishing positive results (among other practices) is leading to a credibility crisis, but there seems less awareness of the problem here

Functional shifts in L2

Data for "Reading Shakespearean tropes in a foreign tongue: Age of L2 acquisition modulates neural responses to functional shifts

Clarifying status of DNNs as models of human vision

On several key issues we agree with the commentators. Perhaps most importantly, everyone seems to agree that psychology has an important role to play in building better models of human vision, and (most) everyone agrees (including us) that DNNs will play an important role in modelling human vision going forward. But there are also disagreements about what models are for, how DNN-human correspondences should be evaluated, the value of alternative modelling approaches, and impact of marketing hype in the literature. In our view, these latter issues are contributing to many unjustified claims regarding DNN-human correspondences in vision and other domains of cognition. We explore all these issues in this response

Introducing the MindSet benchmark for comparing DNNs to human vision

We describe the MindSet benchmark designed to facilitate the testing of DNNs against controlled experiments reported in psychology. MindSet will focus on a range of low-, middle-, and high-level visual findings that provide important constraints for theory, provide the materials for testing DNNs, and provide an example of how to assess a DNN on each experiment using a ResNet152 pretrained on ImageNet. The goal is not to evaluate how well ResNet152 accounts for human vision, but rather, encourage researchers to assess how well various DNNs account for a range of key human visual phenomena

Reading Shakespearean tropes in a foreign tongue: Age of L2 acquisition modulates neural responses to functional shifts

Functional shifts (FSs) – morphosyntactically marked words evoking coherent but novel meanings – are ubiquitous in English and, specially, in Shakespearean literature. While their neural signatures have been explored in native speakers, no study has targeted foreign-language users, let alone comparing early and late bilinguals. Here, we administered a validated FS paradigm to subjects from both populations and evaluated time-frequency modulations evoked by FS and control sentences. Early bilinguals exhibited greater sensitivity towards FSs, indexed by reduced fronto-posterior theta-band oscillations across semantic- and structural-integration windows. Such oscillatory modulations may represent a key marker of age-of-acquisition effects during foreign-language wordplay processing.Fil: Vilas, Martina G.. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Houssay. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Instituto de Neurología Cognitiva. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Fundación Favaloro. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt; ArgentinaFil: Santilli, Micaela. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Houssay. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Instituto de Neurología Cognitiva. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Fundación Favaloro. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt; ArgentinaFil: Mikulan, Ezequiel Pablo. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Houssay. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Instituto de Neurología Cognitiva. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Fundación Favaloro. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt; ArgentinaFil: Gonzalez Adolfi, Federico. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Houssay. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Instituto de Neurología Cognitiva. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Fundación Favaloro. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt; ArgentinaFil: Martorell Caro, Miguel Angel. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Houssay. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Instituto de Neurología Cognitiva. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Fundación Favaloro. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt; ArgentinaFil: Manes, Facundo Francisco. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Houssay. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Instituto de Neurología Cognitiva. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Fundación Favaloro. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt; ArgentinaFil: Herrera, Eduar. Universidad ICESI, ; ColombiaFil: Sedeño, Lucas. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Houssay. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Instituto de Neurología Cognitiva. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Fundación Favaloro. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt; ArgentinaFil: Ibañez, Agustin Mariano. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Houssay. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Instituto de Neurología Cognitiva. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Fundación Favaloro. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt; Argentina. Universidad Autónoma del Caribe; Colombia. Universidad Adolfo Ibañez; ChileFil: García, Adolfo Martín. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Houssay. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Instituto de Neurología Cognitiva. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt | Fundación Favaloro. Instituto de Neurociencia Cognitiva y Traslacional. Fundación Ineco Rosario Sede del Incyt; Argentin