Representational precision in visual cortex reveals outcome encoding and reward modulation during action preparation

According to ideomotor theory, goal-directed action involves the active perceptual anticipation of actions and their associated effects. We used multivariate analysis of fMRI data to test if preparation of an action promotes precision in the perceptual representation of the action. In addition, we tested how reward magnitude modulates this effect. Finally, we examined how expectation and uncertainty impact neural precision in the motor cortex. In line with our predictions, preparation of a hand or face action increased the precision of neural activation patterns in the extrastriate body area (EBA) and fusiform face area (FFA), respectively. The size of this effect of anticipation predicted individuals\u27 efficiency at performing the prepared action. In addition, increasing reward magnitude increased the precision of perceptual representations in both EBA and FFA although this effect was limited to the group of participants that learned to associate face actions with high reward. Surprisingly, examination of representations in the hand motor cortex and face motor cortex yielded effects in the opposite direction. Our findings demonstrate that the precision of representations in visual and motor areas provides an important neural signature of the sensorimotor representations involved in goal-directed action

Backwards is the way forward: feedback in the cortical hierarchy predicts the expected future

Clark offers a powerful description of the brain as a prediction machine, which offers progress on two distinct levels. First, on an abstract conceptual level, it provides a unifying framework for perception, action, and cognition (including subdivisions such as attention, expectation, and imagination). Second, hierarchical prediction offers progress on a concrete descriptive level for testing and constraining conceptual elements and mechanisms of predictive coding models (estimation of predictions, prediction errors, and internal models)

Characterizing and comparing acoustic representations in convolutional neural networks and the human auditory system

Le traitement auditif dans le cerveau humain et dans les systèmes informatiques consiste en une cascade de transformations représentationnelles qui extraient et réorganisent les informations pertinentes pour permettre l'exécution des tâches. Cette thèse s'intéresse à la nature des représentations acoustiques et aux principes de conception et d'apprentissage qui soutiennent leur développement. Les objectifs scientifiques sont de caractériser et de comparer les représentations auditives dans les réseaux de neurones convolutionnels profonds (CNN) et la voie auditive humaine. Ce travail soulève plusieurs questions méta-scientifiques sur la nature du progrès scientifique, qui sont également considérées. L'introduction passe en revue les connaissances actuelles sur la voie auditive des mammifères et présente les concepts pertinents de l'apprentissage profond. Le premier article soutient que les questions philosophiques les plus pressantes à l'intersection de l'intelligence artificielle et biologique concernent finalement la définition des phénomènes à expliquer et ce qui constitue des explications valables de tels phénomènes. Je surligne les théories pertinentes de l'explication scientifique que j’espére fourniront un échafaudage pour de futures discussions. L'article 2 teste un modèle populaire de cortex auditif basé sur des modulations spectro-temporelles. Nous constatons qu'un modèle linéaire entraîné uniquement sur les réponses BOLD aux ondulations dynamiques simples (contenant seulement une fréquence fondamentale, un taux de modulation temporelle et une échelle spectrale) peut se généraliser pour prédire les réponses aux mélanges de deux ondulations dynamiques. Le troisième article caractérise la spécificité linguistique des couches CNN et explore l'effet de l'entraînement figé et des poids aléatoires. Nous avons observé trois régions distinctes de transférabilité: (1) les deux premières couches étaient entièrement transférables, (2) les couches 2 à 8 étaient également hautement transférables, mais nous avons trouvé évidence de spécificité de la langue, (3) les couches suivantes entièrement connectées étaient plus spécifiques à la langue mais pouvaient être adaptées sur la langue cible. Dans l'article 4, nous utilisons l'analyse de similarité pour constater que la performance supérieure de l'entraînement figé obtenues à l'article 3 peuvent être attribuées aux différences de représentation dans l'avant-dernière couche: la deuxième couche entièrement connectée. Nous analysons également les réseaux aléatoires de l'article 3, dont nous concluons que la forme représentationnelle est doublement contrainte par l'architecture et la forme de l'entrée et de la cible. Pour tester si les CNN acoustiques apprennent une hiérarchie de représentation similaire à celle du système auditif humain, le cinquième article compare l'activité des réseaux «freeze trained» de l'article 3 à l'activité IRMf 7T dans l'ensemble du système auditif humain. Nous ne trouvons aucune évidence d'une hiérarchie de représentation partagée et constatons plutôt que tous nos régions auditifs étaient les plus similaires à la première couche entièrement connectée. Enfin, le chapitre de discussion passe en revue les mérites et les limites d'une approche d'apprentissage profond aux neurosciences dans un cadre de comparaison de modèles. Ensemble, ces travaux contribuent à l'entreprise naissante de modélisation du système auditif avec des réseaux de neurones et constituent un petit pas vers une science unifiée de l'intelligence qui étudie les phénomènes qui se manifestent dans l'intelligence biologique et artificielle.Auditory processing in the human brain and in contemporary machine hearing systems consists of a cascade of representational transformations that extract and reorganize relevant information to enable task performance. This thesis is concerned with the nature of acoustic representations and the network design and learning principles that support their development. The primary scientific goals are to characterize and compare auditory representations in deep convolutional neural networks (CNNs) and the human auditory pathway. This work prompts several meta-scientific questions about the nature of scientific progress, which are also considered. The introduction reviews what is currently known about the mammalian auditory pathway and introduces the relevant concepts in deep learning.The first article argues that the most pressing philosophical questions at the intersection of artificial and biological intelligence are ultimately concerned with defining the phenomena to be explained and with what constitute valid explanations of such phenomena. I highlight relevant theories of scientific explanation which we hope will provide scaffolding for future discussion. Article 2 tests a popular model of auditory cortex based on frequency-specific spectrotemporal modulations. We find that a linear model trained only on BOLD responses to simple dynamic ripples (containing only one fundamental frequency, temporal modulation rate, and spectral scale) can generalize to predict responses to mixtures of two dynamic ripples. Both the third and fourth article investigate how CNN representations are affected by various aspects of training. The third article characterizes the language specificity of CNN layers and explores the effect of freeze training and random weights. We observed three distinct regions of transferability: (1) the first two layers were entirely transferable between languages, (2) layers 2--8 were also highly transferable but we found some evidence of language specificity, (3) the subsequent fully connected layers were more language specific but could be successfully finetuned to the target language. In Article 4, we use similarity analysis to find that the superior performance of freeze training achieved in Article 3 can be largely attributed to representational differences in the penultimate layer: the second fully connected layer. We also analyze the random networks from Article 3, from which we conclude that representational form is doubly constrained by architecture and the form of the input and target. To test whether acoustic CNNs learn a similar representational hierarchy as that of the human auditory system, the fifth article presents a similarity analysis to compare the activity of the freeze trained networks from Article 3 to 7T fMRI activity throughout the human auditory system. We find no evidence of a shared representational hierarchy and instead find that all of our auditory regions were most similar to the first fully connected layer. Finally, the discussion chapter reviews the merits and limitations of a deep learning approach to neuroscience in a model comparison framework. Together, these works contribute to the nascent enterprise of modeling the auditory system with neural networks and constitute a small step towards a unified science of intelligence that studies the phenomena that are exhibited in both biological and artificial intelligence

Content representation in the human medial temporal lobe

The transformation of sensory inputs into complex memory representations is fundamental to human experience; yet, little is known about how this crucial process is achieved. When you meet your friend at the new cafe in town, what part of the brain encodes this novel scene into long term memory? What part encoded your friend’s favorite t-shirt, so that the sight of it gives you a feeling of familiarity rather than surprise? It is well-established that the medial temporal lobe (MTL) is crucial to both processes, but the MTL is not a single homogeneous region. In fact, it is composed of several anatomically distinct subregions including hippocampus, perirhinal cortex (PRC) and parahippocampal cortex (PHC). However, the computations performed by each subregion to encode individual events is still unclear. The present research tests the central hypothesis that different forms of event content are transformed into memory by distinct subregions within the MTL. A critical barrier in the study of content representation thus far has been its focus on comparing univariate peak activations in a region to different stimulus materials. To go beyond this limited approach, we employed multivariate statistical analyses that takes into account how event content is represented by distributed activity in MTL subregions. First, we examine the content-specific contributions of MTL subregions to episodic encoding and retrieval. Then, we demonstrate how these distributed representations support memory-based prediction to resolve ambiguities in our environment.Neuroscienc

Neural Processes Underlying the Flexible Control and Learning of Attentional Selection

In every-day life we are usually surrounded by a plethora of stimuli, of which only some may be relevant to us at a given moment in time. The dynamic interaction between internal factors, such as our previous experience and current goals, and external factors, such as salient sensory stimulation, determine where, how and what we attend to in our environment. This dissertation investigated some of the neural mechanisms that underlie successful goal-directed behavior in two conditions 1. when attention was actively cued to a target stimulus, and 2. when the attentional target had to be actively and repeatedly learned, in macaque monkeys and in humans. In Chapter 2, I investigated inter-areal spiketrain correlations in neuron pairs across the fronto-cingulate cortex when macaque monkeys are cued to shift their attention to one of two target stimuli. I found that neuron pairs in anterior cingulate cortex (ACC) and dorsal prefrontal cortex (PFC) with similar spatial preferences correlate their spiketrains at the time when attention needs to be actively shifted, suggesting that the flexible interaction between these two areas may support successful covert attention shifts. In Chapter 3, I show that when the attentional target stimulus needs to be repeatedly learned and is defined by only one of several stimulus features, neurons in macaque frontal and striatal regions encode prediction error signals that carry specific information about the stimulus feature that was selected in the preceding choice. These signals may be involved in identifying those synapses that require updating to allow flexible adjustments in goal-directed behavior. In Chapter 4, I found that when humans must repeatedly learn the identity of an attentional target, a human event-related potential over visual cortex that is thought to index attentional target selection, selectively decreases after successful learning, in particular for the distracting stimulus, and selectively increases for the target stimulus following negative feedback during learning. Overall, this dissertation provides novel insights into some of the complex neural mechanisms that support flexible control and learning of attention across brain regions of the human and non-human primate brain

Mirror Activity in the Macaque Motor System

Mirror neurons (MirNs) within ventral premotor cortex (PMv) and primary motor cortex (M1), including pyramidal tract neurons (PTNs) projecting to the spinal cord, modulate their activity during both the execution and observation of motor acts. However, movement is not produced in the latter condition, and mirror responses cannot be explained by lowlevel muscle activity. Relatively reduced activity in M1 during observation may help to suppress movement. Here, we examined the extent to which activity at different stages of action observation reflects grasp representation and suppression of movement across multiple levels of the mirror system in monkeys and humans. We recorded MirNs in M1 and F5 (rostral PMv), including identified PTNs, in two macaque monkeys as they performed, observed, and withheld reach-to-grasp actions. Time-varying population activity was more distinct between execution and observation in M1 than in F5, and M1 activity in the lead-up to the observation of movement onset shared parallels with movement withholding activity. In separate experiments, modulation of short-latency responses evoked in hand muscles by pyramidal tract stimulation revealed modest grasp-specific facilitation at the spinal level during grasp observation. This contrasted with a relative suppression of excitability prior to observed movement onset or when monkeys simply withheld movement. Additional cortical recording experiments examined how contextual factors, such as observing to imitate, observing while engaged in action, or observation with reduced visual information, modulated mirror activity in M1 and F5. Finally, single-pulse transcranial magnetic stimulation (TMS) in healthy human volunteers was used to examine changes in corticospinal excitability (CSE) during action observation and withholding. Overall, the results reveal distinctions in the profile of mirror activity across premotor and motor areas. While F5 maintains a more abstract representation of grasp independent of the acting agent, a balance of excitation and inhibition in motor cortex and spinal circuitry during action observation may support a flexible dissociation between initiation of grasping actions and representation of observed grasp