Deep Learning Models to Study Sentence Comprehension in the Human Brain

Recent artificial neural networks that process natural language achieve unprecedented performance in tasks requiring sentence-level understanding. As such, they could be interesting models of the integration of linguistic information in the human brain. We review works that compare these artificial language models with human brain activity and we assess the extent to which this approach has improved our understanding of the neural processes involved in natural language comprehension. Two main results emerge. First, the neural representation of word meaning aligns with the context-dependent, dense word vectors used by the artificial neural networks. Second, the processing hierarchy that emerges within artificial neural networks broadly matches the brain, but is surprisingly inconsistent across studies. We discuss current challenges in establishing artificial neural networks as process models of natural language comprehension. We suggest exploiting the highly structured representational geometry of artificial neural networks when mapping representations to brain data

Decoding the Real-Time Neurobiological Properties of Incremental Semantic Interpretation.

Communication through spoken language is a central human capacity, involving a wide range of complex computations that incrementally interpret each word into meaningful sentences. However, surprisingly little is known about the spatiotemporal properties of the complex neurobiological systems that support these dynamic predictive and integrative computations. Here, we focus on prediction, a core incremental processing operation guiding the interpretation of each upcoming word with respect to its preceding context. To investigate the neurobiological basis of how semantic constraints change and evolve as each word in a sentence accumulates over time, in a spoken sentence comprehension study, we analyzed the multivariate patterns of neural activity recorded by source-localized electro/magnetoencephalography (EMEG), using computational models capturing semantic constraints derived from the prior context on each upcoming word. Our results provide insights into predictive operations subserved by different regions within a bi-hemispheric system, which over time generate, refine, and evaluate constraints on each word as it is heard.ERC Horizon202

Learning under uncertainty in the young and older human brain: Common and distinct mechanisms of different attentional and intentional systems

The human brain is able to infer the probability of future events by combining information of past observations with current sensory input. Naturally, we are surrounded by more stimuli than we can pay attention to, so selection of relevant input is crucial. The present thesis aimed at identifying common and distinct neural correlates engaged in predictive processing in spatial attention (selection of attended locations) and motor intention (selection of prepared motor responses). Secondly, age-related influences on probabilistic inference in spatial-attention, feature-based attention (selection of attended color) and motor intention, and the impact of task difficulty were considered. Orienting attention during goal-directed behavior can be supported by visual cues, whereas reorienting to unexpected events following misguiding information is linked to behavioral costs and updating of predictions. These processes can be investigated with a cueing paradigm in which differences in reaction time (RT) between valid and invalidly cued trials increase with higher cue validity (%CV) (Posner, 1980). Bayesian models can describe the experience-dependent learning effects of inferring %CV, following novel events (Vossel et al., 2014c; Vossel, Mathys, Stephan & Friston, 2015). The principle aim of the first experiment was to identify and compare the neural correlates involved in inferring probabilities in the spatial attentional and motor intentional domain. Cues indicated either the possible location or prepared the motor response associated with the target. Instead of a fixed probability context, participants were exposed to a volatile environment, in which the validity of the cue information changed unpredictably over time. Combining functional magnetic resonance imaging (fMRI) data with behavioral estimates derived from a Bayesian learning model (Mathys, Daunizeau, Friston & Stephan, 2011) unveiled domain-specific predictability-dependent responses within the right temporoparietal junction (TPJ) for spatial attention and the left angular gyrus (ANG) and anterior cingulate (ACC) in the motor intention task. The blood oxygen level dependent (BOLD) amplitude particularly increased in accord with violations of cue predictability in high cue validity contexts (i.e. when invalid trials were least expected). Valid trials however, induced no (TPJ and ANG) or decreased modulation (ACC). A further aim was to examine possible commonalities in the neural signatures of predictability-dependent processing. Connectivity analysis uncovered common coupling of all three seed regions involved in predictability-dependent processing with the right anterior hippocampus. Since cognitive functions undergo substantial changes in healthy ageing, a second behavioral study was conducted to test whether age differentially influences probabilistic inference in different attentional subsystems, and how task difficulty impacts on learning performance. Thus, following up on the first experiment, similar tasks and the same computational model was used to assess updating behavior in healthy aging. Older and younger adults performed two separate experiments with different difficulty levels. Each experiment included three versions of a cueing task, entailing predictive spatial- (i.e. location), feature- (i.e. color of target) and motor intention cues (i.e. prepare response). Results of the easier version demonstrated a preserved ability of older adults to generate predictions and profit from all cue types. Interestingly, increased task demand uncovered a reduced ability to use motor intention cues to update predictions in older compared to younger adults. In conclusion, the results provide evidence for a segregated functional anatomy of probabilistic inference in spatial attention and motor intention. Nonetheless a common connectivity profile with the hippocampus also points at commonalities. Finally age seems to differentially impact the efficiency of learning behavior in the motor intention system, supporting the notion of independence of the attentional- and intentional subsystems