The neural basis of audio-visual integration and adaptation

The brain integrates or segregates audio-visual signals effortlessly in everyday life. In order to do so, it needs to infer the causal structure by which the signals were generated. Although behavioural studies extensively characterized causal inference in audio-visual perception, the neural mechanisms are barely explored. The current thesis sheds light on these neural processes and demonstrates how the brain adapts to dynamic as well as long-term changes in the environmental statistics of audio-visual signals. In Chapter 1, I introduce the causal inference problem and demonstrate how spatial audiovisual signals are integrated at the behavioural as well as neural level. In Chapter 2, I describe methodological foundations for the following empirical chapters. In Chapter 3, I present the neural mechanisms of explicit causal inference and the representations of audio-visual space along the human cortical hierarchy. Chapter 4 reveals that the brain is able to use recent past to adapt to the dynamically changing environment. In Chapter 5, I discuss the neural substrates of encoding auditory space and its adaptive changes in response to spatially conflicting visual signals. Finally, in Chapter 6, I summarize the findings of the thesis, its contributions to the literature, and I outline directions for future research

Canonical Correlation Analysis and Partial Least Squares for identifying brain-behaviour associations: a tutorial and a comparative study

Canonical Correlation Analysis (CCA) and Partial Least Squares (PLS) are powerful multivariate methods for capturing associations across two modalities of data (e.g., brain and behaviour). However, when the sample size is similar or smaller than the number of variables in the data, CCA and PLS models may overfit, i.e., find spurious associations that generalise poorly to new data. Dimensionality reduction and regularized extensions of CCA and PLS have been proposed to address this problem, yet most studies using these approaches have some limitations. This work gives a theoretical and practical introduction into the most common CCA/PLS models and their regularized variants. We examine the limitations of standard CCA and PLS when the sample size is similar or smaller than the number of variables. We discuss how dimensionality reduction and regularization techniques address this problem and explain their main advantages and disadvantages. We highlight crucial aspects of the CCA/PLS analysis framework, including optimising the hyperparameters of the model and testing the identified associations for statistical significance. We apply the described CCA/PLS models to simulated data and real data from the Human Connectome Project and the Alzheimer's Disease Neuroimaging Initiative (both of n>500). We use both low and high dimensionality versions of each data (i.e., ratios between sample size and variables in the range of ∼1-10 and ∼0.1-0.01) to demonstrate the impact of data dimensionality on the models. Finally, we summarize the key lessons of the tutorial

Multiple Holdouts With Stability: Improving the Generalizability of Machine Learning Analyses of Brain-Behavior Relationships.

BACKGROUND:In 2009, the National Institute of Mental Health launched the Research Domain Criteria, an attempt to move beyond diagnostic categories and ground psychiatry within neurobiological constructs that combine different levels of measures (e.g., brain imaging and behavior). Statistical methods that can integrate such multimodal data, however, are often vulnerable to overfitting, poor generalization, and difficulties in interpreting the results. METHODS:We propose an innovative machine learning framework combining multiple holdouts and a stability criterion with regularized multivariate techniques, such as sparse partial least squares and kernel canonical correlation analysis, for identifying hidden dimensions of cross-modality relationships. To illustrate the approach, we investigated structural brain-behavior associations in an extensively phenotyped developmental sample of 345 participants (312 healthy and 33 with clinical depression). The brain data consisted of whole-brain voxel-based gray matter volumes, and the behavioral data included item-level self-report questionnaires and IQ and demographic measures. RESULTS:Both sparse partial least squares and kernel canonical correlation analysis captured two hidden dimensions of brain-behavior relationships: one related to age and drinking and the other one related to depression. The applied machine learning framework indicates that these results are stable and generalize well to new data. Indeed, the identified brain-behavior associations are in agreement with previous findings in the literature concerning age, alcohol use, and depression-related changes in brain volume. CONCLUSIONS:Multivariate techniques (such as sparse partial least squares and kernel canonical correlation analysis) embedded in our novel framework are promising tools to link behavior and/or symptoms to neurobiology and thus have great potential to contribute to a biologically grounded definition of psychiatric disorders

Brain-behaviour modes of covariation in healthy and clinically depressed young people

Abstract: Understanding how variations in dimensions of psychometrics, IQ and demographics relate to changes in brain connectivity during the critical developmental period of adolescence and early adulthood is a major challenge. This has particular relevance for mental health disorders where a failure to understand these links might hinder the development of better diagnostic approaches and therapeutics. Here, we investigated this question in 306 adolescents and young adults (14–24 y, 25 clinically depressed) using a multivariate statistical framework, based on canonical correlation analysis (CCA). By linking individual functional brain connectivity profiles to self-report questionnaires, IQ and demographic data we identified two distinct modes of covariation. The first mode mapped onto an externalization/internalization axis and showed a strong association with sex. The second mode mapped onto a well-being/distress axis independent of sex. Interestingly, both modes showed an association with age. Crucially, the changes in functional brain connectivity associated with changes in these phenotypes showed marked developmental effects. The findings point to a role for the default mode, frontoparietal and limbic networks in psychopathology and depression

Audiovisual adaptation is expressed in spatial and decisional codes.

Funder: EC | EC Seventh Framework Programm | FP7 Ideas: European Research Council (FP7-IDEAS-ERC - Specific Programme: ''Ideas'' Implementing the Seventh Framework Programme of the European Community for Research, Technological Development and Demonstration Activities (2007 to 2013)); Grant(s): ERC-2012-StG_20111109 multsensThe brain adapts dynamically to the changing sensory statistics of its environment. Recent research has started to delineate the neural circuitries and representations that support this cross-sensory plasticity. Combining psychophysics and model-based representational fMRI and EEG we characterized how the adult human brain adapts to misaligned audiovisual signals. We show that audiovisual adaptation is associated with changes in regional BOLD-responses and fine-scale activity patterns in a widespread network from Heschl's gyrus to dorsolateral prefrontal cortices. Audiovisual recalibration relies on distinct spatial and decisional codes that are expressed with opposite gradients and time courses across the auditory processing hierarchy. Early activity patterns in auditory cortices encode sounds in a continuous space that flexibly adapts to misaligned visual inputs. Later activity patterns in frontoparietal cortices code decisional uncertainty consistent with these spatial transformations. Our findings suggest that regions within the auditory processing hierarchy multiplex spatial and decisional codes to adapt flexibly to the changing sensory statistics in the environment

Connectome-based reservoir computing with the conn2res toolbox

Abstract The connection patterns of neural circuits form a complex network. How signaling in these circuits manifests as complex cognition and adaptive behaviour remains the central question in neuroscience. Concomitant advances in connectomics and artificial intelligence open fundamentally new opportunities to understand how connection patterns shape computational capacity in biological brain networks. Reservoir computing is a versatile paradigm that uses high-dimensional, nonlinear dynamical systems to perform computations and approximate cognitive functions. Here we present conn2res: an open-source Python toolbox for implementing biological neural networks as artificial neural networks. conn2res is modular, allowing arbitrary network architecture and dynamics to be imposed. The toolbox allows researchers to input connectomes reconstructed using multiple techniques, from tract tracing to noninvasive diffusion imaging, and to impose multiple dynamical systems, from spiking neurons to memristive dynamics. The versatility of the conn2res toolbox allows us to ask new questions at the confluence of neuroscience and artificial intelligence. By reconceptualizing function as computation, conn2res sets the stage for a more mechanistic understanding of structure-function relationships in brain networks

Connectome-based reservoir computing with the conn2res toolbox

AbstractThe connection patterns of neural circuits form a complex network. How signaling in these circuits manifests as complex cognition and adaptive behaviour remains the central question in neuroscience. Concomitant advances in connectomics and artificial intelligence open fundamentally new opportunities to understand how connection patterns shape computational capacity in biological brain networks. Reservoir computing is a versatile paradigm that uses high-dimensional, nonlinear dynamical systems to perform computations and approximate cognitive functions. Here we present : an open-source Python toolbox for implementing biological neural networks as artificial neural networks. is modular, allowing arbitrary network architecture and dynamics to be imposed. The toolbox allows researchers to input connectomes reconstructed using multiple techniques, from tract tracing to noninvasive diffusion imaging, and to impose multiple dynamical systems, from spiking neurons to memristive dynamics. The versatility of the toolbox allows us to ask new questions at the confluence of neuroscience and artificial intelligence. By reconceptualizing function as computation, sets the stage for a more mechanistic understanding of structure-function relationships in brain networks.</jats:p

Multiple Holdouts With Stability: Improving the Generalizability of Machine Learning Analyses of Brain–Behavior Relationships

BACKGROUND: In 2009, the National Institute of Mental Health launched the Research Domain Criteria, an attempt to move beyond diagnostic categories and ground psychiatry within neurobiological constructs that combine different levels of measures (e.g., brain imaging and behavior). Statistical methods that can integrate such multimodal data, however, are often vulnerable to overfitting, poor generalization, and difficulties in interpreting the results. METHODS: We propose an innovative machine learning framework combining multiple holdouts and a stability criterion with regularized multivariate techniques, such as sparse partial least squares and kernel canonical correlation analysis, for identifying hidden dimensions of cross-modality relationships. To illustrate the approach, we investigated structural brain–behavior associations in an extensively phenotyped developmental sample of 345 participants (312 healthy and 33 with clinical depression). The brain data consisted of whole-brain voxel-based gray matter volumes, and the behavioral data included item-level self-report questionnaires and IQ and demographic measures. RESULTS: Both sparse partial least squares and kernel canonical correlation analysis captured two hidden dimensions of brain–behavior relationships: one related to age and drinking and the other one related to depression. The applied machine learning framework indicates that these results are stable and generalize well to new data. Indeed, the identified brain–behavior associations are in agreement with previous findings in the literature concerning age, alcohol use, and depression-related changes in brain volume. CONCLUSIONS: Multivariate techniques (such as sparse partial least squares and kernel canonical correlation analysis) embedded in our novel framework are promising tools to link behavior and/or symptoms to neurobiology and thus have great potential to contribute to a biologically grounded definition of psychiatric disorders.Wellcome Trust Strategic Award No. 095844Wellcome Centre for Human Neuroimaging Grant No. 203147/Z/16/ZWellcome Trust Grant No. WT102845/Z/13/ZFundacao para a Ciencia e a Tecnologia No. SFRH/BD/120640/2016University College London Hospitals (UCLH)National Institute for Health Research (NIHR)Biomedical Research Centre (BRC)Medical Research Council (MRC) Skills Development Fellowship Grant No. MR/ S007806/1NIHR Senior Investigator Award Grant No. NF-SI-0514-10157NIHR Collabora- tion for Leadership in Applied Health Research and Care (CLAHRC) North Thames at Barts Health NHS TrustNational Health ServiceUniversity of CambridgeGlaxoSmithKlin

Two latent dimensions linking multi-featured brain structure to behaviour in healthy adults

The development of solid theories of brain-behaviour relationships is weakened by a replication crisis and publication biases1,2⁠. To overcome this, data-driven multivariate approaches that integrate neurobiology and behaviour have been promoted. For instance, Canonical Correlation Analysis (CCA) can identify sets of behavioural variables that correlate with sets of brain variables (i.e., latent dimensions). With such approach, studies in young healthy adults found latent dimensions associated with mood and cognitive-control/general-intelligence, but brain patterns were relatively inconsistent3–5⁠. More robust results could be found with a regularized version of CCA (RCCA) embedded in a machine learning framework, assessing generalizability (evaluating how the model performs in new data) and stability (if similar variables are selected in different data splits)6⁠. With this framework, we studied robust latent dimensions linking behaviour and multi-featured brain structure in young healthy adults.We used T1 anatomical scans of the Human Connectome Project Young Adult sample (S1200, n: 1047, age: 28.78 ±3.67 (mean ±sd); 560 females). Grey Matter Volume (GMV) was estimated with CAT12, and Cortical Thickness (CT) and Surface Area (SA) with FreeSurfer v5.3. Brain data were averaged by regions (200 cortical, 73 subcortical) using the Schaefer atlas7⁠. Behavioural measures spanned cognition, emotion, mental health and life outcome. RCCA linked the multi-featured structural brain data (concatenating GMV, CT and SA) with behaviour. In the main analysis, age and gender were excluded of the model whilst in a supplementary analysis both were regressed out from brain and behavioural data. A multiple holdout machine learning framework6,8⁠ was used for model selection and statistical inference. In particular, generalizability and stability were used to find the optimal regularization parameters. Significance of the latent dimensions was assessed with 1000 permutations respecting the family structure of the data.We found two latent dimensions. The first dimension (r=0.43-0.55, p=0.015) was positively associated with high cognitive abilities (such as fluid intelligence, working memory and executive functions) and self/goal-driven aspects (Fig. 1A). This dimension was positively associated with anterior temporal and medial prefrontal (pole) regions (GMV and SA), amygdala and hippocampus (GMV), and insula (SA and CT) (Fig 1B). This dimension remained significant when regressing out gender and age. The second dimension (r=0.27-0.41, p=0.015) was positively associated with cognition (including attention, language and episodic memory) and negatively associated with negative social behaviour (such as rule breaking and aggression) (Fig 2A). Brain associations with this dimension were positive in visual regions (CT) and putamen (GMV), and negative in the cerebellum (GMV), superior frontal regions (CT), as well as Broca’s area and visual, paralimbic and language regions (SA) (Fig 2B). Importantly, this dimension did not remain significant when regressing out gender and age.Two dimensions of structural brain-behaviour interindividual variability were found. The first dimension mainly reflected higher cognitive and self-driven functions. Accordingly, it was associated with brain structural variability in higher-hierarchical regions of the self-centric and motivation network. The second dimension appears to mainly reflect education-related performance aspects and was negatively associated with negative social behaviour. Importantly, this dimension did not remain significant when removing variance related to gender categories and age, suggesting the influence of demographical factors on this brain-behaviour dimension. Hence, future studies should investigate the influence of genetic vs environmental factors on these dimensions, in particular with heritability assessment. Better characterizing these dimensions should help to prevent pathological extremes

Heritable and robust latent dimension linking cognition and multi-featured brain structure in healthy adults

Multivariate methods are promoted to study solid brain-behaviour relationships1,2. Canonical Correlation Analysis (CCA) has been used to analyse latent dimensions of brain-behaviour interindividual variability3–5⁠. However, more robust results can be found with regularised CCA (RCCA) and assessing generalizability (model performance in new data) and stability (if different data splits yield similar results)6,7⁠. In addition, the heritability of these dimensions would help understand their genetic load.We studied robust brain-behaviour latent dimensions (Human Connectome Project; n: 1047, mean age: 28.78; 560 females). Brain structure included region-wise Grey Matter Volume (GMV; CAT12), Cortical Thickness and Surface Area (CT and SA respectively; FreeSurfer v5.3). Behavioural data spanned cognition, emotion and mental health. We used RCCA and a novel machine learning framework to optimise generalisability and stability6,7⁠. We also studied the heritability of the latent dimension.We found one significant latent dimension (r=0.46-0.32, p=0.04) positively associated with cognitive-control/executive-functions. CT loadings were negative in associative areas and positive in sensoriomotor areas. SA and GMV loadings were positive on the temporal pole, inferior temporal gyri, pars orbitalis, anterior cingulate cortex and postcentral gyri. GMV loadings were negative in cerebellum and basal ganglia. Heritability of brain (h2=0.85) and behavioural scores (h2=0.82) and their genetic correlation (rhog=0.66) were significant (p<0.000).The dimension found captured variability from higher to lower cognitive-control/executive-functions. Accordingly, brain loadings ranged from lower to higher regions of the brain cortex and were similar to the pattern of cortical expansion during evolution and human development. The brain and behavioural scores were found to be heritable and to have shared genetic factors. Better characterizing these dimensions should help to prevent pathological extremes