Rethinking Gauss-Newton for learning over-parameterized models

This work studies the global convergence and generalization properties of Gauss Newton's (GN) when optimizing one-hidden layer networks in the over-parameterized regime. We first establish a global convergence result for GN in the continuous-time limit exhibiting a faster convergence rate compared to GD due to improved conditioning. We then perform an empirical study on a synthetic regression task to investigate the implicit bias of GN's method. We find that, while GN is consistently faster than GD in finding a global optimum, the performance of the learned model on a test dataset is heavily influenced by both the learning rate and the variance of the randomly initialized network's weights. Specifically, we find that initializing with a smaller variance results in a better generalization, a behavior also observed for GD. However, in contrast to GD where larger learning rates lead to the best generalization, we find that GN achieves an improved generalization when using smaller learning rates, albeit at the cost of slower convergence. This study emphasizes the significance of the learning rate in balancing the optimization speed of GN with the generalization ability of the learned solution

The relevance of posterior thalamo-cortical connectivity for visual short-term memory capacity

Visual short-term memory (vSTM) capacity represents the maximum number of visual items that can be perceived and stored into vSTM. One way to measure it is by using simple psycho-physical experiments together with the theory of visual attention (TVA) computational framework in which visual processing is conceived as a race between objects to be consciously perceived and stored into vSTM. The neural theory of visual attention (NTVA), which gives an interpretation of the TVA at both the cellular and systemic level, suggests that recurrent loops between posterior thalamus and visual cortices are relevant for vSTM capacity. Nevertheless, no clear evidence for the role of posterior thalamus and its connection to visual cortices in vSTM capacity has been found thus far. This thesis investigated the role of posterior thalamo-cortical connectivity in vSTM capacity in healthy young individuals as well as in two populations that have shown to exhibit both vSTM capacity impairments and posterior cortical and subcortical white matter damages: healthy aging and premature birth. We found that vSTM capacity in healthy young adults was significantly associated with the tracts connecting posterior thalamus to occipital cortices and their microstructure. However, this association was modified in elderly individuals and in young adults born prematurely, in which the recruitment of additional, cortico-cortical, tracts, takes place. Together, these findings bring the first structural evidence for the NTVA model with respect to the relevance of posterior thalamo-cortical tracts for vSTM capacity and show how alterations of these tracts affect vSTM capacity

Self-Attention in Colors: Another Take on Encoding Graph Structure in Transformers

We introduce a novel self-attention mechanism, which we call CSA (Chromatic Self-Attention), which extends the notion of attention scores to attention _filters_, independently modulating the feature channels. We showcase CSA in a fully-attentional graph Transformer CGT (Chromatic Graph Transformer) which integrates both graph structural information and edge features, completely bypassing the need for local message-passing components. Our method flexibly encodes graph structure through node-node interactions, by enriching the original edge features with a relative positional encoding scheme. We propose a new scheme based on random walks that encodes both structural and positional information, and show how to incorporate higher-order topological information, such as rings in molecular graphs. Our approach achieves state-of-the-art results on the ZINC benchmark dataset, while providing a flexible framework for encoding graph structure and incorporating higher-order topology

The relevance of posterior thalamo-cortical connectivity for visual short-term memory capacity

Visual short-term memory (vSTM) capacity represents the maximum number of visual items that can be perceived and stored into vSTM. One way to measure it is by using simple psycho-physical experiments together with the theory of visual attention (TVA) computational framework in which visual processing is conceived as a race between objects to be consciously perceived and stored into vSTM. The neural theory of visual attention (NTVA), which gives an interpretation of the TVA at both the cellular and systemic level, suggests that recurrent loops between posterior thalamus and visual cortices are relevant for vSTM capacity. Nevertheless, no clear evidence for the role of posterior thalamus and its connection to visual cortices in vSTM capacity has been found thus far. This thesis investigated the role of posterior thalamo-cortical connectivity in vSTM capacity in healthy young individuals as well as in two populations that have shown to exhibit both vSTM capacity impairments and posterior cortical and subcortical white matter damages: healthy aging and premature birth. We found that vSTM capacity in healthy young adults was significantly associated with the tracts connecting posterior thalamus to occipital cortices and their microstructure. However, this association was modified in elderly individuals and in young adults born prematurely, in which the recruitment of additional, cortico-cortical, tracts, takes place. Together, these findings bring the first structural evidence for the NTVA model with respect to the relevance of posterior thalamo-cortical tracts for vSTM capacity and show how alterations of these tracts affect vSTM capacity

An analysis of MRI derived cortical complexity in premature-born adults : regional patterns, risk factors, and potential significance

Premature birth bears an increased risk for aberrant brain development concerning its structure and function. Cortical complexity (CC) expresses the fractal dimension of the brain surface and changes during neurodevelopment. We hypothesized that CC is altered after premature birth and associated with long-term cognitive development. One-hundred-and-one very premature-born adults (gestational age <32 weeks and/or birth weight <1500 ​g) and 111 term-born adults were assessed by structural MRI and cognitive testing at 26 years of age. CC was measured based on MRI by vertex-wise estimation of fractal dimension. Cognitive performance was measured based on Griffiths-Mental-Development-Scale (at 20 months) and Wechsler-Adult-Intelligence-Scales (at 26 years). In premature-born adults, CC was decreased bilaterally in large lateral temporal and medial parietal clusters. Decreased CC was associated with lower gestational age and birth weight. Furthermore, decreased CC in the medial parietal cortices was linked with reduced full-scale IQ of premature-born adults and mediated the association between cognitive development at 20 months and IQ in adulthood. Results demonstrate that CC is reduced in very premature-born adults in temporoparietal cortices, mediating the impact of prematurity on impaired cognitive development. These data indicate functionally relevant long-term alterations in the brain’s basic geometry of cortical organization in prematurity

Sequelae of premature birth in young adults

Background and Purpose Qualitative studies about the abnormalities appreciated on routine magnetic resonance imaging (MRI) sequences in prematurely born adults are lacking. This article aimed at filling this knowledge gap by (1) qualitatively describing routine imaging findings in prematurely born adults, (2) evaluating measures for routine image interpretation and (3) investigating the impact of perinatal variables related to premature birth. Methods In this study two board-certified radiologists assessed T1-weighted and FLAIR-weighted images of 100 prematurely born adults born very preterm (VP <32 weeks) and/or at very low birth weight (VLBW <1500 g) and 106 controls born at full term (FT) (mean age 26.8 ± 0.7 years). The number of white matter lesions (WML) was counted according to localization. Lateral ventricle volume (LVV) was evaluated subjectively and by measurements of Evans’ index (EI) and frontal-occipital-horn ratio (FOHR). Freesurfer-based volumetry served as reference standard. Miscellaneous incidental findings were noted as free text. Results The LVV was increased in 24.7% of VP/VLBW individuals and significantly larger than in FT controls. This was best identified by measurement of FOHR (AUC = 0.928). Ventricular enlargement was predicted by low gestational age (odds ratio: 0.71, 95% CI 0.51–0.98) and presence of neonatal intracranial hemorrhage (odds ratio: 0.26, 95% CI 0.07–0.92). The numbers of deep and periventricular WML were increased while subcortical WMLs were not. Conclusion Enlargement of the LVV and deep and periventricular WMLs are typical sequelae of premature birth that can be appreciated on routine brain MRI. To increase sensitivity of abnormal LVV detection, measurement of FOHR seems feasible in clinical practice

Complex Grey Matter Structure Segmentation in Brains via Deep Learning: Example of the Claustrum

Segmentationand parcellation of the brain has been widely performed on brain MRI using atlas-based methods. However, segmentation of the claustrum, a thin and sheet-like structure between insular cortex and putamen has not been amenable to automatized segmentation, thus limiting its investigation in larger imaging cohorts. Recently, deep-learning based approaches have been introduced for automated segmentation of brain structures, yielding great potential to overcome preexisting limitations. In the following, we present a multi-view deep-learning based approach to segment the claustrum in T1-weighted MRI scans. We trained and evaluated the proposed method on 181 manual bilateral claustrum annotations by an expert neuroradiologist serving as reference standard. Cross-validation experiments yielded median volumetric similarity, robust Hausdor? distance and Dice score of 93.3%, 1.41mm and 71.8% respectively which represents equal or superior segmentation performance compared to human intra-rater reliability. Leave-one-scanner-out evaluation showed good transfer-ability of the algorithm to images from unseen scanners, however at slightly inferior performance. Furthermore, we found that AI-based claustrum segmentation benefits from multi-view information and requires sample sizes of around 75 MRI scans in the training set. In conclusion, the developed algorithm has large potential in independent study cohorts and to facilitate MRI-based research of the human claustrum through automated segmentation. The software and models of our method are made publicly available.Comment: submitted to a journa

Theory of visual attention's thalamic model for visual short-term memory capacity and top-down control: evidence from a thalamo-cortical structural connectivity analysis

In the theory of visual attention (TVA), it is suggested that objects in a visual scene compete for representation in a visual short-term memory (vSTM) store. The race towards the store is assumed to be biased by top-down controlled weighting of the objects according to their task relevance. Only objects that reach the store before its capacity limitation is reached are represented consciously in a given instant. TVA-based computational modeling of participants' performance in whole- and partial-report tasks permits independent parameters of individual efficiency of top-down control α and vSTM storage capacity K to be extracted. The neural interpretation of the TVA proposes recurrent loops between the posterior thalamus and posterior visual cortices to be relevant for generating attentional weights for competing objects and for maintaining selected objects in vSTM. Accordingly, we tested whether structural connectivity between posterior thalamus and occipital cortices (PT-OC) is associated with estimates of top-down control and vSTM capacity. We applied whole- and partial-report tasks and probabilistic tractography in a sample of 37 healthy adults. We found vSTM capacity K to be associated with left PT-OC structural connectivity and a trend-wise relation between top-down control α and right PT-OC structural connectivity. These findings support the assumption of the relevance of thalamic structures and their connections to visual cortex for top-down control and vSTM capacity

Automated claustrum segmentation in human brain MRI using deep learning

In the last two decades, neuroscience has produced intriguing evidence for a central role of the claustrum in mammalian forebrain structure and function. However, relatively few in vivo studies of the claustrum exist in humans. A reason for this may be the delicate and sheet-like structure of the claustrum lying between the insular cortex and the putamen, which makes it not amenable to conventional segmentation methods. Recently, Deep Learning (DL) based approaches have been successfully introduced for automated segmentation of complex, subcortical brain structures. In the following, we present a multi-view DL-based approach to segment the claustrum in T1-weighted MRI scans. We trained and evaluated the proposed method in 181 individuals, using bilateral manual claustrum annotations by an expert neuroradiologist as reference standard. Cross-validation experiments yielded median volumetric similarity, robust Hausdorff distance, and Dice score of 93.3%, 1.41 mm, and 71.8%, respectively, representing equal or superior segmentation performance compared to human intra-rater reliability. The leave-one-scanner-out evaluation showed good transferability of the algorithm to images from unseen scanners at slightly inferior performance. Furthermore, we found that DL-based claustrum segmentation benefits from multi-view information and requires a sample size of around 75 MRI scans in the training set. We conclude that the developed algorithm allows for robust automated claustrum segmentation and thus yields considerable potential for facilitating MRI-based research of the human claustrum. The software and models of our method are made publicly available