Systematic Overestimation of Machine Learning Performance in Neuroimaging Studies of Depression

We currently observe a disconcerting phenomenon in machine learning studies in psychiatry: While we would expect larger samples to yield better results due to the availability of more data, larger machine learning studies consistently show much weaker performance than the numerous small-scale studies. Here, we systematically investigated this effect focusing on one of the most heavily studied questions in the field, namely the classification of patients suffering from Major Depressive Disorder (MDD) and healthy controls. Drawing upon a balanced sample of $N = 1,868$ MDD patients and healthy controls from our recent international Predictive Analytics Competition (PAC), we first trained and tested a classification model on the full dataset which yielded an accuracy of 61%. Next, we mimicked the process by which researchers would draw samples of various sizes ($N=4$ to $N=150$) from the population and showed a strong risk of overestimation. Specifically, for small sample sizes ($N=20$), we observe accuracies of up to 95%. For medium sample sizes ($N=100$) accuracies up to 75% were found. Importantly, further investigation showed that sufficiently large test sets effectively protect against performance overestimation whereas larger datasets per se do not. While these results question the validity of a substantial part of the current literature, we outline the relatively low-cost remedy of larger test sets

Effects of polygenic risk for major mental disorders and cross-disorder on cortical complexity

Background MRI-derived cortical folding measures are an indicator of largely genetically driven early developmental processes. However, the effects of genetic risk for major mental disorders on early brain development are not well understood. Methods We extracted cortical complexity values from structural MRI data of 580 healthy participants using the CAT12 toolbox. Polygenic risk scores (PRS) for schizophrenia, bipolar disorder, major depression, and cross-disorder (incorporating cumulative genetic risk for depression, schizophrenia, bipolar disorder, autism spectrum disorder, and attention-deficit hyperactivity disorder) were computed and used in separate general linear models with cortical complexity as the regressand. In brain regions that showed a significant association between polygenic risk for mental disorders and cortical complexity, volume of interest (VOI)/region of interest (ROI) analyses were conducted to investigate additional changes in their volume and cortical thickness. Results The PRS for depression was associated with cortical complexity in the right orbitofrontal cortex (right hemisphere: p = 0.006). A subsequent VOI/ROI analysis showed no association between polygenic risk for depression and either grey matter volume or cortical thickness. We found no associations between cortical complexity and polygenic risk for either schizophrenia, bipolar disorder or psychiatric cross-disorder when correcting for multiple testing. Conclusions Changes in cortical complexity associated with polygenic risk for depression might facilitate well-established volume changes in orbitofrontal cortices in depression. Despite the absence of psychopathology, changed cortical complexity that parallels polygenic risk for depression might also change reward systems, which are also structurally affected in patients with depressive syndrome

Severity of current depression and remission status are associated with structural connectome alterations in major depressive disorder

Major depressive disorder (MDD) is associated to affected brain wiring. Little is known whether these changes are stable over time and hence might represent a biological predisposition, or whether these are state markers of current disease severity and recovery after a depressive episode. Human white matter network ("connectome") analysis via network science is a suitable tool to investigate the association between affected brain connectivity and MDD. This study examines structural connectome topology in 464 MDD patients (mean age: 36.6 years) and 432 healthy controls (35.6 years). MDD patients were stratified categorially by current disease status (acute vs. partial remission vs. full remission) based on DSM-IV criteria. Current symptom severity was assessed continuously via the Hamilton Depression Rating Scale (HAMD). Connectome matrices were created via a combination of T1-weighted magnetic resonance imaging (MRI) and tractography methods based on diffusion-weighted imaging. Global tract-based metrics were not found to show significant differences between disease status groups, suggesting conserved global brain connectivity in MDD. In contrast, reduced global fractional anisotropy (FA) was observed specifically in acute depressed patients compared to fully remitted patients and healthy controls. Within the MDD patients, FA in a subnetwork including frontal, temporal, insular, and parietal nodes was negatively associated with HAMD, an effect remaining when correcting for lifetime disease severity. Therefore, our findings provide new evidence of MDD to be associated with structural, yet dynamic, state-dependent connectome alterations, which covary with current disease severity and remission status after a depressive episode

Volume of the human hippocampus and clinical response following electroconvulsive therapy

BACKGROUND: Hippocampal enlargements are commonly reported after electroconvulsive therapy (ECT). To clarify mechanisms, we examined if ECT-induced hippocampal volume change relates to dose (number of ECT sessions and electrode placement) and acts as a biomarker of clinical outcome. METHODS: Longitudinal neuroimaging and clinical data from 10 independent sites participating in the Global ECT-Magnetic Resonance Imaging Research Collaboration (GEMRIC) were obtained for mega-analysis. Hippocampal volumes were extracted from structural magnetic resonance images, acquired before and after patients (n = 281) experiencing a major depressive episode completed an ECT treatment series using right unilateral and bilateral stimulation. Untreated nondepressed control subjects (n = 95) were scanned twice. RESULTS: The linear component of hippocampal volume change was 0.28% (SE 0.08) per ECT session (p < .001). Volume change varied by electrode placement in the left hippocampus (bilateral, 3.3 +/- 2.2%, d = 1.5; right unilateral, 1.6 +/- 2.1%, d = 0.8; p < .0001) but not the right hippocampus (bilateral, 3.0 +/- 1.7%, d = 1.8; right unilateral, 2.7 +/- 2.0%, d = 1.4; p = .36). Volume change for electrode placement per ECT session varied similarly by hemisphere. Individuals with greater treatment-related volume increases had poorer outcomes (Montgomery-Asberg Depression Rating Scale change -1.0 [SE 0.35], per 1% volume increase, p = .005), although the effects were not significant after controlling for ECT number (slope -0.69 [SE 0.38], p = .069). CONCLUSIONS: The number of ECT sessions and electrode placement impacts the extent and laterality of hippocampal enlargement, but volume change is not positively associated with clinical outcome. The results suggest that the high efficacy of ECT is not explained by hippocampal enlargement, which alone might not serve as a viable biomarker for treatment outcome

In vivo hippocampal subfield volumes in bipolar disorder—A mega-analysis from The Enhancing Neuro Imaging Genetics through Meta-Analysis Bipolar Disorder Working Group

The hippocampus consists of anatomically and functionally distinct subfields that may be differentially involved in the pathophysiology of bipolar disorder (BD). Here we, the Enhancing NeuroImaging Genetics through Meta‐Analysis Bipolar Disorder workinggroup, study hippocampal subfield volumetry in BD. T1‐weighted magnetic resonance imaging scans from 4,698 individuals (BD = 1,472, healthy controls [HC] = 3,226) from 23 sites worldwide were processed with FreeSurfer. We used linear mixed‐effects models and mega‐analysis to investigate differences in hippocampal subfield volumes between BD and HC, followed by analyses of clinical characteristics and medication use. BD showed significantly smaller volumes of the whole hippocampus (Cohen's d = −0.20), cornu ammonis (CA)1 (d = −0.18), CA2/3 (d = −0.11), CA4 (d = −0.19), molecular layer (d = −0.21), granule cell layer of dentate gyrus (d = −0.21), hippocampal tail (d = −0.10), subiculum (d = −0.15), presubiculum (d = −0.18), and hippocampal amygdala transition area (d = −0.17) compared to HC. Lithium users did not show volume differences compared to HC, while non‐users did. Antipsychotics or antiepileptic use was associated with smaller volumes. In this largest study of hippocampal subfields in BD to date, we show widespread reductions in nine of 12 subfields studied. The associations were modulated by medication use and specifically the lack of differences between lithium users and HC supports a possible protective role of lithium in BD

Apolipoprotein E Homozygous ε4 Allele Status: A Deteriorating Effect on Visuospatial Working Memory and Global Brain Structure

Theoretical background: The Apolipoprotein E (APOE) ε4 genotype is known to be one of the strongest single-gene predictors for Alzheimer disease, which is characterized by widespread brain structural degeneration progressing along with cognitive impairment. The ε4 allele status has been associated with brain structural alterations and lower cognitive ability in non-demented subjects. However, it remains unclear to what extent the visuospatial cognitive domain is affected, from what age onward changes are detectable and if alterations may interact with cognitive deficits in major depressive disorder (MDD). The current work investigated the effect of APOE ε4 homozygosity on visuospatial working memory (vWM) capacity, and on hippocampal morphometry. Furthermore, potential moderating roles of age and MDD were assessed.Methods: A sample of n = 31 homozygous ε4 carriers was contrasted with n = 31 non-ε4 carriers in a cross-sectional design. The sample consisted of non-demented, young to mid-age participants (mean age = 34.47; SD = 13.48; 51.6% female). Among them were n = 12 homozygous ε4 carriers and n = 12 non-ε4 carriers suffering from MDD (39%). VWM was assessed using the Corsi block-tapping task. Region of interest analyses of hippocampal gray matter density and volume were conducted using voxel-based morphometry (CAT12), and Freesurfer, respectively.Results: Homozygous ε4 carriers showed significantly lower Corsi span capacity than non-ε4 carriers did, and Corsi span capacity was associated with higher gray matter density of the hippocampus. APOE group differences in hippocampal volume could be detected but were no longer present when controlling for total intracranial volume. Hippocampal gray matter density did not differ between APOE groups. We did not find any interaction effects of age and MDD diagnosis on hippocampal morphometry.Conclusion: Our results point toward a negative association of homozygous ε4 allele status with vWM capacity already during mid-adulthood, which emerges independently of MDD diagnosis and age. APOE genotype seems to be associated with global brain structural rather than hippocampus specific alterations in young- to mid-age participants

The genetic architecture of the human cerebral cortex

The cerebral cortex underlies our complex cognitive capabilities, yet little is known about the specific genetic loci that influence human cortical structure. To identify genetic variants that affect cortical structure, we conducted a genome-wide association meta-analysis of brain magnetic resonance imaging data from 51,665 individuals. We analyzed the surface area and average thickness of the whole cortex and 34 regions with known functional specializations. We identified 199 significant loci and found significant enrichment for loci influencing total surface area within regulatory elements that are active during prenatal cortical development, supporting the radial unit hypothesis. Loci that affect regional surface area cluster near genes in Wnt signaling pathways, which influence progenitor expansion and areal identity. Variation in cortical structure is genetically correlated with cognitive function, Parkinson's disease, insomnia, depression, neuroticism, and attention deficit hyperactivity disorder

The genetic architecture of the human cerebral cortex

The cerebral cortex underlies our complex cognitive capabilities, yet little is known about the specific genetic loci that influence human cortical structure. To identify genetic variants that affect cortical structure, we conducted a genome-wide association meta-analysis of brain magnetic resonance imaging data from 51,665 individuals. We analyzed the surface area and average thickness of the whole cortex and 34 regions with known functional specializations. We identified 199 significant loci and found significant enrichment for loci influencing total surface area within regulatory elements that are active during prenatal cortical development, supporting the radial unit hypothesis. Loci that affect regional surface area cluster near genes in Wnt signaling pathways, which influence progenitor expansion and areal identity. Variation in cortical structure is genetically correlated with cognitive function, Parkinson's disease, insomnia, depression, neuroticism, and attention deficit hyperactivity disorder

Using structural MRI to identify bipolar disorders - 13 site machine learning study in 3020 individuals from the ENIGMA Bipolar Disorders Working Group

Bipolar disorders (BDs) are among the leading causes of morbidity and disability. Objective biological markers, such as those based on brain imaging, could aid in clinical management of BD. Machine learning (ML) brings neuroimaging analyses to individual subject level and may potentially allow for their diagnostic use. However, fair and optimal application of ML requires large, multi-site datasets. We applied ML (support vector machines) to MRI data (regional cortical thickness, surface area, subcortical volumes) from 853 BD and 2167 control participants from 13 cohorts in the ENIGMA consortium. We attempted to differentiate BD from control participants, investigated different data handling strategies and studied the neuroimaging/clinical features most important for classification. Individual site accuracies ranged from 45.23% to 81.07%. Aggregate subject-level analyses yielded the highest accuracy (65.23%, 95% CI = 63.47–67.00, ROC-AUC = 71.49%, 95% CI = 69.39–73.59), followed by leave-one-site-out cross-validation (accuracy = 58.67%, 95% CI = 56.70–60.63). Meta-analysis of individual site accuracies did not provide above chance results. There was substantial agreement between the regions that contributed to identification of BD participants in the best performing site and in the aggregate dataset (Cohen’s Kappa = 0.83, 95% CI = 0.829–0.831). Treatment with anticonvulsants and age were associated with greater odds of correct classification. Although short of the 80% clinically relevant accuracy threshold, the results are promising and provide a fair and realistic estimate of classification performance, which can be achieved in a large, ecologically valid, multi-site sample of BD participants based on regional neurostructural measures. Furthermore, the significant classification in different samples was based on plausible and similar neuroanatomical features. Future multi-site studies should move towards sharing of raw/voxelwise neuroimaging data

10Kin1day: A Bottom-Up Neuroimaging Initiative.

We organized 10Kin1day, a pop-up scientific event with the goal to bring together neuroimaging groups from around the world to jointly analyze 10,000+ existing MRI connectivity datasets during a 3-day workshop. In this report, we describe the motivation and principles of 10Kin1day, together with a public release of 8,000+ MRI connectome maps of the human brain