Improving 3D convolutional neural network comprehensibility via interactive visualization of relevance maps: Evaluation in Alzheimer's disease

Background: Although convolutional neural networks (CNN) achieve high diagnostic accuracy for detecting Alzheimer's disease (AD) dementia based on magnetic resonance imaging (MRI) scans, they are not yet applied in clinical routine. One important reason for this is a lack of model comprehensibility. Recently developed visualization methods for deriving CNN relevance maps may help to fill this gap. We investigated whether models with higher accuracy also rely more on discriminative brain regions predefined by prior knowledge. Methods: We trained a CNN for the detection of AD in N=663 T1-weighted MRI scans of patients with dementia and amnestic mild cognitive impairment (MCI) and verified the accuracy of the models via cross-validation and in three independent samples including N=1655 cases. We evaluated the association of relevance scores and hippocampus volume to validate the clinical utility of this approach. To improve model comprehensibility, we implemented an interactive visualization of 3D CNN relevance maps. Results: Across three independent datasets, group separation showed high accuracy for AD dementia vs. controls (AUC$\geq$0.92) and moderate accuracy for MCI vs. controls (AUC$\approx$0.75). Relevance maps indicated that hippocampal atrophy was considered as the most informative factor for AD detection, with additional contributions from atrophy in other cortical and subcortical regions. Relevance scores within the hippocampus were highly correlated with hippocampal volumes (Pearson's r$\approx$-0.86, p<0.001). Conclusion: The relevance maps highlighted atrophy in regions that we had hypothesized a priori. This strengthens the comprehensibility of the CNN models, which were trained in a purely data-driven manner based on the scans and diagnosis labels.Comment: 24 pages, 9 figures/tables, supplementary material, source code available on GitHu

Visual Feature Attribution using Wasserstein GANs

Attributing the pixels of an input image to a certain category is an important and well-studied problem in computer vision, with applications ranging from weakly supervised localisation to understanding hidden effects in the data. In recent years, approaches based on interpreting a previously trained neural network classifier have become the de facto state-of-the-art and are commonly used on medical as well as natural image datasets. In this paper, we discuss a limitation of these approaches which may lead to only a subset of the category specific features being detected. To address this problem we develop a novel feature attribution technique based on Wasserstein Generative Adversarial Networks (WGAN), which does not suffer from this limitation. We show that our proposed method performs substantially better than the state-of-the-art for visual attribution on a synthetic dataset and on real 3D neuroimaging data from patients with mild cognitive impairment (MCI) and Alzheimer's disease (AD). For AD patients the method produces compellingly realistic disease effect maps which are very close to the observed effects.Comment: Accepted to CVPR 201

Explainable deep learning classifiers for disease detection based on structural brain MRI data

In dieser Doktorarbeit wird die Frage untersucht, wie erfolgreich deep learning bei der Diagnostik von neurodegenerativen Erkrankungen unterstützen kann. In 5 experimentellen Studien wird die Anwendung von Convolutional Neural Networks (CNNs) auf Daten der Magnetresonanztomographie (MRT) untersucht. Ein Schwerpunkt wird dabei auf die Erklärbarkeit der eigentlich intransparenten Modelle gelegt. Mit Hilfe von Methoden der erklärbaren künstlichen Intelligenz (KI) werden Heatmaps erstellt, die die Relevanz einzelner Bildbereiche für das Modell darstellen. Die 5 Studien dieser Dissertation zeigen das Potenzial von CNNs zur Krankheitserkennung auf neurologischen MRT, insbesondere bei der Kombination mit Methoden der erklärbaren KI. Mehrere Herausforderungen wurden in den Studien aufgezeigt und Lösungsansätze in den Experimenten evaluiert. Über alle Studien hinweg haben CNNs gute Klassifikationsgenauigkeiten erzielt und konnten durch den Vergleich von Heatmaps zur klinischen Literatur validiert werden. Weiterhin wurde eine neue CNN Architektur entwickelt, spezialisiert auf die räumlichen Eigenschaften von Gehirn MRT Bildern.Deep learning and especially convolutional neural networks (CNNs) have a high potential of being implemented into clinical decision support software for tasks such as diagnosis and prediction of disease courses. This thesis has studied the application of CNNs on structural MRI data for diagnosing neurological diseases. Specifically, multiple sclerosis and Alzheimer’s disease were used as classification targets due to their high prevalence, data availability and apparent biomarkers in structural MRI data. The classification task is challenging since pathology can be highly individual and difficult for human experts to detect and due to small sample sizes, which are caused by the high acquisition cost and sensitivity of medical imaging data. A roadblock in adopting CNNs to clinical practice is their lack of interpretability. Therefore, after optimizing the machine learning models for predictive performance (e.g. balanced accuracy), we have employed explainability methods to study the reliability and validity of the trained models. The deep learning models achieved good predictive performance of over 87% balanced accuracy on all tasks and the explainability heatmaps showed coherence with known clinical biomarkers for both disorders. Explainability methods were compared quantitatively using brain atlases and shortcomings regarding their robustness were revealed. Further investigations showed clear benefits of transfer-learning and image registration on the model performance. Lastly, a new CNN layer type was introduced, which incorporates a prior on the spatial homogeneity of neuro-MRI data. CNNs excel when used on natural images which possess spatial heterogeneity, and even though MRI data and natural images share computational similarities, the composition and orientation of neuro-MRI is very distinct. The introduced patch-individual filter (PIF) layer breaks the assumption of spatial invariance of CNNs and reduces convergence time on different data sets without reducing predictive performance. The presented work highlights many challenges that CNNs for disease diagnosis face on MRI data and defines as well as tests strategies to overcome those

Large-scale neuroimaging in Alzheimer’s disease and normal aging

Large-scale neuroimaging data is becoming increasingly available, providing a rich data source with which to study neurological conditions. In this thesis, I demonstrate the utility of large-scale neuroimaging as it applies to Alzheimer’s disease (AD) and normal aging, using univariate parametric mapping, regional analysis, and advanced machine learning. Specifically, this thesis covers: 1) validation and extension of prior studies using large-scale datasets; 2) AD diagnosis and normal aging evaluation empowered by large-scale datasets and advanced deep learning algorithms; 3) enhancement of cerebral blood volume (CBV) fMRI utility with retrospective CBV-fMRI technique. First, I demonstrated the utility of large-scale datasets for validating and extending prior studies using univariate analytics. I presented a study localizing AD-vulnerable regions more reliably and with better anatomical resolution using data from more than 350 subjects. Following a similar approach, I investigated the structural characteristics of healthy APOE ε4 homozygous subjects screened from a large-scale community-based study. To study the neuroimaging signatures of normal aging, we performed a large-scale joint CBV-fMRI and structural MRI study covering age 20-70s, and a structural MRI study of normal aging covering the full age-span, with the elder group screened from a large-scale clinic-based study ensuring no evidence of AD using both longitudinal follow-up and cerebrospinal fluid (CSF) biomarkers evidences. Second, I performed deep learning neuroimaging studies for AD diagnosis and normal aging evaluation, and investigated the regionality associated with each task. I developed an AD diagnosis method using a 3D convolutional neural network model trained and evaluated on ~4,600 structural MRI scans and further investigated a series of novel regionality analyses. I further extensively studied the utility of the structural MRI summary measure derived from the deep learning model in prodromal AD detection. This study constitutes a general analytic framework, which was followed to evaluate normal aging by performing deep learning-based age estimation in cognitively normal population using more than 6,000 scans. The deep learning neuroimaging models classified AD and estimated age with high accuracy, and also revealed regional patterns conforming to neuropathophysiology. The deep learning derived MRI measure demonstrated potential clinical utility, outperforming other AD pathology measures and biomarkers. In addition, I explored the utility of deep learning on positron emission tomography (PET) data for AD diagnosis and regionality analyses, further demonstrating the broad utility and generalizability of the method. Finally, I introduced a technique enabling CBV generation retrospectively from clinical contrast-enhanced scans. The derivation of meaningful functional measures from such clinical scans is only possible through calibration to a reference, which was built from the largest collection of research CBV-fMRI scans from our lab. This method was validated in an epilepsy study and demonstrated the potential to enhance the utility of CBV-fMRI by enriching the CBV-fMRI dataset. This technique is also applicable to AD and normal aging studies, and potentially enables deep learning based analytic approaches applied on CBV-fMRI with similar pipelines used in structural MRI. Collectively, this thesis demonstrates how mechanistic and diagnostic information on brain disorders can be extracted from large-scale neuroimaging data, using both classical statistical methods and advanced machine learning

Deep Networks Based Energy Models for Object Recognition from Multimodality Images

Object recognition has been extensively investigated in computer vision area, since it is a fundamental and essential technique in many important applications, such as robotics, auto-driving, automated manufacturing, and security surveillance. According to the selection criteria, object recognition mechanisms can be broadly categorized into object proposal and classification, eye fixation prediction and saliency object detection. Object proposal tends to capture all potential objects from natural images, and then classify them into predefined groups for image description and interpretation. For a given natural image, human perception is normally attracted to the most visually important regions/objects. Therefore, eye fixation prediction attempts to localize some interesting points or small regions according to human visual system (HVS). Based on these interesting points and small regions, saliency object detection algorithms propagate the important extracted information to achieve a refined segmentation of the whole salient objects. In addition to natural images, object recognition also plays a critical role in clinical practice. The informative insights of anatomy and function of human body obtained from multimodality biomedical images such as magnetic resonance imaging (MRI), transrectal ultrasound (TRUS), computed tomography (CT) and positron emission tomography (PET) facilitate the precision medicine. Automated object recognition from biomedical images empowers the non-invasive diagnosis and treatments via automated tissue segmentation, tumor detection and cancer staging. The conventional recognition methods normally utilize handcrafted features (such as oriented gradients, curvature, Haar features, Haralick texture features, Laws energy features, etc.) depending on the image modalities and object characteristics. It is challenging to have a general model for object recognition. Superior to handcrafted features, deep neural networks (DNN) can extract self-adaptive features corresponding with specific task, hence can be employed for general object recognition models. These DNN-features are adjusted semantically and cognitively by over tens of millions parameters corresponding to the mechanism of human brain, therefore leads to more accurate and robust results. Motivated by it, in this thesis, we proposed DNN-based energy models to recognize object on multimodality images. For the aim of object recognition, the major contributions of this thesis can be summarized below: 1. We firstly proposed a new comprehensive autoencoder model to recognize the position and shape of prostate from magnetic resonance images. Different from the most autoencoder-based methods, we focused on positive samples to train the model in which the extracted features all come from prostate. After that, an image energy minimization scheme was applied to further improve the recognition accuracy. The proposed model was compared with three classic classifiers (i.e. support vector machine with radial basis function kernel, random forest, and naive Bayes), and demonstrated significant superiority for prostate recognition on magnetic resonance images. We further extended the proposed autoencoder model for saliency object detection on natural images, and the experimental validation proved the accurate and robust saliency object detection results of our model. 2. A general multi-contexts combined deep neural networks (MCDN) model was then proposed for object recognition from natural images and biomedical images. Under one uniform framework, our model was performed in multi-scale manner. Our model was applied for saliency object detection from natural images as well as prostate recognition from magnetic resonance images. Our experimental validation demonstrated that the proposed model was competitive to current state-of-the-art methods. 3. We designed a novel saliency image energy to finely segment salient objects on basis of our MCDN model. The region priors were taken into account in the energy function to avoid trivial errors. Our method outperformed state-of-the-art algorithms on five benchmarking datasets. In the experiments, we also demonstrated that our proposed saliency image energy can boost the results of other conventional saliency detection methods

Deep Interpretability Methods for Neuroimaging

Brain dynamics are highly complex and yet hold the key to understanding brain function and dysfunction. The dynamics captured by resting-state functional magnetic resonance imaging data are noisy, high-dimensional, and not readily interpretable. The typical approach of reducing this data to low-dimensional features and focusing on the most predictive features comes with strong assumptions and can miss essential aspects of the underlying dynamics. In contrast, introspection of discriminatively trained deep learning models may uncover disorder-relevant elements of the signal at the level of individual time points and spatial locations. Nevertheless, the difficulty of reliable training on high-dimensional but small-sample datasets and the unclear relevance of the resulting predictive markers prevent the widespread use of deep learning in functional neuroimaging. In this dissertation, we address these challenges by proposing a deep learning framework to learn from high-dimensional dynamical data while maintaining stable, ecologically valid interpretations. The developed model is pre-trainable and alleviates the need to collect an enormous amount of neuroimaging samples to achieve optimal training. We also provide a quantitative validation module, Retain and Retrain (RAR), that can objectively verify the higher predictability of the dynamics learned by the model. Results successfully demonstrate that the proposed framework enables learning the fMRI dynamics directly from small data and capturing compact, stable interpretations of features predictive of function and dysfunction. We also comprehensively reviewed deep interpretability literature in the neuroimaging domain. Our analysis reveals the ongoing trend of interpretability practices in neuroimaging studies and identifies the gaps that should be addressed for effective human-machine collaboration in this domain. This dissertation also proposed a post hoc interpretability method, Geometrically Guided Integrated Gradients (GGIG), that leverages geometric properties of the functional space as learned by a deep learning model. With extensive experiments and quantitative validation on MNIST and ImageNet datasets, we demonstrate that GGIG outperforms integrated gradients (IG), which is considered to be a popular interpretability method in the literature. As GGIG is able to identify the contours of the discriminative regions in the input space, GGIG may be useful in various medical imaging tasks where fine-grained localization as an explanation is beneficial

State-dependent changes of connectivity patterns and functional brain network topology in Autism Spectrum Disorder

Anatomical and functional brain studies have converged to the hypothesis that Autism Spectrum Disorders (ASD) are associated with atypical connectivity. Using a modified resting-state paradigm to drive subjects' attention, we provide evidence of a very marked interaction between ASD brain functional connectivity and cognitive state. We show that functional connectivity changes in opposite ways in ASD and typicals as attention shifts from external world towards one's body generated information. Furthermore, ASD subject alter more markedly than typicals their connectivity across cognitive states. Using differences in brain connectivity across conditions, we classified ASD subjects at a performance around 80% while classification based on the connectivity patterns in any given cognitive state were close to chance. Connectivity between the Anterior Insula and dorsal-anterior Cingulate Cortex showed the highest classification accuracy and its strength increased with ASD severity. These results pave the path for diagnosis of mental pathologies based on functional brain networks obtained from a library of mental states

Gene-SGAN: a method for discovering disease subtypes with imaging and genetic signatures via multi-view weakly-supervised deep clustering

Disease heterogeneity has been a critical challenge for precision diagnosis and treatment, especially in neurologic and neuropsychiatric diseases. Many diseases can display multiple distinct brain phenotypes across individuals, potentially reflecting disease subtypes that can be captured using MRI and machine learning methods. However, biological interpretability and treatment relevance are limited if the derived subtypes are not associated with genetic drivers or susceptibility factors. Herein, we describe Gene-SGAN - a multi-view, weakly-supervised deep clustering method - which dissects disease heterogeneity by jointly considering phenotypic and genetic data, thereby conferring genetic correlations to the disease subtypes and associated endophenotypic signatures. We first validate the generalizability, interpretability, and robustness of Gene-SGAN in semi-synthetic experiments. We then demonstrate its application to real multi-site datasets from 28,858 individuals, deriving subtypes of Alzheimer's disease and brain endophenotypes associated with hypertension, from MRI and SNP data. Derived brain phenotypes displayed significant differences in neuroanatomical patterns, genetic determinants, biological and clinical biomarkers, indicating potentially distinct underlying neuropathologic processes, genetic drivers, and susceptibility factors. Overall, Gene-SGAN is broadly applicable to disease subtyping and endophenotype discovery, and is herein tested on disease-related, genetically-driven neuroimaging phenotypes

Association between resting-state functional connectivity, glucose metabolism and task-related activity of neural networks

The brain is organized into several large-scale functional networks. Such networks are primarily characterized by intrinsic functional connectivity, i.e. temporally synchronous activity between the different brain regions of a network. The functional connectivity of networks can be identified via functional MRI during resting state, i.e. without engaging the subject in a particular task. Resting-state fMRI is thus less demanding on the subject and therefore of particular interest from a clinical point of view to detect alterations in brain function. Applied to neurodegenerative disease including Alzheimer’s disease, resting-state fMRI has shown alterations in several resting-state networks, suggesting that basic network function is altered in AD. However, the interpretation of alterations in resting-state fMRI connectivity is inherently limited since no cognitive states are explicitly expressed during fMRI. In this regard, we aimed to elucidate how resting-state fMRI connectivity relates to 1) cognition-related brain activity and 2) markers of pathologically altered brain function in AD. In order to understand at a more basic level the association between resting-state and task-related fMRI, we first examined, in a group of elderly healthy subjects, the association between functional connectivity of major networks assessed during resting-state fMRI with those acquired during memory-task related fMRI, in the same individuals. Secondly, in order to assess whether alterations in AD are associated with already well-established markers of pathological brain function in AD, we compared resting-state fMRI functional network connectivity with that in FDG-PET metabolism in AD. Project 1: We investigated the association between functional connectivity acquired during rest and the level of activation obtained during an episodic memory task that included the encoding and forced-choice recognition of face-name pairs in elderly cognitively normal subjects. Independent component analysis (ICA) was used to identify major resting-state networks in the brain. Next, we applied ICA to the task-fMRI data to determine the components (networks) that were significantly associated with the task regressors of successful vs unsuccessful learning or recognition trials. Spatial correlation analysis between the resulting extracted resting-state and task-related fMRI components showed a spatial match in several components such as medial temporal lobe centered components and posterior components. However, apart from the spatial correspondence, the level of resting state functional connectivity did not predict the level of task-related functional connectivity in spatially matching components. Together these results suggested that particular resting-state networks are activated during a memory task, however, the level of baseline connectivity does not predict to what extent a network becomes activated during a task. Future studies may assess whether pathological resting-state connectivity predicts altered task-related connectivity in the same networks in AD. Project 2: We examined the association between resting-state fMRI functional connectivity within major functional networks and FDG-PET metabolism in those networks, assessed in elderly healthy controls, subjects with prodromal AD (mild cognitive impairment and amyloid PET biomarker confirmed AD etiology) and AD dementia. We found that FDG-PET was generally reduced in all networks in the course of AD. The main finding was that lower network functional connectivity was associated with lower FDG-PET uptake in the Default mode network and fronto-parietal attention network across the whole group and specifically in prodromal AD, suggesting that both modalities are associated in higher networks affected in the course of AD. These results provide insightful comprehension of the hypometabolism patterns that are typically found in AD