53 research outputs found
Uncovering convolutional neural network decisions for diagnosing multiple sclerosis on conventional MRI using layer-wise relevance propagation
Machine learning-based imaging diagnostics has recently reached or even
superseded the level of clinical experts in several clinical domains. However,
classification decisions of a trained machine learning system are typically
non-transparent, a major hindrance for clinical integration, error tracking or
knowledge discovery. In this study, we present a transparent deep learning
framework relying on convolutional neural networks (CNNs) and layer-wise
relevance propagation (LRP) for diagnosing multiple sclerosis (MS). MS is
commonly diagnosed utilizing a combination of clinical presentation and
conventional magnetic resonance imaging (MRI), specifically the occurrence and
presentation of white matter lesions in T2-weighted images. We hypothesized
that using LRP in a naive predictive model would enable us to uncover relevant
image features that a trained CNN uses for decision-making. Since imaging
markers in MS are well-established this would enable us to validate the
respective CNN model. First, we pre-trained a CNN on MRI data from the
Alzheimer's Disease Neuroimaging Initiative (n = 921), afterwards specializing
the CNN to discriminate between MS patients and healthy controls (n = 147).
Using LRP, we then produced a heatmap for each subject in the holdout set
depicting the voxel-wise relevance for a particular classification decision.
The resulting CNN model resulted in a balanced accuracy of 87.04% and an area
under the curve of 96.08% in a receiver operating characteristic curve. The
subsequent LRP visualization revealed that the CNN model focuses indeed on
individual lesions, but also incorporates additional information such as lesion
location, non-lesional white matter or gray matter areas such as the thalamus,
which are established conventional and advanced MRI markers in MS. We conclude
that LRP and the proposed framework have the capability to make diagnostic
decisions of..
Uncovering convolutional neural network decisions for diagnosing multiple sclerosis on conventional MRI using layer-wise relevance propagation
Machine learning-based imaging diagnostics has recently reached or even surpassed the level of clinical experts in several clinical domains. However, classification decisions of a trained machine learning system are typically non-transparent, a major hindrance for clinical integration, error tracking or knowledge discovery. In this study, we present a transparent deep learning framework relying on 3D convolutional neural networks (CNNs) and layer-wise relevance propagation (LRP) for diagnosing multiple sclerosis (MS), the most widespread autoimmune neuroinflammatory disease. MS is commonly diagnosed utilizing a combination of clinical presentation and conventional magnetic resonance imaging (MRI), specifically the occurrence and presentation of white matter lesions in T2-weighted images. We hypothesized that using LRP in a naive predictive model would enable us to uncover relevant image features that a trained CNN uses for decision-making. Since imaging markers in MS are well-established this would enable us to validate the respective CNN model. First, we pre-trained a CNN on MRI data from the Alzheimer's Disease Neuroimaging Initiative (n = 921), afterwards specializing the CNN to discriminate between MS patients (n = 76) and healthy controls (n = 71). Using LRP, we then produced a heatmap for each subject in the holdout set depicting the voxel-wise relevance for a particular classification decision. The resulting CNN model resulted in a balanced accuracy of 87.04% and an area under the curve of 96.08% in a receiver operating characteristic curve. The subsequent LRP visualization revealed that the CNN model focuses indeed on individual lesions, but also incorporates additional information such as lesion location, non-lesional white matter or gray matter areas such as the thalamus, which are established conventional and advanced MRI markers in MS. We conclude that LRP and the proposed framework have the capability to make diagnostic decisions of CNN models transparent, which could serve to justify classification decisions for clinical review, verify diagnosis-relevant features and potentially gather new disease knowledge
Explainable deep learning classifiers for disease detection based on structural brain MRI data
In dieser Doktorarbeit wird die Frage untersucht, wie erfolgreich deep learning bei der Diagnostik von neurodegenerativen Erkrankungen unterstützen kann. In 5 experimentellen Studien wird die Anwendung von Convolutional Neural Networks (CNNs) auf Daten der Magnetresonanztomographie (MRT) untersucht. Ein Schwerpunkt wird dabei auf die Erklärbarkeit der eigentlich intransparenten Modelle gelegt. Mit Hilfe von Methoden der erklärbaren künstlichen Intelligenz (KI) werden Heatmaps erstellt, die die Relevanz einzelner Bildbereiche für das Modell darstellen.
Die 5 Studien dieser Dissertation zeigen das Potenzial von CNNs zur Krankheitserkennung auf neurologischen MRT, insbesondere bei der Kombination mit Methoden der erklärbaren KI. Mehrere Herausforderungen wurden in den Studien aufgezeigt und Lösungsansätze in den Experimenten evaluiert. Über alle Studien hinweg haben CNNs gute Klassifikationsgenauigkeiten erzielt und konnten durch den Vergleich von Heatmaps zur klinischen Literatur validiert werden. Weiterhin wurde eine neue CNN Architektur entwickelt, spezialisiert auf die räumlichen Eigenschaften von Gehirn MRT Bildern.Deep learning and especially convolutional neural networks (CNNs) have a high potential of being implemented into clinical decision support software for tasks such as diagnosis and prediction of disease courses. This thesis has studied the application of CNNs on structural MRI data for diagnosing neurological diseases. Specifically, multiple sclerosis and Alzheimer’s disease were used as classification targets due to their high prevalence, data availability and apparent biomarkers in structural MRI data. The classification task is challenging since pathology can be highly individual and difficult for human experts to detect and due to small sample sizes, which are caused by the high acquisition cost and sensitivity of medical imaging data. A roadblock in adopting CNNs to clinical practice is their lack of interpretability. Therefore, after optimizing the machine learning models for predictive performance (e.g. balanced accuracy), we have employed explainability methods to study the reliability and validity of the trained models. The deep learning models achieved good predictive performance of over 87% balanced accuracy on all tasks and the explainability heatmaps showed coherence with known clinical biomarkers for both disorders. Explainability methods were compared quantitatively using brain atlases and shortcomings regarding their robustness were revealed. Further investigations showed clear benefits of transfer-learning and image registration on the model performance. Lastly, a new CNN layer type was introduced, which incorporates a prior on the spatial homogeneity of neuro-MRI data. CNNs excel when used on natural images which possess spatial heterogeneity, and even though MRI data and natural images share computational similarities, the composition and orientation of neuro-MRI is very distinct. The introduced patch-individual filter (PIF) layer breaks the assumption of spatial invariance of CNNs and reduces convergence time on different data sets without reducing predictive performance. The presented work highlights many challenges that CNNs for disease diagnosis face on MRI data and defines as well as tests strategies to overcome those
GAMER-MRIL identifies Disability-Related Brain Changes in Multiple Sclerosis
Objective: Identifying disability-related brain changes is important for
multiple sclerosis (MS) patients. Currently, there is no clear understanding
about which pathological features drive disability in single MS patients. In
this work, we propose a novel comprehensive approach, GAMER-MRIL, leveraging
whole-brain quantitative MRI (qMRI), convolutional neural network (CNN), and an
interpretability method from classifying MS patients with severe disability to
investigating relevant pathological brain changes. Methods:
One-hundred-sixty-six MS patients underwent 3T MRI acquisitions. qMRI
informative of microstructural brain properties was reconstructed, including
quantitative T1 (qT1), myelin water fraction (MWF), and neurite density index
(NDI). To fully utilize the qMRI, GAMER-MRIL extended a gated-attention-based
CNN (GAMER-MRI), which was developed to select patch-based qMRI important for a
given task/question, to the whole-brain image. To find out disability-related
brain regions, GAMER-MRIL modified a structure-aware interpretability method,
Layer-wise Relevance Propagation (LRP), to incorporate qMRI. Results: The test
performance was AUC=0.885. qT1 was the most sensitive measure related to
disability, followed by NDI. The proposed LRP approach obtained more
specifically relevant regions than other interpretability methods, including
the saliency map, the integrated gradients, and the original LRP. The relevant
regions included the corticospinal tract, where average qT1 and NDI
significantly correlated with patients' disability scores (=-0.37 and
0.44). Conclusion: These results demonstrated that GAMER-MRIL can classify
patients with severe disability using qMRI and subsequently identify brain
regions potentially important to the integrity of the mobile function.
Significance: GAMER-MRIL holds promise for developing biomarkers and increasing
clinicians' trust in NN
Testing the robustness of attribution methods for convolutional neural networks in MRI-based Alzheimer's disease classification
Attribution methods are an easy to use tool for investigating and validating
machine learning models. Multiple methods have been suggested in the literature
and it is not yet clear which method is most suitable for a given task. In this
study, we tested the robustness of four attribution methods, namely
gradient*input, guided backpropagation, layer-wise relevance propagation and
occlusion, for the task of Alzheimer's disease classification. We have
repeatedly trained a convolutional neural network (CNN) with identical training
settings in order to separate structural MRI data of patients with Alzheimer's
disease and healthy controls. Afterwards, we produced attribution maps for each
subject in the test data and quantitatively compared them across models and
attribution methods. We show that visual comparison is not sufficient and that
some widely used attribution methods produce highly inconsistent outcomes
Deciphering multiple sclerosis disability with deep learning attention maps on clinical MRI
Deep learning; Disability; Structural MRIAprendizaje profundo; Discapacidad; Resonancia magnética estructuralAprenentatge profund; Discapacitat; Ressonància magnètica estructuralThe application of convolutional neural networks (CNNs) to MRI data has emerged as a promising approach to achieving unprecedented levels of accuracy when predicting the course of neurological conditions, including multiple sclerosis, by means of extracting image features not detectable through conventional methods. Additionally, the study of CNN-derived attention maps, which indicate the most relevant anatomical features for CNN-based decisions, has the potential to uncover key disease mechanisms leading to disability accumulation.
From a cohort of patients prospectively followed up after a first demyelinating attack, we selected those with T1-weighted and T2-FLAIR brain MRI sequences available for image analysis and a clinical assessment performed within the following six months (N = 319). Patients were divided into two groups according to expanded disability status scale (EDSS) score: ≥3.0 and < 3.0. A 3D-CNN model predicted the class using whole-brain MRI scans as input. A comparison with a logistic regression (LR) model using volumetric measurements as explanatory variables and a validation of the CNN model on an independent dataset with similar characteristics (N = 440) were also performed. The layer-wise relevance propagation method was used to obtain individual attention maps.
The CNN model achieved a mean accuracy of 79% and proved to be superior to the equivalent LR-model (77%). Additionally, the model was successfully validated in the independent external cohort without any re-training (accuracy = 71%). Attention-map analyses revealed the predominant role of frontotemporal cortex and cerebellum for CNN decisions, suggesting that the mechanisms leading to disability accrual exceed the mere presence of brain lesions or atrophy and probably involve how damage is distributed in the central nervous system.MS PATHS is funded by Biogen. This study has been possible thanks to a Junior Leader La Caixa Fellowship awarded to C. Tur (fellowship code is LCF/BQ/PI20/11760008) by “la Caixa” Foundation (ID 100010434). The salaries of C. Tur and Ll. Coll are covered by this award
Harnessing spatial homogeneity of neuroimaging data: patch individual filter layers for CNNs
Neuroimaging data, e.g. obtained from magnetic resonance imaging (MRI), is
comparably homogeneous due to (1) the uniform structure of the brain and (2)
additional efforts to spatially normalize the data to a standard template using
linear and non-linear transformations. Convolutional neural networks (CNNs), in
contrast, have been specifically designed for highly heterogeneous data, such
as natural images, by sliding convolutional filters over different positions in
an image. Here, we suggest a new CNN architecture that combines the idea of
hierarchical abstraction in neural networks with a prior on the spatial
homogeneity of neuroimaging data: Whereas early layers are trained globally
using standard convolutional layers, we introduce for higher, more abstract
layers patch individual filters (PIF). By learning filters in individual image
regions (patches) without sharing weights, PIF layers can learn abstract
features faster and with fewer samples. We thoroughly evaluated PIF layers for
three different tasks and data sets, namely sex classification on UK Biobank
data, Alzheimer's disease detection on ADNI data and multiple sclerosis
detection on private hospital data. We demonstrate that CNNs using PIF layers
result in higher accuracies, especially in low sample size settings, and need
fewer training epochs for convergence. To the best of our knowledge, this is
the first study which introduces a prior on brain MRI for CNN learning
Deciphering multiple sclerosis disability with deep learning attention maps on clinical MRI
The application of convolutional neural networks (CNNs) to MRI data has emerged as a promising approach to achieving unprecedented levels of accuracy when predicting the course of neurological conditions, including multiple sclerosis, by means of extracting image features not detectable through conventional methods. Additionally, the study of CNN-derived attention maps, which indicate the most relevant anatomical features for CNN-based decisions, has the potential to uncover key disease mechanisms leading to disability accumulation. From a cohort of patients prospectively followed up after a first demyelinating attack, we selected those with T1-weighted and T2-FLAIR brain MRI sequences available for image analysis and a clinical assessment performed within the following six months (N = 319). Patients were divided into two groups according to expanded disability status scale (EDSS) score: ≥3.0 and < 3.0. A 3D-CNN model predicted the class using whole-brain MRI scans as input. A comparison with a logistic regression (LR) model using volumetric measurements as explanatory variables and a validation of the CNN model on an independent dataset with similar characteristics (N = 440) were also performed. The layer-wise relevance propagation method was used to obtain individual attention maps. The CNN model achieved a mean accuracy of 79% and proved to be superior to the equivalent LR-model (77%). Additionally, the model was successfully validated in the independent external cohort without any re-training (accuracy = 71%). Attention-map analyses revealed the predominant role of frontotemporal cortex and cerebellum for CNN decisions, suggesting that the mechanisms leading to disability accrual exceed the mere presence of brain lesions or atrophy and probably involve how damage is distributed in the central nervous system
Improving nuclear medicine with deep learning and explainability: two real-world use cases in parkinsonian syndrome and safety dosimetry
Computer vision in the area of medical imaging has rapidly improved during recent years as a consequence of developments in deep learning and explainability algorithms. In addition, imaging in nuclear medicine is becoming increasingly sophisticated, with the emergence of targeted radiotherapies that enable treatment and imaging on a molecular level (“theranostics”) where radiolabeled targeted molecules are directly injected into the bloodstream. Based on our recent work, we present two use-cases in nuclear medicine as follows: first, the impact of automated organ segmentation required for personalized dosimetry in patients with neuroendocrine tumors and second, purely data-driven identification and verification of brain regions for diagnosis of Parkinson’s disease. Convolutional neural network was used for automated organ segmentation on computed tomography images. The segmented organs were used for calculation of the energy deposited into the organ-at-risk for patients treated with a radiopharmaceutical. Our method resulted in faster and cheaper dosimetry and only differed by 7% from dosimetry performed by two medical physicists. The identification of brain regions, however was analyzed on dopamine-transporter single positron emission tomography images using convolutional neural network and explainability, i.e., layer-wise relevance propagation algorithm. Our findings confirm that the extra-striatal brain regions, i.e., insula, amygdala, ventromedial prefrontal cortex, thalamus, anterior temporal cortex, superior frontal lobe, and pons contribute to the interpretation of images beyond the striatal regions. In current common diagnostic practice, however, only the striatum is the reference region, while extra-striatal regions are neglected. We further demonstrate that deep learning-based diagnosis combined with explainability algorithm can be recommended to support interpretation of this image modality in clinical routine for parkinsonian syndromes, with a total computation time of three seconds which is compatible with busy clinical workflow.
Overall, this thesis shows for the first time that deep learning with explainability can achieve results competitive with human performance and generate novel hypotheses, thus paving the way towards improved diagnosis and treatment in nuclear medicine
- …