2,406 research outputs found
HeMIS: Hetero-Modal Image Segmentation
We introduce a deep learning image segmentation framework that is extremely
robust to missing imaging modalities. Instead of attempting to impute or
synthesize missing data, the proposed approach learns, for each modality, an
embedding of the input image into a single latent vector space for which
arithmetic operations (such as taking the mean) are well defined. Points in
that space, which are averaged over modalities available at inference time, can
then be further processed to yield the desired segmentation. As such, any
combinatorial subset of available modalities can be provided as input, without
having to learn a combinatorial number of imputation models. Evaluated on two
neurological MRI datasets (brain tumors and MS lesions), the approach yields
state-of-the-art segmentation results when provided with all modalities;
moreover, its performance degrades remarkably gracefully when modalities are
removed, significantly more so than alternative mean-filling or other synthesis
approaches.Comment: Accepted as an oral presentation at MICCAI 201
MyoPS-Net: Myocardial Pathology Segmentation with Flexible Combination of Multi-Sequence CMR Images
Myocardial pathology segmentation (MyoPS) can be a prerequisite for the
accurate diagnosis and treatment planning of myocardial infarction. However,
achieving this segmentation is challenging, mainly due to the inadequate and
indistinct information from an image. In this work, we develop an end-to-end
deep neural network, referred to as MyoPS-Net, to flexibly combine
five-sequence cardiac magnetic resonance (CMR) images for MyoPS. To extract
precise and adequate information, we design an effective yet flexible
architecture to extract and fuse cross-modal features. This architecture can
tackle different numbers of CMR images and complex combinations of modalities,
with output branches targeting specific pathologies. To impose anatomical
knowledge on the segmentation results, we first propose a module to regularize
myocardium consistency and localize the pathologies, and then introduce an
inclusiveness loss to utilize relations between myocardial scars and edema. We
evaluated the proposed MyoPS-Net on two datasets, i.e., a private one
consisting of 50 paired multi-sequence CMR images and a public one from
MICCAI2020 MyoPS Challenge. Experimental results showed that MyoPS-Net could
achieve state-of-the-art performance in various scenarios. Note that in
practical clinics, the subjects may not have full sequences, such as missing
LGE CMR or mapping CMR scans. We therefore conducted extensive experiments to
investigate the performance of the proposed method in dealing with such complex
combinations of different CMR sequences. Results proved the superiority and
generalizability of MyoPS-Net, and more importantly, indicated a practical
clinical application
FastSurfer-HypVINN: Automated sub-segmentation of the hypothalamus and adjacent structures on high-resolutional brain MRI
The hypothalamus plays a crucial role in the regulation of a broad range of
physiological, behavioural, and cognitive functions. However, despite its
importance, only a few small-scale neuroimaging studies have investigated its
substructures, likely due to the lack of fully automated segmentation tools to
address scalability and reproducibility issues of manual segmentation. While
the only previous attempt to automatically sub-segment the hypothalamus with a
neural network showed promise for 1.0 mm isotropic T1-weighted (T1w) MRI, there
is a need for an automated tool to sub-segment also high-resolutional (HiRes)
MR scans, as they are becoming widely available, and include structural detail
also from multi-modal MRI. We, therefore, introduce a novel, fast, and fully
automated deep learning method named HypVINN for sub-segmentation of the
hypothalamus and adjacent structures on 0.8 mm isotropic T1w and T2w brain MR
images that is robust to missing modalities. We extensively validate our model
with respect to segmentation accuracy, generalizability, in-session test-retest
reliability, and sensitivity to replicate hypothalamic volume effects (e.g.
sex-differences). The proposed method exhibits high segmentation performance
both for standalone T1w images as well as for T1w/T2w image pairs. Even with
the additional capability to accept flexible inputs, our model matches or
exceeds the performance of state-of-the-art methods with fixed inputs. We,
further, demonstrate the generalizability of our method in experiments with 1.0
mm MR scans from both the Rhineland Study and the UK Biobank. Finally, HypVINN
can perform the segmentation in less than a minute (GPU) and will be available
in the open source FastSurfer neuroimaging software suite, offering a
validated, efficient, and scalable solution for evaluating imaging-derived
phenotypes of the hypothalamus.Comment: Submitted to Imaging Neuroscienc
A Survey on Deep Learning in Medical Image Analysis
Deep learning algorithms, in particular convolutional networks, have rapidly
become a methodology of choice for analyzing medical images. This paper reviews
the major deep learning concepts pertinent to medical image analysis and
summarizes over 300 contributions to the field, most of which appeared in the
last year. We survey the use of deep learning for image classification, object
detection, segmentation, registration, and other tasks and provide concise
overviews of studies per application area. Open challenges and directions for
future research are discussed.Comment: Revised survey includes expanded discussion section and reworked
introductory section on common deep architectures. Added missed papers from
before Feb 1st 201
Enhancing Modality-Agnostic Representations via Meta-Learning for Brain Tumor Segmentation
In medical vision, different imaging modalities provide complementary
information. However, in practice, not all modalities may be available during
inference or even training. Previous approaches, e.g., knowledge distillation
or image synthesis, often assume the availability of full modalities for all
patients during training; this is unrealistic and impractical due to the
variability in data collection across sites. We propose a novel approach to
learn enhanced modality-agnostic representations by employing a meta-learning
strategy in training, even when only limited full modality samples are
available. Meta-learning enhances partial modality representations to full
modality representations by meta-training on partial modality data and
meta-testing on limited full modality samples. Additionally, we co-supervise
this feature enrichment by introducing an auxiliary adversarial learning
branch. More specifically, a missing modality detector is used as a
discriminator to mimic the full modality setting. Our segmentation framework
significantly outperforms state-of-the-art brain tumor segmentation techniques
in missing modality scenarios.Comment: Accepted in ICCV 202
MMMNA-Net for Overall Survival Time Prediction of Brain Tumor Patients
Overall survival (OS) time is one of the most important evaluation indices
for gliomas situations. Multimodal Magnetic Resonance Imaging (MRI) scans play
an important role in the study of glioma prognosis OS time. Several deep
learning-based methods are proposed for the OS time prediction on multi-modal
MRI problems. However, these methods usually fuse multi-modal information at
the beginning or at the end of the deep learning networks and lack the fusion
of features from different scales. In addition, the fusion at the end of
networks always adapts global with global (eg. fully connected after
concatenation of global average pooling output) or local with local (eg.
bilinear pooling), which loses the information of local with global. In this
paper, we propose a novel method for multi-modal OS time prediction of brain
tumor patients, which contains an improved nonlocal features fusion module
introduced on different scales. Our method obtains a relative 8.76% improvement
over the current state-of-art method (0.6989 vs. 0.6426 on accuracy). Extensive
testing demonstrates that our method could adapt to situations with missing
modalities. The code is available at
https://github.com/TangWen920812/mmmna-net.Comment: Accepted EMBC 202
Modality-Agnostic Learning for Medical Image Segmentation Using Multi-modality Self-distillation
Medical image segmentation of tumors and organs at risk is a time-consuming
yet critical process in the clinic that utilizes multi-modality imaging (e.g,
different acquisitions, data types, and sequences) to increase segmentation
precision. In this paper, we propose a novel framework, Modality-Agnostic
learning through Multi-modality Self-dist-illation (MAG-MS), to investigate the
impact of input modalities on medical image segmentation. MAG-MS distills
knowledge from the fusion of multiple modalities and applies it to enhance
representation learning for individual modalities. Thus, it provides a
versatile and efficient approach to handle limited modalities during testing.
Our extensive experiments on benchmark datasets demonstrate the high efficiency
of MAG-MS and its superior segmentation performance than current
state-of-the-art methods. Furthermore, using MAG-MS, we provide valuable
insight and guidance on selecting input modalities for medical image
segmentation tasks
Brain tumor segmentation with missing modalities via latent multi-source correlation representation
Multimodal MR images can provide complementary information for accurate brain
tumor segmentation. However, it's common to have missing imaging modalities in
clinical practice. Since there exists a strong correlation between multi
modalities, a novel correlation representation block is proposed to specially
discover the latent multi-source correlation. Thanks to the obtained
correlation representation, the segmentation becomes more robust in the case of
missing modalities. The model parameter estimation module first maps the
individual representation produced by each encoder to obtain independent
parameters, then, under these parameters, the correlation expression module
transforms all the individual representations to form a latent multi-source
correlation representation. Finally, the correlation representations across
modalities are fused via the attention mechanism into a shared representation
to emphasize the most important features for segmentation. We evaluate our
model on BraTS 2018 datasets, it outperforms the current state-of-the-art
method and produces robust results when one or more modalities are missing.Comment: 9 pages, 6 figures, accepted by MICCAI 202
- …