325 research outputs found
Unpaired multi-modal segmentation via knowledge distillation
Multi-modal learning is typically performed with network architectures containing modality-specific layers and shared layers, utilizing co-registered images of different modalities. We propose a novel learning scheme for unpaired cross-modality image segmentation, with a highly compact architecture achieving superior segmentation accuracy. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI, and only employ modality-specific internal normalization layers which compute respective statistics. To effectively train such a highly compact model, we introduce a novel loss term inspired by knowledge distillation, by explicitly constraining the KL-divergence of our derived prediction distributions between modalities. We have extensively validated our approach on two multi-class segmentation problems: i) cardiac structure segmentation, and ii) abdominal organ segmentation. Different network settings, i.e., 2D dilated network and 3D U-net, are utilized to investigate our method's general efficacy. Experimental results on both tasks demonstrate that our novel multi-modal learning scheme consistently outperforms single-modal training and previous multi-modal approaches
Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation
The success of deep convolutional neural networks is partially attributed to
the massive amount of annotated training data. However, in practice, medical
data annotations are usually expensive and time-consuming to be obtained.
Considering multi-modality data with the same anatomic structures are widely
available in clinic routine, in this paper, we aim to exploit the prior
knowledge (e.g., shape priors) learned from one modality (aka., assistant
modality) to improve the segmentation performance on another modality (aka.,
target modality) to make up annotation scarcity. To alleviate the learning
difficulties caused by modality-specific appearance discrepancy, we first
present an Image Alignment Module (IAM) to narrow the appearance gap between
assistant and target modality data.We then propose a novel Mutual Knowledge
Distillation (MKD) scheme to thoroughly exploit the modality-shared knowledge
to facilitate the target-modality segmentation. To be specific, we formulate
our framework as an integration of two individual segmentors. Each segmentor
not only explicitly extracts one modality knowledge from corresponding
annotations, but also implicitly explores another modality knowledge from its
counterpart in mutual-guided manner. The ensemble of two segmentors would
further integrate the knowledge from both modalities and generate reliable
segmentation results on target modality. Experimental results on the public
multi-class cardiac segmentation data, i.e., MMWHS 2017, show that our method
achieves large improvements on CT segmentation by utilizing additional MRI data
and outperforms other state-of-the-art multi-modality learning methods.Comment: Accepted by AAAI 202
LightVessel: Exploring Lightweight Coronary Artery Vessel Segmentation via Similarity Knowledge Distillation
In recent years, deep convolution neural networks (DCNNs) have achieved great
prospects in coronary artery vessel segmentation. However, it is difficult to
deploy complicated models in clinical scenarios since high-performance
approaches have excessive parameters and high computation costs. To tackle this
problem, we propose \textbf{LightVessel}, a Similarity Knowledge Distillation
Framework, for lightweight coronary artery vessel segmentation. Primarily, we
propose a Feature-wise Similarity Distillation (FSD) module for semantic-shift
modeling. Specifically, we calculate the feature similarity between the
symmetric layers from the encoder and decoder. Then the similarity is
transferred as knowledge from a cumbersome teacher network to a non-trained
lightweight student network. Meanwhile, for encouraging the student model to
learn more pixel-wise semantic information, we introduce the Adversarial
Similarity Distillation (ASD) module. Concretely, the ASD module aims to
construct the spatial adversarial correlation between the annotation and
prediction from the teacher and student models, respectively. Through the ASD
module, the student model obtains fined-grained subtle edge segmented results
of the coronary artery vessel. Extensive experiments conducted on Clinical
Coronary Artery Vessel Dataset demonstrate that LightVessel outperforms various
knowledge distillation counterparts.Comment: 5 pages, 7 figures, conferenc
Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentation
The ability to scene understanding in adverse visual conditions, e.g.,
nighttime, has sparked active research for RGB-Thermal (RGB-T) semantic
segmentation. However, it is essentially hampered by two critical problems: 1)
the day-night gap of RGB images is larger than that of thermal images, and 2)
the class-wise performance of RGB images at night is not consistently higher or
lower than that of thermal images. we propose the first test-time adaptation
(TTA) framework, dubbed Night-TTA, to address the problems for nighttime RGBT
semantic segmentation without access to the source (daytime) data during
adaptation. Our method enjoys three key technical parts. Firstly, as one
modality (e.g., RGB) suffers from a larger domain gap than that of the other
(e.g., thermal), Imaging Heterogeneity Refinement (IHR) employs an interaction
branch on the basis of RGB and thermal branches to prevent cross-modal
discrepancy and performance degradation. Then, Class Aware Refinement (CAR) is
introduced to obtain reliable ensemble logits based on pixel-level distribution
aggregation of the three branches. In addition, we also design a specific
learning scheme for our TTA framework, which enables the ensemble logits and
three student logits to collaboratively learn to improve the quality of
predictions during the testing phase of our Night TTA. Extensive experiments
show that our method achieves state-of-the-art (SoTA) performance with a 13.07%
boost in mIoU
- …