With the development of deep convolutional neural networks, medical image
segmentation has achieved a series of breakthroughs in recent years. However,
the higher-performance convolutional neural networks always mean numerous
parameters and high computation costs, which will hinder the applications in
clinical scenarios. Meanwhile, the scarceness of large-scale annotated medical
image datasets further impedes the application of high-performance networks. To
tackle these problems, we propose Graph Flow, a comprehensive knowledge
distillation framework, for both network-efficiency and annotation-efficiency
medical image segmentation. Specifically, our core Graph Flow Distillation
transfer the essence of cross-layer variations from a well-trained cumbersome
teacher network to a non-trained compact student network. In addition, an
unsupervised Paraphraser Module is designed to purify the knowledge of the
teacher network, which is also beneficial for the stabilization of training
procedure. Furthermore, we build a unified distillation framework by
integrating the adversarial distillation and the vanilla logits distillation,
which can further refine the final predictions of the compact network.
Extensive experiments conducted on Gastric Cancer Segmentation Dataset and
Synapse Multi-organ Segmentation Dataset demonstrate the prominent ability of
our method which achieves state-of-the-art performance on these
different-modality and multi-category medical image datasets. Moreover, we
demonstrate the effectiveness of our Graph Flow through a new semi-supervised
paradigm for dual efficient medical image segmentation. Our code will be
available at Graph Flow