563 research outputs found
Bi-Modality Medical Image Synthesis Using Semi-Supervised Sequential Generative Adversarial Networks
In this paper, we propose a bi-modality medical image synthesis approach
based on sequential generative adversarial network (GAN) and semi-supervised
learning. Our approach consists of two generative modules that synthesize
images of the two modalities in a sequential order. A method for measuring the
synthesis complexity is proposed to automatically determine the synthesis order
in our sequential GAN. Images of the modality with a lower complexity are
synthesized first, and the counterparts with a higher complexity are generated
later. Our sequential GAN is trained end-to-end in a semi-supervised manner. In
supervised training, the joint distribution of bi-modality images are learned
from real paired images of the two modalities by explicitly minimizing the
reconstruction losses between the real and synthetic images. To avoid
overfitting limited training images, in unsupervised training, the marginal
distribution of each modality is learned based on unpaired images by minimizing
the Wasserstein distance between the distributions of real and fake images. We
comprehensively evaluate the proposed model using two synthesis tasks based on
three types of evaluate metrics and user studies. Visual and quantitative
results demonstrate the superiority of our method to the state-of-the-art
methods, and reasonable visual quality and clinical significance. Code is made
publicly available at
https://github.com/hustlinyi/Multimodal-Medical-Image-Synthesis
Recommended from our members
Deep learning for cardiac image segmentation: A review
Deep learning has become the most widely used approach for cardiac image segmentation in recent years. In this paper, we provide a review of over 100 cardiac image segmentation papers using deep learning, which covers common imaging modalities including magnetic resonance imaging (MRI), computed tomography (CT), and ultrasound (US) and major anatomical structures of interest (ventricles, atria and vessels). In addition, a summary of publicly available cardiac image datasets and code repositories are included to provide a base for encouraging reproducible research. Finally, we discuss the challenges and limitations with current deep learning-based approaches (scarcity of labels, model generalizability across different domains, interpretability) and suggest potential directions for future research
Learning Disentangled Representations in the Imaging Domain
Disentangled representation learning has been proposed as an approach to
learning general representations even in the absence of, or with limited,
supervision. A good general representation can be fine-tuned for new target
tasks using modest amounts of data, or used directly in unseen domains
achieving remarkable performance in the corresponding task. This alleviation of
the data and annotation requirements offers tantalising prospects for
applications in computer vision and healthcare. In this tutorial paper, we
motivate the need for disentangled representations, present key theory, and
detail practical building blocks and criteria for learning such
representations. We discuss applications in medical imaging and computer vision
emphasising choices made in exemplar key works. We conclude by presenting
remaining challenges and opportunities.Comment: Submitted. This paper follows a tutorial style but also surveys a
considerable (more than 200 citations) number of work
YoloCurvSeg: You Only Label One Noisy Skeleton for Vessel-style Curvilinear Structure Segmentation
Weakly-supervised learning (WSL) has been proposed to alleviate the conflict
between data annotation cost and model performance through employing
sparsely-grained (i.e., point-, box-, scribble-wise) supervision and has shown
promising performance, particularly in the image segmentation field. However,
it is still a very challenging problem due to the limited supervision,
especially when only a small number of labeled samples are available.
Additionally, almost all existing WSL segmentation methods are designed for
star-convex structures which are very different from curvilinear structures
such as vessels and nerves. In this paper, we propose a novel sparsely
annotated segmentation framework for curvilinear structures, named YoloCurvSeg,
based on image synthesis. A background generator delivers image backgrounds
that closely match real distributions through inpainting dilated skeletons. The
extracted backgrounds are then combined with randomly emulated curves generated
by a Space Colonization Algorithm-based foreground generator and through a
multilayer patch-wise contrastive learning synthesizer. In this way, a
synthetic dataset with both images and curve segmentation labels is obtained,
at the cost of only one or a few noisy skeleton annotations. Finally, a
segmenter is trained with the generated dataset and possibly an unlabeled
dataset. The proposed YoloCurvSeg is evaluated on four publicly available
datasets (OCTA500, CORN, DRIVE and CHASEDB1) and the results show that
YoloCurvSeg outperforms state-of-the-art WSL segmentation methods by large
margins. With only one noisy skeleton annotation (respectively 0.14%, 0.03%,
1.40%, and 0.65% of the full annotation), YoloCurvSeg achieves more than 97% of
the fully-supervised performance on each dataset. Code and datasets will be
released at https://github.com/llmir/YoloCurvSeg.Comment: 11 pages, 10 figures, submitted to IEEE Transactions on Medical
Imaging (TMI
Generation of annotated multimodal ground truth datasets for abdominal medical image registration
Sparsity of annotated data is a major limitation in medical image processing
tasks such as registration. Registered multimodal image data are essential for
the diagnosis of medical conditions and the success of interventional medical
procedures. To overcome the shortage of data, we present a method that allows
the generation of annotated multimodal 4D datasets. We use a CycleGAN network
architecture to generate multimodal synthetic data from the 4D extended
cardiac-torso (XCAT) phantom and real patient data. Organ masks are provided by
the XCAT phantom, therefore the generated dataset can serve as ground truth for
image segmentation and registration. Realistic simulation of respiration and
heartbeat is possible within the XCAT framework. To underline the usability as
a registration ground truth, a proof of principle registration is performed.
Compared to real patient data, the synthetic data showed good agreement
regarding the image voxel intensity distribution and the noise characteristics.
The generated T1-weighted magnetic resonance imaging (MRI), computed tomography
(CT), and cone beam CT (CBCT) images are inherently co-registered. Thus, the
synthetic dataset allowed us to optimize registration parameters of a
multimodal non-rigid registration, utilizing liver organ masks for evaluation.
Our proposed framework provides not only annotated but also multimodal
synthetic data which can serve as a ground truth for various tasks in medical
imaging processing. We demonstrated the applicability of synthetic data for the
development of multimodal medical image registration algorithms.Comment: 12 pages, 5 figures. This work has been published in the
International Journal of Computer Assisted Radiology and Surgery volum
- …