Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-Based CT Image Augmentation for Object Detection

Accurate Computer-Assisted Diagnosis, relying on large-scale annotated pathological images, can alleviate the risk of overlooking the diagnosis. Unfortunately, in medical imaging, most available datasets are small/fragmented. To tackle this, as a Data Augmentation (DA) method, 3D conditional Generative Adversarial Networks (GANs) can synthesize desired realistic/diverse 3D images as additional training data. However, no 3D conditional GAN-based DA approach exists for general bounding box-based 3D object detection, while it can locate disease areas with physicians' minimum annotation cost, unlike rigorous 3D segmentation. Moreover, since lesions vary in position/size/attenuation, further GAN-based DA performance requires multiple conditions. Therefore, we propose 3D Multi-Conditional GAN (MCGAN) to generate realistic/diverse 32 X 32 X 32 nodules placed naturally on lung Computed Tomography images to boost sensitivity in 3D object detection. Our MCGAN adopts two discriminators for conditioning: the context discriminator learns to classify real vs synthetic nodule/surrounding pairs with noise box-centered surroundings; the nodule discriminator attempts to classify real vs synthetic nodules with size/attenuation conditions. The results show that 3D Convolutional Neural Network-based detection can achieve higher sensitivity under any nodule size/attenuation at fixed False Positive rates and overcome the medical data paucity with the MCGAN-generated realistic nodules---even expert physicians fail to distinguish them from the real ones in Visual Turing Test.Comment: 9 pages, 6 figures, accepted to 3DV 201

Conditional Generation of Medical Images via Disentangled Adversarial Inference

Synthetic medical image generation has a huge potential for improving healthcare through many applications, from data augmentation for training machine learning systems to preserving patient privacy. Conditional Adversarial Generative Networks (cGANs) use a conditioning factor to generate images and have shown great success in recent years. Intuitively, the information in an image can be divided into two parts: 1) content which is presented through the conditioning vector and 2) style which is the undiscovered information missing from the conditioning vector. Current practices in using cGANs for medical image generation, only use a single variable for image generation (i.e., content) and therefore, do not provide much flexibility nor control over the generated image. In this work we propose a methodology to learn from the image itself, disentangled representations of style and content, and use this information to impose control over the generation process. In this framework, style is learned in a fully unsupervised manner, while content is learned through both supervised learning (using the conditioning vector) and unsupervised learning (with the inference mechanism). We undergo two novel regularization steps to ensure content-style disentanglement. First, we minimize the shared information between content and style by introducing a novel application of the gradient reverse layer (GRL); second, we introduce a self-supervised regularization method to further separate information in the content and style variables. We show that in general, two latent variable models achieve better performance and give more control over the generated image. We also show that our proposed model (DRAI) achieves the best disentanglement score and has the best overall performance.Comment: Published in Medical Image Analysi

Unsupervised and Weakly-Supervised Learning of Localized Texture Patterns of Lung Diseases on Computed Tomography

Computed tomography (CT) imaging enables in vivo assessment of lung parenchyma and several lung diseases. CT scans are key in particular for the diagnosis of 1) chronic obstructive pulmonary disease (COPD), which is the fourth leading cause of death worldwide, and largely overlaps with pulmonary emphysema; and 2) lung cancer, which is the first leading cause of cancer-related death, and manifests in its early stage with the presence of lung nodules. Most lung CT image analysis methods to-date have relied on supervised learning requiring manually annotated local regions of interest (ROIs), which are slow and labor-intensive to obtain. Machine learning models requiring less or no manual annotations are important for a sustainable development of computer-aided diagnosis (CAD) systems. This thesis focused on exploiting CT scans for lung disease characterization via two learning strategies: 1) fully unsupervised learning on a very large amount of unannotated image patches to discover novel lung texture patterns for pulmonary emphysema; and 2) weakly-supervised learning to generate voxel-level localization of lung nodules from CT whole-slice labels. In the first part of this thesis, we proposed an original unsupervised approach to learn emphysema-specific radiological texture patterns. We have designed dedicated spatial and texture features and a two-stage learning strategy incorporating clustering and graph partitioning. Learning was performed on a cohort of 2,922 high-resolution full-lung CT scans, which included a high prevalence of smokers and COPD subjects. Experiments lead to discovering 10 highly-reproducible spatially-informed lung texture patterns and 6 quantitative emphysema subtypes (QES). Our discovered QES were associated independently with distinct risk of symptoms, physiological changes, exacerbations and mortality. Genome-wide association studies identified loci associated with four subtypes. Then we designed a deep-learning approach, using unsupervised domain adaptation with adversarial training, to label the QES on cardiac CT scans, which included approximately 70% of the lung. Our proposed method accounted for the differences in CT image qualities, and enabled us to study the progression of QES on a cohort of 17,039 longitudinal cardiac and full-lung CT scans. Overall, the discovered QES provide novel emphysema sub-phenotyping that may facilitate future study of emphysema development, understanding the stages of COPD and the design of personalized therapies. In the second part of the thesis, we have designed a deep-learning method for lung nodule detection with weak labels, using classification convolutional neural networks (CNNs) with skip-connections to generate high-quality discriminative class activation maps, and a novel candidate screening framework to reduce the number of false positives. Given that the vast majority of annotated nodules are benign, we further exploited a data augmentation framework with a generative adversarial network (GAN) to address the issue of data imbalance for lung cancer prediction. Our weakly-supervised lung nodule detection on 1,000s CT scans achieved competitive performance compared to a fully-supervised method, while requiring 100 times less annotations. Our data augmentation framework enabled synthesizing nodules with high fidelity in specified categories, and is beneficial for predicting nodule malignancy scores and hence improving the accuracy / reliability of lung cancer screening