MEDICAL MACHINE INTELLIGENCE: DATA-EFFICIENCY AND KNOWLEDGE-AWARENESS

Traditional clinician diagnosis requires massive manual labor from experienced doctors, which is time-consuming and costly. Computer-aided systems are therefore proposed to reduce doctors’ efforts by using machines to automatically make diagnosis and treatment recommendations. The recent success in deep learning has largely advanced the field of computer-aided diagnosis by offering an avenue to deliver automated medical image analysis. Despite such progress, there remain several challenges towards medical machine intelligence, such as unsatisfactory performance regarding challenging small targets, insufficient training data, high annotation cost, the lack of domain-specific knowledge, etc. These challenges cultivate the need for developing data-efficient and knowledge-aware deep learning techniques which can generalize to different medical tasks without requiring intensive manual labeling efforts, and incorporate domain-specific knowledge in the learning process. In this thesis, we rethink the current progress of deep learning in medical image analysis, with a focus on the aforementioned challenges, and present different data-efficient and knowledge-aware deep learning approaches to address them accordingly. Firstly, we introduce coarse-to-fine mechanisms which use the prediction from the first (coarse) stage to shrink the input region for the second (fine) stage, to enhance the model performance especially for segmenting small challenging structures, such as the pancreas which occupies only a very small fraction (e.g., < 0.5%) of the entire CT volume. The method achieved the state-of-the-art result on the NIH pancreas segmentation dataset. Further extensions also demonstrated effectiveness for segmenting neoplasms such as pancreatic cysts or multiple organs. Secondly, we present a semi-supervised learning framework for medical image segmentation by leveraging both limited labeled data and abundant unlabeled data. Our learning method encourages the segmentation output to be consistent for the same input under different viewing conditions. More importantly, the outputs from different viewing directions are fused altogether to improve the quality of the target, which further enhances the overall performance. The comparison with fully-supervised methods on multi-organ segmentation confirms the effectiveness of this method. Thirdly, we discuss how to incorporate knowledge priors for multi-organ segmentation. Noticing that the abdominal organ sizes exhibit similar distributions across different cohorts, we propose to explicitly incorporate anatomical priors on abdominal organ sizes, guiding the training process with domain-specific knowledge. The approach achieves 84.97% on the MICCAI 2015 challenge “Multi-Atlas Labeling Beyond the Cranial Vault”, which significantly outperforms previous state-of-the-art even using fewer annotations. Lastly, by rethinking how radiologists interpret medical images, we identify one limitation for existing deep-learning-based works on detecting pancreatic ductal adenocarcinoma is the lack of knowledge integration from multi-phase images. Thereby, we introduce a dual-path network where different paths are connected for multi-phase information exchange, and an additional loss is added for removing view divergence. By effectively incorporating multi-phase information, the presented method shows superior performance than prior arts on this matter

TOWARDS DEEP LEARNING ROBUSTNESS FOR COMPUTER VISION IN THE REAL WORLD

Deep learning has been successful in computer vision in recent years. Deep learning models achieve state-of-the-art results on many popular visual benchmarks with additional benefits compared with previous models. However, many recent studies illustrate that deep learning models are not robust towards imperceptible or perceptible changes. This robustness gap makes applying deep learning models to real-world applications challenging due to safety and reliability concerns. This thesis mainly focuses on the robustness of deep learning models in the real world. In the real world, the attackers usually don't know the details of the deep learning models. Besides, even though there are no attackers, the deep learning models are still challenged by many complex cases such as input corruptions, stylized images, and out-of-distribution data. In the first part of this thesis, we study the adversarial robustness in the real world: (1) we successfully attack several deep learning models for different tasks, and then defend against those attacks; (2) we develop universal perturbations that successfully attack unseen deep learning models without knowing architectures, parameters, and tasks. In the second part of this thesis, we discuss more general types of robustness in the real world. Besides adversarial perturbations, we address the more commonly occurred complex cases in the real world, such as input corruptions, natural adversarial examples, stylized images, and out-of-distribution data. We found two strategies that can effectively improve the robustness: (1) address the short-cut learning issue of the deep neural network so that models can collect all helpful information from the input image; (2) use complementary information from different modalities