End-to-end deep auto-encoder for segmenting a moving object with limited training data

Deep learning-based approaches have been widely used in various applications, including segmentation and classification. However, a large amount of data is required to train such techniques. Indeed, in the surveillance video domain, there are few accessible data due to acquisition and experiment complexity. In this paper, we propose an end-to-end deep auto-encoder system for object segmenting from surveillance videos. Our main purpose is to enhance the process of distinguishing the foreground object when only limited data are available. To this end, we propose two approaches based on transfer learning and multi-depth auto-encoders to avoid over-fitting by combining classical data augmentation and principal component analysis (PCA) techniques to improve the quality of training data. Our approach achieves good results outperforming other popular models, which used the same principle of training with limited data. In addition, a detailed explanation of these techniques and some recommendations are provided. Our methodology constitutes a useful strategy for increasing samples in the deep learning domain and can be applied to improve segmentation accuracy. We believe that our strategy has a considerable interest in various applications such as medical and biological fields, especially in the early stages of experiments where there are few samples

Leveraging Supervoxels for Medical Image Volume Segmentation With Limited Supervision

The majority of existing methods for machine learning-based medical image segmentation are supervised models that require large amounts of fully annotated images. These types of datasets are typically not available in the medical domain and are difficult and expensive to generate. A wide-spread use of machine learning based models for medical image segmentation therefore requires the development of data-efficient algorithms that only require limited supervision. To address these challenges, this thesis presents new machine learning methodology for unsupervised lung tumor segmentation and few-shot learning based organ segmentation. When working in the limited supervision paradigm, exploiting the available information in the data is key. The methodology developed in this thesis leverages automatically generated supervoxels in various ways to exploit the structural information in the images. The work on unsupervised tumor segmentation explores the opportunity of performing clustering on a population-level in order to provide the algorithm with as much information as possible. To facilitate this population-level across-patient clustering, supervoxel representations are exploited to reduce the number of samples, and thereby the computational cost. In the work on few-shot learning-based organ segmentation, supervoxels are used to generate pseudo-labels for self-supervised training. Further, to obtain a model that is robust to the typically large and inhomogeneous background class, a novel anomaly detection-inspired classifier is proposed to ease the modelling of the background. To encourage the resulting segmentation maps to respect edges defined in the input space, a supervoxel-informed feature refinement module is proposed to refine the embedded feature vectors during inference. Finally, to improve trustworthiness, an architecture-agnostic mechanism to estimate model uncertainty in few-shot segmentation is developed. Results demonstrate that supervoxels are versatile tools for leveraging structural information in medical data when training segmentation models with limited supervision

Semi-supervised Learning for Medical Image Segmentation

Medical image segmentation is a fundamental step in many computer aided clinical applications, such as tumour detection and quantification, organ measurement and feature learning, etc. However, manually delineating the target of interest on medical images (2D and 3D) is highly labour intensive and time-consuming, even for clinical experts. To address this problem, this thesis focuses on exploring and developing solutions of interactive and fully automated methods to achieve efficient and accurate medical image segmentation. First of all, an interactive semi-automatic segmentation software is developed for the purpose of efficiently annotating any given medical image in 2D and 3D. By converting the segmentation task into a graph optimisation problem using Conditional Random Field, the software allows interactive image segmentation using scribbles. It can also suggest the best image slice to annotate for segmentation refinement in 3D images. Moreover, an “one size for all” parameter setting is experimentally determined using different image modalities, dimensionalities and resolutions, hence no parameter adjustment is required for different unseen medical images. This software can be used for the segmentation of individual medical images in clinical applications or can be used as an annotation tool to generate training examples for machine learning methods. The software can be downloaded from bit.ly/interactive-seg-tool. The developed interactive image segmentation software is efficient, but annotating a large amount of images (hundreds or thousands) for fully supervised machine learning to achieve automatic segmentation is still time-consuming. Therefore, a semi-supervised image segmentation method is developed to achieve fully automatic segmentation by training on a small number of annotated images. An ensemble learning based method is proposed, which is an encoder-decoder based Deep Convolutional Neural Network (DCNN). It is initially trained using a few annotated training samples. This initially trained model is then duplicated as sub-models and improved iteratively using random subsets of unannotated data with pseudo masks generated from models trained in the previous iteration. The number of sub-models is gradually decreased to one in the final iteration. To the best of our knowledge, this is the first use of ensemble learning and DCNN to achieve semi-supervised learning. By evaluating it on a public skin lesion segmentation dataset, it outperforms both the fully supervised learning method using only annotated data and the state-of-the-art methods using similar pseudo labelling ideas. In the context of medical image segmentation, many targets of interest have common geometric shapes across populations (e.g. brain, bone, kidney, liver, etc.). In this case, deformable image registration (alignment) technique can be applied to annotate an unseen image by deforming an annotated template image. Deep learning methods also advanced the field of image registration, but many existing methods can only successfully align images with small deformations. In this thesis, an encoder-decoder DCNN based image registration method is proposed to deal with large deformations. Specifically, a multi-resolution encoder is applied across different image scales for feature extraction. In the decoder, multi-resolution displacement fields are estimated in each scale and then successively combined to produce the final displacement field for transforming the source image to the target image space. The method outperforms many other methods on a local 2D dataset and a public 3D dataset with large deformations. More importantly, the method is further improved by using segmentation masks to guide the image registration to focus on specified local regions, which improves the performance of both segmentation and registration significantly. Finally, to combine the advantages of both image segmentation and image registration. A unified framework that combines a DCNN based segmentation model and the above developed registration model is developed to achieve semi-supervised learning. Initially, the segmentation model is pre-trained using a small number of annotated images, and the registration model is pre-trained using unsupervised learning of all training images. Subsequently, soft pseudo masks of unannotated images are generated by the registration model and segmentation model. The soft Dice loss function is applied to iteratively improve both models using these pseudo labelled images. It is shown that the proposed framework allows both models to mutually improve each other. This approach produces excellent segmentation results only using a small number of annotated images for training, which is better than the segmentation results produced by each model separately. More importantly, once finished training, the framework is able to perform both image segmentation and image registration in high quality

Deep learning in medical imaging and radiation therapy

Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/146980/1/mp13264_am.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/146980/2/mp13264.pd

Deep learning for fast and robust medical image reconstruction and analysis

Medical imaging is an indispensable component of modern medical research as well as clinical practice. Nevertheless, imaging techniques such as magnetic resonance imaging (MRI) and computational tomography (CT) are costly and are less accessible to the majority of the world. To make medical devices more accessible, affordable and efficient, it is crucial to re-calibrate our current imaging paradigm for smarter imaging. In particular, as medical imaging techniques have highly structured forms in the way they acquire data, they provide us with an opportunity to optimise the imaging techniques holistically by leveraging data. The central theme of this thesis is to explore different opportunities where we can exploit data and deep learning to improve the way we extract information for better, faster and smarter imaging. This thesis explores three distinct problems. The first problem is the time-consuming nature of dynamic MR data acquisition and reconstruction. We propose deep learning methods for accelerated dynamic MR image reconstruction, resulting in up to 10-fold reduction in imaging time. The second problem is the redundancy in our current imaging pipeline. Traditionally, imaging pipeline treated acquisition, reconstruction and analysis as separate steps. However, we argue that one can approach them holistically and optimise the entire pipeline jointly for a specific target goal. To this end, we propose deep learning approaches for obtaining high fidelity cardiac MR segmentation directly from significantly undersampled data, greatly exceeding the undersampling limit for image reconstruction. The final part of this thesis tackles the problem of interpretability of the deep learning algorithms. We propose attention-models that can implicitly focus on salient regions in an image to improve accuracy for ultrasound scan plane detection and CT segmentation. More crucially, these models can provide explainability, which is a crucial stepping stone for the harmonisation of smart imaging and current clinical practice.Open Acces

Development of Deep Learning Methods for Magnetic Resonance Phase Imaging of Neurological Disease

Magnetic resonance imaging (MRI) is a high-resolution, non-invasive medical imaging modality that is widely used in human brain. In recent years, susceptibility weighted imaging (SWI) and quantitative susceptibility mapping (QSM) have been proposed to utilize MR phase signal to generate contrast from tissue magnetic susceptibility and even quantify the property. On the other hand, deep learning, especially deep convolutional neural networks (DCNNs), have achieved state-of-the-art performances in numerous computer vision tasks and gained significant attention in the field of medical imaging in the recent years. This dissertation combined the idea of deep learning with the two MR phase imaging methods. To combined deep learning with SWI, we designed and trained a 3D deep residual network that can distinguish false positive detected candidates from cerebral microbleeds (CMBs) and built an automatic CMB detection pipeline with high performance. We further confirmed the generalizability of this deep learning-based pipeline using multiple dataset with different scan parameters and pathologies and provided lessons for application and generalization of generic deep learning based medical imaging methods.To combine deep learning with QSM, we developed a 3D U-Net based network that learns to perform dipole inversion from gold standard QSM acquired from data with multiple orientation. The model was further improved with adversarial training strategy and achieved significantly lower reconstruction error than traditional QSM algorithms. In addition, we also performed various background removal and dipole inversion algorithms on both brain tumor patients and healthy volunteers to study and compare their performances. The results could provide guidance on future application of QSM in different scenarios

Combining Shape and Learning for Medical Image Analysis

Automatic methods with the ability to make accurate, fast and robust assessments of medical images are highly requested in medical research and clinical care. Excellent automatic algorithms are characterized by speed, allowing for scalability, and an accuracy comparable to an expert radiologist. They should produce morphologically and physiologically plausible results while generalizing well to unseen and rare anatomies. Still, there are few, if any, applications where today\u27s automatic methods succeed to meet these requirements.\ua0The focus of this thesis is two tasks essential for enabling automatic medical image assessment, medical image segmentation and medical image registration. Medical image registration, i.e. aligning two separate medical images, is used as an important sub-routine in many image analysis tools as well as in image fusion, disease progress tracking and population statistics. Medical image segmentation, i.e. delineating anatomically or physiologically meaningful boundaries, is used for both diagnostic and visualization purposes in a wide range of applications, e.g. in computer-aided diagnosis and surgery.The thesis comprises five papers addressing medical image registration and/or segmentation for a diverse set of applications and modalities, i.e. pericardium segmentation in cardiac CTA, brain region parcellation in MRI, multi-organ segmentation in CT, heart ventricle segmentation in cardiac ultrasound and tau PET registration. The five papers propose competitive registration and segmentation methods enabled by machine learning techniques, e.g. random decision forests and convolutional neural networks, as well as by shape modelling, e.g. multi-atlas segmentation and conditional random fields

Role of deep learning techniques in non-invasive diagnosis of human diseases.

Machine learning, a sub-discipline in the domain of artificial intelligence, concentrates on algorithms able to learn and/or adapt their structure (e.g., parameters) based on a set of observed data. The adaptation is performed by optimizing over a cost function. Machine learning obtained a great attention in the biomedical community because it offers a promise for improving sensitivity and/or specificity of detection and diagnosis of diseases. It also can increase objectivity of the decision making, decrease the time and effort on health care professionals during the process of disease detection and diagnosis. The potential impact of machine learning is greater than ever due to the increase in medical data being acquired, the presence of novel modalities being developed and the complexity of medical data. In all of these scenarios, machine learning can come up with new tools for interpreting the complex datasets that confront clinicians. Much of the excitement for the application of machine learning to biomedical research comes from the development of deep learning which is modeled after computation in the brain. Deep learning can help in attaining insights that would be impossible to obtain through manual analysis. Deep learning algorithms and in particular convolutional neural networks are different from traditional machine learning approaches. Deep learning algorithms are known by their ability to learn complex representations to enhance pattern recognition from raw data. On the other hand, traditional machine learning requires human engineering and domain expertise to design feature extractors and structure data. With increasing demands upon current radiologists, there are growing needs for automating the diagnosis. This is a concern that deep learning is able to address. In this dissertation, we present four different successful applications of deep learning for diseases diagnosis. All the work presented in the dissertation utilizes medical images. In the first application, we introduce a deep-learning based computer-aided diagnostic system for the early detection of acute renal transplant rejection. The system is based on the fusion of both imaging markers (apparent diffusion coefficients derived from diffusion-weighted magnetic resonance imaging) and clinical biomarkers (creatinine clearance and serum plasma creatinine). The fused data is then used as an input to train and test a convolutional neural network based classifier. The proposed system is tested on scans collected from 56 subjects from geographically diverse populations and different scanner types/image collection protocols. The overall accuracy of the proposed system is 92.9% with 93.3% sensitivity and 92.3% specificity in distinguishing non-rejected kidney transplants from rejected ones. In the second application, we propose a novel deep learning approach for the automated segmentation and quantification of the LV from cardiac cine MR images. We aimed at achieving lower errors for the estimated heart parameters compared to the previous studies by proposing a novel deep learning segmentation method. Using fully convolutional neural networks, we proposed novel methods for the extraction of a region of interest that contains the left ventricle, and the segmentation of the left ventricle. Following myocardial segmentation, functional and mass parameters of the left ventricle are estimated. Automated Cardiac Diagnosis Challenge dataset was used to validate our framework, which gave better segmentation, accurate estimation of cardiac parameters, and produced less error compared to other methods applied on the same dataset. Furthermore, we showed that our segmentation approach generalizes well across different datasets by testing its performance on a locally acquired dataset. In the third application, we propose a novel deep learning approach for automated quantification of strain from cardiac cine MR images of mice. For strain analysis, we developed a Laplace-based approach to track the LV wall points by solving the Laplace equation between the LV contours of each two successive image frames over the cardiac cycle. Following tracking, the strain estimation is performed using the Lagrangian-based approach. This new automated system for strain analysis was validated by comparing the outcome of these analysis with the tagged MR images from the same mice. There were no significant differences between the strain data obtained from our algorithm using cine compared to tagged MR imaging. In the fourth application, we demonstrate how a deep learning approach can be utilized for the automated classification of kidney histopathological images. Our approach can classify four classes: the fat, the parenchyma, the clear cell renal cell carcinoma, and the unusual cancer which has been discovered recently, called clear cell papillary renal cell carcinoma. Our framework consists of three convolutional neural networks and the whole-slide kidney images were divided into patches with three different sizes to be inputted to the networks. Our approach can provide patch-wise and pixel-wise classification. Our approach classified the four classes accurately and surpassed other state-of-the-art methods such as ResNet (pixel accuracy: 0.89 Resnet18, 0.93 proposed). In conclusion, the results of our proposed systems demonstrate the potential of deep learning for the efficient, reproducible, fast, and affordable disease diagnosis

Semi-supervised Learning for Medical Image Segmentation

Medical image segmentation is a fundamental step in many computer aided clinical applications, such as tumour detection and quantification, organ measurement and feature learning, etc. However, manually delineating the target of interest on medical images (2D and 3D) is highly labour intensive and time-consuming, even for clinical experts. To address this problem, this thesis focuses on exploring and developing solutions of interactive and fully automated methods to achieve efficient and accurate medical image segmentation. First of all, an interactive semi-automatic segmentation software is developed for the purpose of efficiently annotating any given medical image in 2D and 3D. By converting the segmentation task into a graph optimisation problem using Conditional Random Field, the software allows interactive image segmentation using scribbles. It can also suggest the best image slice to annotate for segmentation refinement in 3D images. Moreover, an “one size for all” parameter setting is experimentally determined using different image modalities, dimensionalities and resolutions, hence no parameter adjustment is required for different unseen medical images. This software can be used for the segmentation of individual medical images in clinical applications or can be used as an annotation tool to generate training examples for machine learning methods. The software can be downloaded from bit.ly/interactive-seg-tool. The developed interactive image segmentation software is efficient, but annotating a large amount of images (hundreds or thousands) for fully supervised machine learning to achieve automatic segmentation is still time-consuming. Therefore, a semi-supervised image segmentation method is developed to achieve fully automatic segmentation by training on a small number of annotated images. An ensemble learning based method is proposed, which is an encoder-decoder based Deep Convolutional Neural Network (DCNN). It is initially trained using a few annotated training samples. This initially trained model is then duplicated as sub-models and improved iteratively using random subsets of unannotated data with pseudo masks generated from models trained in the previous iteration. The number of sub-models is gradually decreased to one in the final iteration. To the best of our knowledge, this is the first use of ensemble learning and DCNN to achieve semi-supervised learning. By evaluating it on a public skin lesion segmentation dataset, it outperforms both the fully supervised learning method using only annotated data and the state-of-the-art methods using similar pseudo labelling ideas. In the context of medical image segmentation, many targets of interest have common geometric shapes across populations (e.g. brain, bone, kidney, liver, etc.). In this case, deformable image registration (alignment) technique can be applied to annotate an unseen image by deforming an annotated template image. Deep learning methods also advanced the field of image registration, but many existing methods can only successfully align images with small deformations. In this thesis, an encoder-decoder DCNN based image registration method is proposed to deal with large deformations. Specifically, a multi-resolution encoder is applied across different image scales for feature extraction. In the decoder, multi-resolution displacement fields are estimated in each scale and then successively combined to produce the final displacement field for transforming the source image to the target image space. The method outperforms many other methods on a local 2D dataset and a public 3D dataset with large deformations. More importantly, the method is further improved by using segmentation masks to guide the image registration to focus on specified local regions, which improves the performance of both segmentation and registration significantly. Finally, to combine the advantages of both image segmentation and image registration. A unified framework that combines a DCNN based segmentation model and the above developed registration model is developed to achieve semi-supervised learning. Initially, the segmentation model is pre-trained using a small number of annotated images, and the registration model is pre-trained using unsupervised learning of all training images. Subsequently, soft pseudo masks of unannotated images are generated by the registration model and segmentation model. The soft Dice loss function is applied to iteratively improve both models using these pseudo labelled images. It is shown that the proposed framework allows both models to mutually improve each other. This approach produces excellent segmentation results only using a small number of annotated images for training, which is better than the segmentation results produced by each model separately. More importantly, once finished training, the framework is able to perform both image segmentation and image registration in high quality