Localisation in 3D Images Using Cross-features Correlation Learning

Object detection and segmentation have evolved drastically over the past two decades thanks to the continuous advancement in the field of deep learning. Substantial research efforts have been dedicated towards integrating object detection techniques into a wide range of real-world prob-lems. Most existing methods take advantage of the successful application and representational ability of convolutional neural networks (CNNs). Generally, these methods target mainstream applications that are typically based on 2D imaging scenarios. Additionally, driven by the strong correlation between the quality of the feature embedding and the performance in CNNs, most works focus on design characteristics of CNNs, e.g., depth and width, to enhance their modelling capacity and discriminative ability. Limited research was directed towards exploiting feature-level dependencies, which can be feasibly used to enhance the performance of CNNs. More-over, directly adopting such approaches into more complex imaging domains that target data of higher dimensions (e.g., 3D multi-modal and volumetric images) is not straightforwardly appli-cable due to the different nature and complexity of the problem. In this thesis, we explore the possibility of incorporating feature-level correspondence and correlations into object detection and segmentation contexts that target the localisation of 3D objects from 3D multi-modal and volumetric image data. Accordingly, we first explore the detection problem of 3D solar active regions in multi-spectral solar imagery where different imaging bands correspond to different 2D layers (altitudes) in the 3D solar atmosphere.We propose a joint analysis approach in which information from different imaging bands is first individually analysed using band-specific network branches to extract inter-band features that are then dynamically cross-integrated and jointly analysed to investigate spatial correspon-dence and co-dependencies between the different bands. The aggregated embeddings are further analysed using band-specific detection network branches to predict separate sets of results (one for each band). Throughout our study, we evaluate different types of feature fusion, using convo-lutional embeddings of different semantic levels, as well as the impact of using different numbers of image bands inputs to perform the joint analysis. We test the proposed approach over different multi-modal datasets (multi-modal solar images and brain MRI) and applications. The proposed joint analysis based framework consistently improves the CNN’s performance when detecting target regions in contrast to single band based baseline methods.We then generalise our cross-band joint analysis detection scheme into the 3D segmentation problem using multi-modal images. We adopt the joint analysis principles into a segmentation framework where cross-band information is dynamically analysed and cross-integrated at vari-ous semantic levels. The proposed segmentation network also takes advantage of band-specific skip connections to maximise the inter-band information and assist the network in capturing fine details using embeddings of different spatial scales. Furthermore, a recursive training strat-egy, based on weak labels (e.g., bounding boxes), is proposed to overcome the difficulty of producing dense labels to train the segmentation network. We evaluate the proposed segmen-tation approach using different feature fusion approaches, over different datasets (multi-modal solar images, brain MRI, and cloud satellite imagery), and using different levels of supervisions. Promising results were achieved and demonstrate an improved performance in contrast to single band based analysis and state-of-the-art segmentation methods.Additionally, we investigate the possibility of explicitly modelling objective driven feature-level correlations, in a localised manner, within 3D medical imaging scenarios (3D CT pul-monary imaging) to enhance the effectiveness of the feature extraction process in CNNs and subsequently the detection performance. Particularly, we present a framework to perform the 3D detection of pulmonary nodules as an ensemble of two stages, candidate proposal and a false positive reduction. We propose a 3D channel attention block in which cross-channel informa-tion is incorporated to infer channel-wise feature importance with respect to the target objective. Unlike common attention approaches that rely on heavy dimensionality reduction and computa-tionally expensive multi-layer perceptron networks, the proposed approach utilises fully convo-lutional networks to allow directly exploiting rich 3D descriptors and performing the attention in an efficient manner. We also propose a fully convolutional 3D spatial attention approach that elevates cross-sectional information to infer spatial attention. We demonstrate the effectiveness of the proposed attention approaches against a number of popular channel and spatial attention mechanisms. Furthermore, for the False positive reduction stage, in addition to attention, we adopt a joint analysis based approach that takes into account the variable nodule morphology by aggregating spatial information from different contextual levels. We also propose a Zoom-in convolutional path that incorporates semantic information of different spatial scales to assist the network in capturing fine details. The proposed detection approach demonstrates considerable gains in performance in contrast to state-of-the-art lung nodule detection methods.We further explore the possibility of incorporating long-range dependencies between arbi-trary positions in the input features using Transformer networks to infer self-attention, in the context of 3D pulmonary nodule detection, in contrast to localised (convolutional based) atten-tion . We present a hybrid 3D detection approach that takes advantage of both, the Transformers ability in modelling global context and correlations and the spatial representational characteris-tics of convolutional neural networks, providing complementary information and subsequently improving the discriminative ability of the detection model. We propose two hybrid Transformer CNN variants where we investigate the impact of exploiting a deeper Transformer design –in which more Transformer layers and trainable parameters are incorporated– is used along with high-level convolutional feature inputs of a single spatial resolution, in contrast to a shallower Transformer design –of less Transformer layers and trainable parameters– while exploiting con-volutional embeddings of different semantic levels and relatively higher resolution.Extensive quantitative and qualitative analyses are presented for the proposed methods in this thesis and demonstrate the feasibility of exploiting feature-level relations, either implicitly or explicitly, in different detection and segmentation problems

Cancer diagnosis using deep learning: A bibliographic review

In this paper, we first describe the basics of the field of cancer diagnosis, which includes steps of cancer diagnosis followed by the typical classification methods used by doctors, providing a historical idea of cancer classification techniques to the readers. These methods include Asymmetry, Border, Color and Diameter (ABCD) method, seven-point detection method, Menzies method, and pattern analysis. They are used regularly by doctors for cancer diagnosis, although they are not considered very efficient for obtaining better performance. Moreover, considering all types of audience, the basic evaluation criteria are also discussed. The criteria include the receiver operating characteristic curve (ROC curve), Area under the ROC curve (AUC), F1 score, accuracy, specificity, sensitivity, precision, dice-coefficient, average accuracy, and Jaccard index. Previously used methods are considered inefficient, asking for better and smarter methods for cancer diagnosis. Artificial intelligence and cancer diagnosis are gaining attention as a way to define better diagnostic tools. In particular, deep neural networks can be successfully used for intelligent image analysis. The basic framework of how this machine learning works on medical imaging is provided in this study, i.e., pre-processing, image segmentation and post-processing. The second part of this manuscript describes the different deep learning techniques, such as convolutional neural networks (CNNs), generative adversarial models (GANs), deep autoencoders (DANs), restricted Boltzmann’s machine (RBM), stacked autoencoders (SAE), convolutional autoencoders (CAE), recurrent neural networks (RNNs), long short-term memory (LTSM), multi-scale convolutional neural network (M-CNN), multi-instance learning convolutional neural network (MIL-CNN). For each technique, we provide Python codes, to allow interested readers to experiment with the cited algorithms on their own diagnostic problems. The third part of this manuscript compiles the successfully applied deep learning models for different types of cancers. Considering the length of the manuscript, we restrict ourselves to the discussion of breast cancer, lung cancer, brain cancer, and skin cancer. The purpose of this bibliographic review is to provide researchers opting to work in implementing deep learning and artificial neural networks for cancer diagnosis a knowledge from scratch of the state-of-the-art achievements

Aprendendo a suprimir não-máximos para aperfeiçoar a detecção de nódulos pulmonares em imagens de CT

Orientador: Alexandre Xavier FalcãoDissertação (mestrado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: O câncer de pulmão é o tipo mais comum de câncer em homens e o terceiro mais comum em mulheres. Devido ao mau prognóstico, o câncer de pulmão é responsável pela maior taxa de mortalidade, atingindo 1,8 milhão de mortes por ano. O diagnóstico e o tratamento nos estágios iniciais podem aumentar as chances de sobrevivência. A tomografia computadorizada (TC) é a modalidade de imagem preferida para detectar e diagnosticar câncer de pulmão, pois fornece imagens 3D do tórax em alta resolução, facilitando a detecção de pequenos nódulos. No entanto, a natureza 3D das imagens dificulta sua análise visual. Como conseqüência, o número de falsos positivos ainda é alto e, mesmo contando com a opinião de vários especialistas, o diagnóstico é frequentemente sujeito a alguma falta de consenso. Os sistemas de Diagnóstico Assistida por Computador (CAD) foram desenvolvidos para solucionar o problema, auxiliando especialistas na tarefa de detecção e classificação mais rápidas e precisas de anormalidades. As técnicas usadas nos sistemas CAD podem ser divididas em dois grupos: sistemas CAD que exploram features de imagem baseados em conhecimento e sistemas CAD que aprendem os features de imagens anotadas, principalmente baseadas em aprendizado profundo por meio de redes neurais convolucionais (CNNs). Na última década, muitos métodos computacionais (sistemas CAD) foram desenvolvidos para auxiliar os médicos na detecção de nódulos pulmonares. Tais métodos são baseados principalmente em CNNs, que alcançaram resultados promissores na detecção precoce de nódulos pulmonares. No entanto, esses métodos geram várias regiões candidatas por nódulo, de modo que um algoritmo de não-máxima supressão (NMS) é necessário para selecionar uma única região por nódulo, eliminando as redundantes. O GossipNet é uma rede neural 1D para NMS, que pode aprender os parâmetros do NMS em vez de confiar nos parâmetros artesanais. No entanto, o GossipNet não tira proveito dos features de imagem para aprender NMS. Neste trabalho, propomos um sistema CAD automatizado para detecção de nódulos pulmonares, que consiste em quatro módulos: pré-processamento, a definição de uma região de interesse (por exemplo, por segmentação pulmonar), detecção de nódulos e a eliminação de candidatos redundantes. Para a segmentação pulmonar, usamos uma abordagem recente baseada em sequências de transformações florestais de imagem (IFTs) denominada ALTIS, fornecendo uma segmentação mais precisa dos pulmões em comparação com o método usado no desafio LUNA16. Para a detecção de nódulos e a eliminação de candidatos redundantes, usamos o 3D Faster R-CNN com ResNet18 para a detecção de regiões candidatas com nódulos e apresentamos FeatureNMS --- uma rede neural que fornece features de imagem adicionais à entrada do GossipNet, que resultam de uma transformação sobre as intensidades de voxel de cada região candidata na imagem da TC. Para validação, usamos o conjunto de dados de desafio LUNA16Abstract: Lung cancer is the most common type of cancer in men and the third most common one in women. Due to poor prognosis, lung cancer is responsible for the largest mortality rate, reaching 1.8 million deaths per year. Diagnosis and treatment at the early stages can increase the chances of survival. Computerized Tomography (CT) is the imaging modality of preference to detect and diagnose lung cancer since it provides high-resolution 3D images of the thorax, facilitating the detection of small nodules. However, the 3D nature of the images makes their visual analysis difficult. As a consequence, the number of false positives is still high and, even by counting on the opinion of multiple specialists, the diagnosis is often subjected to some lack of consensus. Computer-Aided Detection (CAD) systems have been developed to address the problem, assisting to specialists in the task of quicker and more accurate detection and classification of abnormalities. The techniques used in CAD systems may be divided into two groups: CAD systems that explore knowledge-based image features and CAD systems that learn the features from annotated images, mostly based on deep learning through Convolutional Neural Networks (CNNs). In the last decade, many computational methods (CAD systems) have been developed to assist physicians in lung nodule detection. Such methods are mostly based on CNNs, which have achieved promising results in early detection of lung nodules. However, these methods generate several candidate regions per nodule, such that a Non-Maximum Suppression (NMS) algorithm is required to select a single region per nodule while eliminating the redundant ones. GossipNet is a 1D Neural Network (NN) for NMS, which can learn the NMS parameters rather than relying on handcrafted ones. However, GossipNet does not take advantage of image features to learn NMS. In this work, we propose an automated CAD system for lung nodule detection which consists of four modules: pre-processing, the definition of a region of interest (e.g., by lung segmentation), nodule detection, and the elimination of redundant candidates. For lung segmentation, we use a recent approach based on sequences of Image Foresting Transforms (IFTs) named ALTIS providing a more accurate segmentation of the lungs compared to the method used in the LUNA16 challenge. For nodule detection and the elimination of redundant candidates, we use 3D Faster R-CNN with ResNet18 for the detection of candidate regions with nodules and present \emph{FeatureNMS} --- a neural network that provides additional image features to the input of GossipNet, which result from a transformation over the voxel intensities of each candidate region in the CT image. For validation, we use the LUNA16 challenge datasetMestradoCiência da ComputaçãoMestre em Ciência da Computação171063/2017-12014/12236-1CNPQFAPES

Going Deep in Medical Image Analysis: Concepts, Methods, Challenges and Future Directions

Medical Image Analysis is currently experiencing a paradigm shift due to Deep Learning. This technology has recently attracted so much interest of the Medical Imaging community that it led to a specialized conference in `Medical Imaging with Deep Learning' in the year 2018. This article surveys the recent developments in this direction, and provides a critical review of the related major aspects. We organize the reviewed literature according to the underlying Pattern Recognition tasks, and further sub-categorize it following a taxonomy based on human anatomy. This article does not assume prior knowledge of Deep Learning and makes a significant contribution in explaining the core Deep Learning concepts to the non-experts in the Medical community. Unique to this study is the Computer Vision/Machine Learning perspective taken on the advances of Deep Learning in Medical Imaging. This enables us to single out `lack of appropriately annotated large-scale datasets' as the core challenge (among other challenges) in this research direction. We draw on the insights from the sister research fields of Computer Vision, Pattern Recognition and Machine Learning etc.; where the techniques of dealing with such challenges have already matured, to provide promising directions for the Medical Imaging community to fully harness Deep Learning in the future

A Review on Deep Learning in Medical Image Reconstruction

Medical imaging is crucial in modern clinics to guide the diagnosis and treatment of diseases. Medical image reconstruction is one of the most fundamental and important components of medical imaging, whose major objective is to acquire high-quality medical images for clinical usage at the minimal cost and risk to the patients. Mathematical models in medical image reconstruction or, more generally, image restoration in computer vision, have been playing a prominent role. Earlier mathematical models are mostly designed by human knowledge or hypothesis on the image to be reconstructed, and we shall call these models handcrafted models. Later, handcrafted plus data-driven modeling started to emerge which still mostly relies on human designs, while part of the model is learned from the observed data. More recently, as more data and computation resources are made available, deep learning based models (or deep models) pushed the data-driven modeling to the extreme where the models are mostly based on learning with minimal human designs. Both handcrafted and data-driven modeling have their own advantages and disadvantages. One of the major research trends in medical imaging is to combine handcrafted modeling with deep modeling so that we can enjoy benefits from both approaches. The major part of this article is to provide a conceptual review of some recent works on deep modeling from the unrolling dynamics viewpoint. This viewpoint stimulates new designs of neural network architectures with inspirations from optimization algorithms and numerical differential equations. Given the popularity of deep modeling, there are still vast remaining challenges in the field, as well as opportunities which we shall discuss at the end of this article.Comment: 31 pages, 6 figures. Survey pape

A Novel Hybrid CNN Denoising Technique (HDCNN) for Image Denoising with Improved Performance

Photo denoising has been tackled by deep convolutional neural networks (CNNs) with powerful learning capabilities. Unfortunately, some CNNs perform badly on complex displays because they only train one deep network for their image blurring models. We recommend a hybrid CNN denoising technique (HDCNN) to address this problem. An HDCNN consists of a dilated interfere with, a RepVGG block, an attribute sharpening interferes with, as well as one inversion. To gather more context data, DB incorporates a stretched convolution, data sequential normalization (BN), shared convergence, and the activating function called the ReLU. Convolution, BN, and reLU are combined in parallel by RVB to obtain complimentary width characteristics. The RVB's refining characteristics are used to refine FB, which is then utilized to collect more precise data. To create a crisp image, a single convolution works in conjunction with a residual learning process. These crucial elements enable the HDCNN to carry out visual denoising efficiently. The suggested HDCNN has a good denoising performance in open data sets, according to experiments

Artificial Intelligence in Image-Based Screening, Diagnostics, and Clinical Care of Cardiopulmonary Diseases

Cardiothoracic and pulmonary diseases are a significant cause of mortality and morbidity worldwide. The COVID-19 pandemic has highlighted the lack of access to clinical care, the overburdened medical system, and the potential of artificial intelligence (AI) in improving medicine. There are a variety of diseases affecting the cardiopulmonary system including lung cancers, heart disease, tuberculosis (TB), etc., in addition to COVID-19-related diseases. Screening, diagnosis, and management of cardiopulmonary diseases has become difficult owing to the limited availability of diagnostic tools and experts, particularly in resource-limited regions. Early screening, accurate diagnosis and staging of these diseases could play a crucial role in treatment and care, and potentially aid in reducing mortality. Radiographic imaging methods such as computed tomography (CT), chest X-rays (CXRs), and echo ultrasound (US) are widely used in screening and diagnosis. Research on using image-based AI and machine learning (ML) methods can help in rapid assessment, serve as surrogates for expert assessment, and reduce variability in human performance. In this Special Issue, “Artificial Intelligence in Image-Based Screening, Diagnostics, and Clinical Care of Cardiopulmonary Diseases”, we have highlighted exemplary primary research studies and literature reviews focusing on novel AI/ML methods and their application in image-based screening, diagnosis, and clinical management of cardiopulmonary diseases. We hope that these articles will help establish the advancements in AI

Tuberculosis diagnosis from pulmonary chest x-ray using deep learning.

Doctoral Degree. University of KwaZulu-Natal, Durban.Tuberculosis (TB) remains a life-threatening disease, and it is one of the leading causes of mortality in developing countries. This is due to poverty and inadequate medical resources. While treatment for TB is possible, it requires an accurate diagnosis first. Several screening tools are available, and the most reliable is Chest X-Ray (CXR), but the radiological expertise for accurately interpreting the CXR images is often lacking. Over the years, CXR has been manually examined; this process results in delayed diagnosis, is time-consuming, expensive, and is prone to misdiagnosis, which could further spread the disease among individuals. Consequently, an algorithm could increase diagnosis efficiency, improve performance, reduce the cost of manual screening and ultimately result in early/timely diagnosis. Several algorithms have been implemented to diagnose TB automatically. However, these algorithms are characterized by low accuracy and sensitivity leading to misdiagnosis. In recent years, Convolutional Neural Networks (CNN), a class of Deep Learning, has demonstrated tremendous success in object detection and image classification task. Hence, this thesis proposed an efficient Computer-Aided Diagnosis (CAD) system with high accuracy and sensitivity for TB detection and classification. The proposed model is based firstly on novel end-to-end CNN architecture, then a pre-trained Deep CNN model that is fine-tuned and employed as a features extractor from CXR. Finally, Ensemble Learning was explored to develop an Ensemble model for TB classification. The Ensemble model achieved a new stateof- the-art diagnosis accuracy of 97.44% with a 99.18% sensitivity, 96.21% specificity and 0.96% AUC. These results are comparable with state-of-the-art techniques and outperform existing TB classification models.Author's Publications listed on page iii

Lung infection segmentation for COVID-19 pneumonia based on a cascade convolutional network from CT images

The COVID-19 pandemic is a global, national, and local public health which causing a significant outbreak in all countries and regions for both males and females around the world. Automated detection of lung infections and their boundaries from medical images offers a great potential to augment the patient treatment healthcare strategies for tackling COVID-19 and its impacts. Detecting this disease from lung CT scan images is perhaps one of the fastest ways to diagnose the patients. However, finding the presence of infected tissues and segment them from CT slices faces numerous challenges, including similar adjacent tissues, vague boundary, and erratic infections. To overcome the mentioned problems, we propose a two-route convolutional neural network (CNN) by extracting global and local features for detecting and classifying COVID-19 infection from CT images. Each pixel from the image is classified into normal and infected tissue. For improving the classification accuracy, we used two different strategies including Fuzzy c-mean clustering and local directional pattern (LDN) encoding methods to represent the input image differently. This allows us to find a more complex pattern from the image. To overcome the overfitting problems due to small samples, an augmentation approach is utilized. The results demonstrated that the proposed framework achieved Precision 96%, Recall 97%, F-score, average surface distance (ASD) of 2.8\pm0.3\ mm and volume overlap error (VOE) of 5.6\pm1.2%