Attention Mechanisms in the Classification of Histological Images

Recently, there has been an increase in the number of medical exams prescribed by medical doctors, not only to diagnose but also to keep track of the evolution of pathologies. In this sense, one of the medical specialties where the mentioned increase in the prescription rate has been observed is oncology. In this regard, not only to efficiently diagnose but also to monitor the evolution of the mentioned diseases, CT (Computed Tomography) scans, MRIs (Magnetic Resonance Imaging), and Biopsies are imaging techniques commonly used. After the exams are performed and the results retrieved by the respective health professionals, their analysis and interpretation are mandatory. This process, carried out by medical experts, is usually a time-consuming and tiring task. In this sense and to reduce the workload of these experts and support decision making, the research community start proposing several computer-aided systems, whose primary goal is to efficiently distinguish between healthy images and tumoral ones. Despite the success achieved by these methodologies, it become evident that the distinction of the two mentioned image categories (healthy and not-healthy) was associated with small regions of the images, and therefore not all image regions were equally important for diagnostic purposes. In this line of thinking, attention mechanisms start being considered to highlight important regions and neglect unimportant ones, leading to more correct predictions. In this thesis, we aim to study the impact of such mechanisms in the extraction of features from histopathological images of the epithelium from the oral cavity. In order to access the quality of the generated features for diagnostic purposes, those features were used to distinguish healthy from cancerous histopathological images.Recentemente, tem-se observado uma tendência crescente no número de exames médicos prescritos por médicos, no sentido de diagnosticar e acompanhar a evolução de patologias. Deste modo, uma das especialidades médicas onde a referida taxa de prescrição se assinala bastante elevada é a oncologia. No sentido de não só diagnosticar com eficácia, mas também para que a evolução das patologias seja devidamente seguida, é comum recorrer-se a técnicas de imagiologia como TACs (Tomografia Axial Computorizadas), RMs (Ressonâncias Magnéticas) ou Biópsias. Após a recepção dos respectivos exames médicos é necessário a sua análise e interpretação pelos profissionais competentes. Este processo é frequentemente moroso e cansativo para estes profissionais. No sentido de reduzir o labor destes profissionais e apoiar a tomada de decisão, começaram a surgir na literatura diversos sistemas computacionais cujo objectivo é distinguir imagens saudáveis de imagens não-saudáveis. Apesar do sucesso alcançado por estes sistemas, rapidamente se verificou que a distinção das duas classes de imagens é dependente de pequenas regiões, neste sentido nem todas as regiões constituintes da imagem são igualmente importantes para a distinção acima indicada. Posto isto, foram considerados mecanismos de atenção no sentido de maior importância dar a porções relevantes da imagem e negligenciar menos importantes, conduzindo a previsões mais correctas. Nesta dissertação pretende-se fazer um estudo do impacto destes mecanismos na extracção de features de imagens histopatológicas da mucosa oral. No sentido de avaliar a qualidade das features extraídas para o diagnóstico, estas são usadas por classificadores para a distinção de imagens saudáveis e cancerígenas

Classification of histological images of thyroid nodules based on a combination of Deep Features and Machine Learning

Background: Thyroid nodules are a prevalent worldwide disease with complex pathological types. They can be classified as either benign or malignant. This paper presents a tool for automatically classifying histological images of thyroid nodules, with a focus on papillary carcinoma and follicular adenoma. Methods: In this work, two pre-trained Convolutional Neural Network (CNN) architectures, VGG16 and VGG19, are used to extract deep features. Then, a principal component analysis was used to reduce the dimensionality of the vectors. Then, three machine learning algorithms (Support Vector Machine, K-Nearest Neighbor, and Random Forest) were used for classification. These investigations were applied to our database collection, Results: The proposed investigations have been applied to our private database collection with a total of 112 histological images. The highest results were obtained by the VGG16 transfer deep feature and the SVM classifier with an accuracy rate equal to 100%

Geometry-Aware Latent Representation Learning for Modeling Disease Progression of Barrett's Esophagus

Barrett's Esophagus (BE) is the only precursor known to Esophageal Adenocarcinoma (EAC), a type of esophageal cancer with poor prognosis upon diagnosis. Therefore, diagnosing BE is crucial in preventing and treating esophageal cancer. While supervised machine learning supports BE diagnosis, high interobserver variability in histopathological training data limits these methods. Unsupervised representation learning via Variational Autoencoders (VAEs) shows promise, as they map input data to a lower-dimensional manifold with only useful features, characterizing BE progression for improved downstream tasks and insights. However, the VAE's Euclidean latent space distorts point relationships, hindering disease progression modeling. Geometric VAEs provide additional geometric structure to the latent space, with RHVAE assuming a Riemannian manifold and $\mathcal{S}$-VAE a hyperspherical manifold. Our study shows that $\mathcal{S}$-VAE outperforms vanilla VAE with better reconstruction losses, representation classification accuracies, and higher-quality generated images and interpolations in lower-dimensional settings. By disentangling rotation information from the latent space, we improve results further using a group-based architecture. Additionally, we take initial steps towards $\mathcal{S}$-AE, a novel autoencoder model generating qualitative images without a variational framework, but retaining benefits of autoencoders such as stability and reconstruction quality

Towards generalizable machine learning models for computer-aided diagnosis in medicine

Hidden stratification represents a phenomenon in which a training dataset contains unlabeled (hidden) subsets of cases that may affect machine learning model performance. Machine learning models that ignore the hidden stratification phenomenon--despite promising overall performance measured as accuracy and sensitivity--often fail at predicting the low prevalence cases, but those cases remain important. In the medical domain, patients with diseases are often less common than healthy patients, and a misdiagnosis of a patient with a disease can have significant clinical impacts. Therefore, to build a robust and trustworthy CAD system and a reliable treatment effect prediction model, we cannot only pursue machine learning models with high overall accuracy, but we also need to discover any hidden stratification in the data and evaluate the proposing machine learning models with respect to both overall performance and the performance on certain subsets (groups) of the data, such as the ‘worst group’. In this study, I investigated three approaches for data stratification: a novel algorithmic deep learning (DL) approach that learns similarities among cases and two schema completion approaches that utilize domain expert knowledge. I further proposed an innovative way to integrate the discovered latent groups into the loss functions of DL models to allow for better model generalizability under the domain shift scenario caused by the data heterogeneity. My results on lung nodule Computed Tomography (CT) images and breast cancer histopathology images demonstrate that learning homogeneous groups within heterogeneous data significantly improves the performance of the computer-aided diagnosis (CAD) system, particularly for low-prevalence or worst-performing cases. This study emphasizes the importance of discovering and learning the latent stratification within the data, as it is a critical step towards building ML models that are generalizable and reliable. Ultimately, this discovery can have a profound impact on clinical decision-making, particularly for low-prevalence cases

Integrated Graph Theoretic, Radiomics, and Deep Learning Framework for Personalized Clinical Diagnosis, Prognosis, and Treatment Response Assessment of Body Tumors

Purpose: A new paradigm is beginning to emerge in radiology with the advent of increased computational capabilities and algorithms. The future of radiological reading rooms is heading towards a unique collaboration between computer scientists and radiologists. The goal of computational radiology is to probe the underlying tissue using advanced algorithms and imaging parameters and produce a personalized diagnosis that can be correlated to pathology. This thesis presents a complete computational radiology framework (I GRAD) for personalized clinical diagnosis, prognosis and treatment planning using an integration of graph theory, radiomics, and deep learning. Methods: There are three major components of the I GRAD framework–image segmentation, feature extraction, and clinical decision support. Image Segmentation: I developed the multiparametric deep learning (MPDL) tissue signature model for segmentation of normal and abnormal tissue from multiparametric (mp) radiological images. The segmentation MPDL network was constructed from stacked sparse autoencoders (SSAE) with five hidden layers. The MPDL network parameters were optimized using k-fold cross-validation. In addition, the MPDL segmentation network was tested on an independent dataset. Feature Extraction: I developed the radiomic feature mapping (RFM) and contribution scattergram (CSg) methods for characterization of spatial and inter-parametric relationships in multiparametric imaging datasets. The radiomic feature maps were created by filtering radiological images with first and second order statistical texture filters followed by the development of standardized features for radiological correlation to biology and clinical decision support. The contribution scattergram was constructed to visualize and understand the inter-parametric relationships of the breast MRI as a complex network. This multiparametric imaging complex network was modeled using manifold learning and evaluated using graph theoretic analysis. Feature Integration: The different clinical and radiological features extracted from multiparametric radiological images and clinical records were integrated using a hybrid multiview manifold learning technique termed the Informatics Radiomics Integration System (IRIS). IRIS uses hierarchical clustering in combination with manifold learning to visualize the high-dimensional patient space on a two-dimensional heatmap. The heatmap highlights the similarity and dissimilarity between different patients and variables. Results: All the algorithms and techniques presented in this dissertation were developed and validated using breast cancer as a model for diagnosis and prognosis using multiparametric breast magnetic resonance imaging (MRI). The deep learning MPDL method demonstrated excellent dice similarity of 0.87±0.05 and 0.84±0.07 for segmentation of lesions on malignant and benign breast patients, respectively. Furthermore, each of the methods, MPDL, RFM, and CSg demonstrated excellent results for breast cancer diagnosis with area under the receiver (AUC) operating characteristic (ROC) curve of 0.85, 0.91, and 0.87, respectively. Furthermore, IRIS classified patients with low risk of breast cancer recurrence from patients with medium and high risk with an AUC of 0.93 compared to OncotypeDX, a 21 gene assay for breast cancer recurrence. Conclusion: By integrating advanced computer science methods into the radiological setting, the I-GRAD framework presented in this thesis can be used to model radiological imaging data in combination with clinical and histopathological data and produce new tools for personalized diagnosis, prognosis or treatment planning by physicians

Medical diagnosis using NIR and THz tissue imaging and machine learning methods

The problem of extracting useful information for medical diagnosis from 2D and 3D optical imaging experimental data is of great importance. We are discussing challenges and perspectives of medical diagnosis using machine learning analysis of NIR and THz tissue imaging. The peculiarities of tissue optical clearing for tissue imaging in NIR and THz spectral ranges aiming the improvement of content data analysis, methods of extracting of informative features from experimental data and creating of prognostic models for medical diagnosis using machine learning methods are discussed

Accurate Diagnosis of Colorectal Cancer Based On Histopathology Images Using Artificial Intelligence

Background: Accurate and robust pathological image analysis for colorectal cancer (CRC) diagnosis is time-consuming and knowledge-intensive, but is essential for CRC patients’ treatment. The current heavy workload of pathologists in clinics/hospitals may easily lead to unconscious misdiagnosis of CRC based on daily image analyses. Methods: Based on a state-of-the-art transfer-learned deep convolutional neural network in artificial intelligence (AI), we proposed a novel patch aggregation strategy for clinic CRC diagnosis using weakly labeled pathological whole-slide image (WSI) patches. This approach was trained and validated using an unprecedented and enormously large number of 170,099 patches, \u3e 14,680 WSIs, from \u3e 9631 subjects that covered diverse and representative clinical cases from multi-independent-sources across China, the USA, and Germany. Results: Our innovative AI tool consistently and nearly perfectly agreed with (average Kappa statistic 0.896) and even often better than most of the experienced expert pathologists when tested in diagnosing CRC WSIs from multicenters. The average area under the receiver operating characteristics curve (AUC) of AI was greater than that of the pathologists (0.988 vs 0.970) and achieved the best performance among the application of other AI methods to CRC diagnosis. Our AI-generated heatmap highlights the image regions of cancer tissue/cells. Conclusions: This first-ever generalizable AI system can handle large amounts of WSIs consistently and robustly without potential bias due to fatigue commonly experienced by clinical pathologists. It will drastically alleviate the heavy clinical burden of daily pathology diagnosis and improve the treatment for CRC patients. This tool is generalizable to other cancer diagnosis based on image recognition

Set Representation Learning: A Framework for Learning Gigapixel Images

In Machine Learning, we often encounter data as a set of instances such Point Clouds (set of x,y, and z coordinates), patches from gigapixel images (Digital Pathology, Satellite Imagery, Astronomical Images, etc.), Weakly Supervised Learning, Multiple Instance Learning, and so on. It is then convenient to have Machine Learning or AI algorithms that can learn set representation. However, most of the progress made in the last two decades has been limited to single instance-based algorithms and smaller image datasets such as MNIST, CIFAR10, and CIFAR100. In this work, I present novel algorithms for Set Representation Learning. The contribution of this work is two-fold: 1. This work introduces three novel methods for learning Set Representations; Memory based Exchangeable model (MEM), Graph Neural Network based Set Representation Learning method, and a Hierarchical Set Representation Learning method. 2. This work demonstrates that learning gigapixel images can be formulated as a set representation problem and provides a framework for efficiently learning gigapixel image representations. Different themes are explored for Set Representation Learning. This work investigates Permutation Invariant Representations for Set Learning and introduces a new Permutation Invariant method - ‘MEM’. Memory-based Exchangeable (MEM) model uses a Permutation Invariant architecture and memory networks to learn inter-dependencies/relation between different elements of the set. Subsequently, Graph Neural Networks (GNNs) are studied for Set Representation Learning, and a new GNN based Set Representation Learning method is proposed. Motivated by learning inter-dependencies among different elements in MEM, the proposed method learns an equivalent graphical representation to model interaction and interdependencies among different elements of the set. Lastly, this work introduces a new learning scheme for learning Hierarchical Set Representations. To demonstrate the efficacy of the proposed algorithms, they are validated and benchmarked on a variety of synthetic and real-world datasets such as MNIST, Point Clouds, and Gaussian Distributions. Histopathology Images are used to demonstrate the application of Set Representation Learning for learning gigapixel images. State-of-the-art results on all datasets are achieved, thus demonstrating efficacy