Computer Vision for Tissue Characterization and Outcome Prediction in Cancer

The aim of this dissertation was to investigate the use of computer vision for tissue characterization and patient outcome prediction in cancer. This work focused on analysis of digitized tissue specimens, which were stained only for basic morphology (i.e. hematoxylin and eosin). The applicability of texture analysis and convolutional neural networks was evaluated for detection of biologically and clinically relevant features. Moreover, novel approaches to guide ground-truth annotation and outcome-supervised learning for prediction of patient survival directly from the tumor tissue images without expert guidance was investigated. We first studied quantification of tumor viability through segmentation of necrotic and viable tissue compartments. We developed a regional texture analysis method, which was trained and tested on whole sections of mouse xenograft models of human lung cancer. Our experiments showed that the proposed segmentation was able to discriminate between viable and non-viable tissue regions with high accuracy when compared to human expert assessment. We next investigated the feasibility of pre-trained convolutional neural networks in analysis of breast cancer tissue, aiming to quantify tumor-infiltrating lymphocytes in the specimens. Interestingly, our results showed that pre-trained convolutional neural networks can be adapted for analysis of histological image data, outperforming texture analysis. The results also indicated that the computerized assessment was on par with pathologist assessments. Moreover, the study presented an image annotation technique guided by specific antibody staining for improved ground-truth labeling. Direct outcome prediction in breast cancer was then studied using a nationwide patient cohort. A computerized pipeline, which incorporated orderless feature aggregation and convolutional image descriptors for outcome-supervised classification, resulted in a risk grouping that was predictive of both disease-specific and overall survival. Surprisingly, further analysis suggested that the computerized risk prediction was also an independent prognostic factor that provided information complementary to the standard clinicopathological factors. This doctoral thesis demonstrated how computer-vision methods can be powerful tools in analysis of cancer tissue samples, highlighting strategies for supervised characterization of tissue entities and an approach for identification of novel prognostic morphological features.Kudosnäytteiden mikroskooppisten piirteiden visuaalinen tarkastelu on yksi tärkeimmistä määrityksistä syöpäpotilaiden diagnosoinnissa ja hoidon suunnittelussa. Edistyneet kuvantamisteknologiat ovat mahdollistaneet histologisten kasvainkudosnäytteiden digitalisoinnin tarkalla resoluutiolla. Näytteiden digitalisoinnin seurauksena niiden analysointiin voidaan soveltaa edistyneitä koneoppimiseen perustuvia konenäön menetelmiä. Tämä väitöskirja tutkii konenäön menetelmien soveltamista syöpäkudosnäytteiden laskennalliseen analyysiin. Työssä tutkitaan yksittäisten histologisten entiteettien, kuten nekroottisen kudoksen ja immuunisolujen automaattista kvantifiointia. Lisäksi työssä esitellään menetelmä potilaan selviytymisen ennustamiseen pelkkään kudosmorfologiaan perustuen

Deep learning-based instance segmentation for the precise automated quantification of digital breast cancer immunohistochemistry images

After the 24 months embargo, this version of the article was accepted for publication, after peer review and does not reflect post-acceptance improvements, or any corrections. The published version is available online (2022-01-14) at: https://doi.org/10.1016/j.eswa.2021.116471.The quantification of biomarkers on immunohistochemistry breast cancer images is essential for defining appropriate therapy for breast cancer patients, as well as for extracting relevant information on disease prognosis. This is an arduous and time-consuming task that may introduce a bias in the results due to intra- and inter-observer variability which could be alleviated by making use of automatic quantification tools. However, this is not a simple processing task given the heterogeneity of breast tumors that results in non-uniformly distributed tumor cells exhibiting different staining colors and intensity, size, shape, and texture, of the nucleus, cytoplasm and membrane. In this research work we demonstrate the feasibility of using a deep learning-based instance segmentation architecture for the automatic quantification of both nuclear and membrane biomarkers applied to IHC-stained slides. We have solved the cumbersome task of training set generation with the design and implementation of a web platform, which has served as a hub for communication and feedback between researchers and pathologists as well as a system for the validation of the automatic image processing models. Through this tool, we have collected annotations over samples of HE, ER and Ki-67 (nuclear biomarkers) and HER2 (membrane biomarker) IHC-stained images. Using the same deep learning network architecture, we have trained two models, so-called nuclei- and membrane-aware segmentation models, which, once successfully validated, have revealed to be a promising method to segment nuclei instances in IHC-stained images. The quantification method proposed in this work has been integrated into the developed web platform and is currently being used as a decision support tool by pathologists

Mammography

In this volume, the topics are constructed from a variety of contents: the bases of mammography systems, optimization of screening mammography with reference to evidence-based research, new technologies of image acquisition and its surrounding systems, and case reports with reference to up-to-date multimodality images of breast cancer. Mammography has been lagged in the transition to digital imaging systems because of the necessity of high resolution for diagnosis. However, in the past ten years, technical improvement has resolved the difficulties and boosted new diagnostic systems. We hope that the reader will learn the essentials of mammography and will be forward-looking for the new technologies. We want to express our sincere gratitude and appreciation?to all the co-authors who have contributed their work to this volume

Artificial intelligence for breast cancer precision pathology

Breast cancer is the most common cancer type in women globally but is associated with a continuous decline in mortality rates. The improved prognosis can be partially attributed to effective treatments developed for subgroups of patients. However, nowadays, it remains challenging to optimise treatment plans for each individual. To improve disease outcome and to decrease the burden associated with unnecessary treatment and adverse drug effects, the current thesis aimed to develop artificial intelligence based tools to improve individualised medicine for breast cancer patients. In study I, we developed a deep learning based model (DeepGrade) to stratify patients that were associated with intermediate risks. The model was optimised with haematoxylin and eosin (HE) stained whole slide images (WSIs) with grade 1 and 3 tumours and applied to stratify grade 2 tumours into grade 1-like (DG2-low) and grade 3-like (DG2-high) subgroups. The efficacy of the DeepGrade model was validated using recurrence free survival where the dichotomised groups exhibited an adjusted hazard ratio (HR) of 2.94 (95% confidence interval [CI] 1.24-6.97, P = 0.015). The observation was further confirmed in the external test cohort with an adjusted HR of 1.91 (95% CI: 1.11-3.29, P = 0.019). In study II, we investigated whether deep learning models were capable of predicting gene expression levels using the morphological patterns from tumours. We optimised convolutional neural networks (CNNs) to predict mRNA expression for 17,695 genes using HE stained WSIs from the training set. An initial evaluation on the validation set showed that a significant correlation between the RNA-seq measurements and model predictions was observed for 52.75% of the genes. The models were further tested in the internal and external test sets. Besides, we compared the model's efficacy in predicting RNA-seq based proliferation scores. Lastly, the ability of capturing spatial gene expression variations for the optimised CNNs was evaluated and confirmed using spatial transcriptomics profiling. In study III, we investigated the relationship between intra-tumour gene expression heterogeneity and patient survival outcomes. Deep learning models optimised from study II were applied to generate spatial gene expression predictions for the PAM50 gene panel. A set of 11 texture based features and one slide average gene expression feature per gene were extracted as input to train a Cox proportional hazards regression model with elastic net regularisation to predict patient risk of recurrence. Through nested cross-validation, the model dichotomised the training cohort into low and high risk groups with an adjusted HR of 2.1 (95% CI: 1.30-3.30, P = 0.002). The model was further validated on two external cohorts. In study IV, we investigated the agreement between the Stratipath Breast, which is the modified, commercialised DeepGrade model developed in study I, and the Prosigna® test. Both tests sought to stratify patients with distinct prognosis. The outputs from Stratipath Breast comprise a risk score and a two-level risk stratification whereas the outputs from Prosigna® include the risk of recurrence score and a three-tier risk stratification. By comparing the number of patients assigned to ‘low’ or ‘high’ risk groups, we found an overall moderate agreement (76.09%) between the two tests. Besides, the risk scores by two tests also revealed a good correlation (Spearman's rho = 0.59, P = 1.16E-08). In addition, a good correlation was observed between the risk score from each test and the Ki67 index. The comparison was also carried out in the subgroup of patients with grade 2 tumours where similar but slightly dropped correlations were found

Added benefits of computer-assisted analysis of Hematoxylin-Eosin stained breast histopathological digital slides

This thesis aims at determining if computer-assisted analysis can be used to better understand pathologists’ perception of mitotic figures on Hematoxylin-Eosin (HE) stained breast histopathological digital slides. It also explores the feasibility of reproducible histologic nuclear atypia scoring by incorporating computer-assisted analysis to cytological scores given by a pathologist. In addition, this thesis investigates the possibility of computer-assisted diagnosis for categorizing HE breast images into different subtypes of cancer or benign masses. In the first study, a data set of 453 mitoses and 265 miscounted non-mitoses within breast cancer digital slides were considered. Different features were extracted from the objects in different channels of eight colour spaces. The findings from the first research study suggested that computer-aided image analysis can provide a better understanding of image-related features related to discrepancies among pathologists in recognition of mitoses. Two tasks done routinely by the pathologists are making diagnosis and grading the breast cancer. In the second study, a new tool for reproducible nuclear atypia scoring in breast cancer histological images was proposed. The third study proposed and tested MuDeRN (MUlti-category classification of breast histopathological image using DEep Residual Networks), which is a framework for classifying hematoxylin-eosin stained breast digital slides either as benign or cancer, and then categorizing cancer and benign cases into four different subtypes each. The studies indicated that computer-assisted analysis can aid in both nuclear grading (COMPASS) and breast cancer diagnosis (MuDeRN). The results could be used to improve current status of breast cancer prognosis estimation through reducing the inter-pathologist disagreement in counting mitotic figures and reproducible nuclear grading. It can also improve providing a second opinion to the pathologist for making a diagnosis

Machine Learning Strategies to Analyze Quantitative Ultrasound Multi-Parametric Images for Prediction of Therapy Response in Breast Cancer Patients

In this thesis project, two novel machine learning strategies were investigated to predict tumor response to neoadjuvant chemotherapy (NAC) at pre-treatment using quantitative ultrasound (QUS) multi-parametric images. The ultrasound data for analytical development and evaluation of the methodologies investigated in this project were acquired from 181 patients diagnosed with locally advanced breast cancer (LABC) and planned for NAC followed by surgery. The QUS multi-parametric images were generated using spectral analyses on the raw ultrasound radiofrequency (RF) data acquired before starting the NAC. In the first machine learning approach investigated in this project, distinct intra-tumor regions were identified within the parametric maps using a hidden Markov random field (HMRF) and its expectation-maximization (EM) algorithm. Several hand-crafted features characterizing the tumor, intra-tumor regions, and the tumor margin were extracted from different parametric images. A multi-step feature selection procedure was applied to construct a QUS biomarker for response prediction. Evaluation results on an independent test set indicated that the developed biomarker using the characteristics of intra-tumor regions and tumor margin in conjunction with a decision tree model with adaptive boosting (AdaBoost) as the classifier could predict the treatment response of patients at pre-treatment with an accuracy of 85.4% and an area under the receiver operating characteristic (ROC) curve (AUC) of 0.89. In the second machine learning approach investigated in this project, two deep convolutional neural network (DCNN) architectures including the residual network (ResNet) and residual attention network (RAN) were explored for extracting optimal feature maps from the parametric images, with a fully connected network for response prediction. Results demonstrated that the developed model with the RAN architecture to extract feature maps from the expanded parametric images of the tumor core and margin had a superior performance with an accuracy of 0.88 and an AUC of 0.86 on the independent test set. Also, survival analysis demonstrated a statistically significant difference between survival curves of the two response cohorts identified at pre-treatment based on both the conventional machine learning method and the deep learning model. Obtained results in this study demonstrated a great promise of QUS multi-parametric imaging integrated with both unsupervised learning methods in identifying distinct breast cancer intra-tumor regions and traditional classification techniques, as well as deep convolutional neural networks in predicting tumor response to NAC prior to start of treatment

Breast dynamic contrast-enhanced-magnetic resonance imaging and radiomics: State of art

Breast cancer represents the most common malignancy in women, being one of the most frequent cause of cancer-related mortality. Ultrasound, mammography, and magnetic resonance imaging (MRI) play a pivotal role in the diagnosis of breast lesions, with different levels of accuracy. Particularly, dynamic contrast-enhanced MRI has shown high diagnostic value in detecting multifocal, multicentric, or contralateral breast cancers. Radiomics is emerging as a promising tool for quantitative tumor evaluation, allowing the extraction of additional quantitative data from radiological imaging acquired with different modalities. Radiomics analysis may provide novel information through the quantification of lesions heterogeneity, that may be relevant in clinical practice for the characterization of breast lesions, prediction of tumor response to systemic therapies and evaluation of prognosis in patients with breast cancers. Several published studies have explored the value of radiomics with good-to-excellent diagnostic and prognostic performances for the evaluation of breast lesions. Particularly, the integrations of radiomics data with other clinical and histopathological parameters have demonstrated to improve the prediction of tumor aggressiveness with high accuracy and provided precise models that will help to guide clinical decisions and patients management. The purpose of this article in to describe the current application of radiomics in breast dynamic contrast-enhanced MRI

Machine Learning Strategies to Analyze Quantitative Ultrasound Multi-Parametric Images for Prediction of Therapy Response in Breast Cancer Patients

In this thesis project, two novel machine learning strategies were investigated to predict tumor response to neoadjuvant chemotherapy (NAC) at pre-treatment using quantitative ultrasound (QUS) multi-parametric images. The ultrasound data for analytical development and evaluation of the methodologies investigated in this project were acquired from 181 patients diagnosed with locally advanced breast cancer (LABC) and planned for NAC followed by surgery. The QUS multi-parametric images were generated using spectral analyses on the raw ultrasound radiofrequency (RF) data acquired before starting the NAC. In the first machine learning approach investigated in this project, distinct intra-tumor regions were identified within the parametric maps using a hidden Markov random field (HMRF) and its expectation-maximization (EM) algorithm. Several hand-crafted features characterizing the tumor, intra-tumor regions, and the tumor margin were extracted from different parametric images. A multi-step feature selection procedure was applied to construct a QUS biomarker for response prediction. Evaluation results on an independent test set indicated that the developed biomarker using the characteristics of intra-tumor regions and tumor margin in conjunction with a decision tree model with adaptive boosting (AdaBoost) as the classifier could predict the treatment response of patients at pre-treatment with an accuracy of 85.4% and an area under the receiver operating characteristic (ROC) curve (AUC) of 0.89. In the second machine learning approach investigated in this project, two deep convolutional neural network (DCNN) architectures including the residual network (ResNet) and residual attention network (RAN) were explored for extracting optimal feature maps from the parametric images, with a fully connected network for response prediction. Results demonstrated that the developed model with the RAN architecture to extract feature maps from the expanded parametric images of the tumor core and margin had a superior performance with an accuracy of 0.88 and an AUC of 0.86 on the independent test set. Also, survival analysis demonstrated a statistically significant difference between survival curves of the two response cohorts identified at pre-treatment based on both the conventional machine learning method and the deep learning model. Obtained results in this study demonstrated a great promise of QUS multi-parametric imaging integrated with both unsupervised learning methods in identifying distinct breast cancer intra-tumor regions and traditional classification techniques, as well as deep convolutional neural networks in predicting tumor response to NAC prior to start of treatment