Deep-Learning–Based Characterization of Tumor-Infiltrating Lymphocytes in Breast Cancers From Histopathology Images and Multiomics Data

Purpose: Tumor-infiltrating lymphocytes (TILs) and their spatial characterizations on whole-slide images (WSIs) of histopathology sections have become crucial in diagnosis, prognosis, and treatment response prediction for different cancers. However, fully automatic assessment of TILs on WSIs currently remains a great challenge because of the heterogeneity and large size of WSIs. We present an automatic pipeline based on a cascade-training U-net to generate high-resolution TIL maps on WSIs. Methods: We present global cell-level TIL maps and 43 quantitative TIL spatial image features for 1,000 WSIs of The Cancer Genome Atlas patients with breast cancer. For more specific analysis, all the patients were divided into three subtypes, namely, estrogen receptor (ER)-positive, ER-negative, and triple-negative groups. The associations between TIL scores and gene expression and somatic mutation were examined separately in three breast cancer subtypes. Both univariate and multivariate survival analyses were performed on 43 TIL image features to examine the prognostic value of TIL spatial patterns in different breast cancer subtypes. Results: The TIL score was in strong association with immune response pathway and genes (eg, programmed death-1 and CLTA4). Different breast cancer subtypes showed TIL score in association with mutations from different genes suggesting that different genetic alterations may lead to similar phenotypes. Spatial TIL features that represent density and distribution of TIL clusters were important indicators of the patient outcomes. Conclusion: Our pipeline can facilitate computational pathology-based discovery in cancer immunology and research on immunotherapy. Our analysis results are available for the research community to generate new hypotheses and insights on breast cancer immunology and development

Deep learning integrates histopathology and proteogenomics at a pan-cancer level

We introduce a pioneering approach that integrates pathology imaging with transcriptomics and proteomics to identify predictive histology features associated with critical clinical outcomes in cancer. We utilize 2,755 H&E-stained histopathological slides from 657 patients across 6 cancer types from CPTAC. Our models effectively recapitulate distinctions readily made by human pathologists: tumor vs. normal (AUROC = 0.995) and tissue-of-origin (AUROC = 0.979). We further investigate predictive power on tasks not normally performed from H&E alone, including TP53 prediction and pathologic stage. Importantly, we describe predictive morphologies not previously utilized in a clinical setting. The incorporation of transcriptomics and proteomics identifies pathway-level signatures and cellular processes driving predictive histology features. Model generalizability and interpretability is confirmed using TCGA. We propose a classification system for these tasks, and suggest potential clinical applications for this integrated human and machine learning approach. A publicly available web-based platform implements these models

HEROHE Challenge: Predicting HER2 Status in Breast Cancer from Hematoxylin–Eosin Whole-Slide Imaging

Breast cancer is the most common malignancy in women worldwide, and is responsible for more than half a million deaths each year. The appropriate therapy depends on the evaluation of the expression of various biomarkers, such as the human epidermal growth factor receptor 2 (HER2) transmembrane protein, through specialized techniques, such as immunohistochemistry or in situ hybridization. In this work, we present the HER2 on hematoxylin and eosin (HEROHE) challenge, a parallel event of the 16th European Congress on Digital Pathology, which aimed to predict the HER2 status in breast cancer based only on hematoxylin–eosin-stained tissue samples, thus avoiding specialized techniques. The challenge consisted of a large, annotated, whole-slide images dataset (509), specifically collected for the challenge. Models for predicting HER2 status were presented by 21 teams worldwide. The best-performing models are presented by detailing the network architectures and key parameters. Methods are compared and approaches, core methodologies, and software choices contrasted. Different evaluation metrics are discussed, as well as the performance of the presented models for each of these metrics. Potential differences in ranking that would result from different choices of evaluation metrics highlight the need for careful consideration at the time of their selection, as the results show that some metrics may misrepresent the true potential of a model to solve the problem for which it was developed. The HEROHE dataset remains publicly available to promote advances in the field of computational pathology

An imaging biomarker of tumor-infiltrating lymphocytes to risk-stratify patients with HPV-associated oropharyngeal cancer

BACKGROUND: Human papillomavirus (HPV)-associated oropharyngeal squamous cell carcinoma (OPSCC) has excellent control rates compared to nonvirally associated OPSCC. Multiple trials are actively testing whether de-escalation of treatment intensity for these patients can maintain oncologic equipoise while reducing treatment-related toxicity. We have developed OP-TIL, a biomarker that characterizes the spatial interplay between tumor-infiltrating lymphocytes (TILs) and surrounding cells in histology images. Herein, we sought to test whether OP-TIL can segregate stage I HPV-associated OPSCC patients into low-risk and high-risk groups and aid in patient selection for de-escalation clinical trials. METHODS: Association between OP-TIL and patient outcome was explored on whole slide hematoxylin and eosin images from 439 stage I HPV-associated OPSCC patients across 6 institutional cohorts. One institutional cohort (n = 94) was used to identify the most prognostic features and train a Cox regression model to predict risk of recurrence and death. Survival analysis was used to validate the algorithm as a biomarker of recurrence or death in the remaining 5 cohorts (n = 345). All statistical tests were 2-sided. RESULTS: OP-TIL separated stage I HPV-associated OPSCC patients with 30 or less pack-year smoking history into low-risk (2-year disease-free survival [DFS] = 94.2%; 5-year DFS = 88.4%) and high-risk (2-year DFS = 82.5%; 5-year DFS = 74.2%) groups (hazard ratio = 2.56, 95% confidence interval = 1.52 to 4.32; P \u3c .001), even after adjusting for age, smoking status, T and N classification, and treatment modality on multivariate analysis for DFS (hazard ratio = 2.27, 95% confidence interval = 1.32 to 3.94; P = .003). CONCLUSIONS: OP-TIL can identify stage I HPV-associated OPSCC patients likely to be poor candidates for treatment de-escalation. Following validation on previously completed multi-institutional clinical trials, OP-TIL has the potential to be a biomarker, beyond clinical stage and HPV status, that can be used clinically to optimize patient selection for de-escalation

Better prognostic markers for nonmuscle invasive papillary urothelial carcinomas

Bladder cancer is a common type of cancer, especially among men in developed countries. Most cancers in the urinary bladder are papillary urothelial carcinomas. They are characterized by a high recurrence frequency (up to 70 %) after local resection. It is crucial for prognosis to discover these recurrent tumours at an early stage, especially before they become muscle-invasive. Reliable prognostic biomarkers for tumour recurrence and stage progression are lacking. This is why patients diagnosed with a non-muscle invasive bladder cancer follow extensive follow-up regimens with possible serious side effects and with high costs for the healthcare systems. WHO grade and tumour stage are two central biomarkers currently having great impact on both treatment decisions and follow-up regimens. However, there are concerns regarding the reproducibility of WHO grading, and stage classification is challenging in small and fragmented tumour material. In Paper I, we examined the reproducibility and the prognostic value of all the individual microscopic features making up the WHO grading system. Among thirteen extracted features there was considerable variation in both reproducibility and prognostic value. The only feature being both reasonably reproducible and statistically significant prognostic was cell polarity. We concluded that further validation studies are needed on these features, and that future grading systems should be based on well-defined features with true prognostic value. With the implementation of immunotherapy, there is increasing interest in tumour immune response and the tumour microenvironment. In a search for better prognostic biomarkers for tumour recurrence and stage progression, in Paper II, we investigated the prognostic value of tumour infiltrating immune cells (CD4, CD8, CD25 and CD138) and previously investigated cell proliferation markers (Ki-67, PPH3 and MAI). Low Ki 67 and tumour multifocality were associated with increased recurrence risk. Recurrence risk was not affected by the composition of immune cells. For stage progression, the only prognostic immune cell marker was CD25. High values for MAI was also strongly associated with stage progression. However, in a multivariate analysis, the most prognostic feature was a combination of MAI and CD25. BCG-instillations in the bladder are indicated in intermediate and high-risk non-muscle invasive bladder cancer patients. This old-fashion immunotherapy has proved to reduce both recurrence- and progression-risk, although it is frequently followed by unpleasant side-effects. As many as 30-50% of high-risk patients receiving BCG instillations, fail by develop high-grade recurrences. They do not only suffer from unnecessary side-effects, but will also have a delay in further treatment. Together with colleagues at three different Dutch hospitals, in Paper III, we looked at the prognostic and predictive value of T1-substaging. A T1-tumour invades the lamina propria, and we wanted to separate those with micro- from those with extensive invasion. We found that BCG-failure was more common among patients with extensive invasion. Furthermore, T1-substaging was associated with both high-grade recurrence-free and progression-free survival. Finally, in Paper IV, we wanted to investigate the prognostic value of two classical immunohistochemical markers, p53 and CK20, and compare them with previously investigated proliferation markers. p53 is a surrogate marker for mutations in the gene TP53, considered to be a main characteristic for muscle-invasive tumours. CK20 is a surrogate marker for luminal tumours in the molecular classification of bladder cancer, and is frequently used to distinguish reactive urothelial changes from urothelial carcinoma in situ. We found both positivity for p53 and CK20 to be significantly associated with stage progression, although not performing better than WHO grade and stage. The proliferation marker MAI, had the highest prognostic value in our study. Any combination of variables did not perform better in a multivariate analysis than MAI alone

Computer Vision for Tissue Characterization and Outcome Prediction in Cancer

The aim of this dissertation was to investigate the use of computer vision for tissue characterization and patient outcome prediction in cancer. This work focused on analysis of digitized tissue specimens, which were stained only for basic morphology (i.e. hematoxylin and eosin). The applicability of texture analysis and convolutional neural networks was evaluated for detection of biologically and clinically relevant features. Moreover, novel approaches to guide ground-truth annotation and outcome-supervised learning for prediction of patient survival directly from the tumor tissue images without expert guidance was investigated. We first studied quantification of tumor viability through segmentation of necrotic and viable tissue compartments. We developed a regional texture analysis method, which was trained and tested on whole sections of mouse xenograft models of human lung cancer. Our experiments showed that the proposed segmentation was able to discriminate between viable and non-viable tissue regions with high accuracy when compared to human expert assessment. We next investigated the feasibility of pre-trained convolutional neural networks in analysis of breast cancer tissue, aiming to quantify tumor-infiltrating lymphocytes in the specimens. Interestingly, our results showed that pre-trained convolutional neural networks can be adapted for analysis of histological image data, outperforming texture analysis. The results also indicated that the computerized assessment was on par with pathologist assessments. Moreover, the study presented an image annotation technique guided by specific antibody staining for improved ground-truth labeling. Direct outcome prediction in breast cancer was then studied using a nationwide patient cohort. A computerized pipeline, which incorporated orderless feature aggregation and convolutional image descriptors for outcome-supervised classification, resulted in a risk grouping that was predictive of both disease-specific and overall survival. Surprisingly, further analysis suggested that the computerized risk prediction was also an independent prognostic factor that provided information complementary to the standard clinicopathological factors. This doctoral thesis demonstrated how computer-vision methods can be powerful tools in analysis of cancer tissue samples, highlighting strategies for supervised characterization of tissue entities and an approach for identification of novel prognostic morphological features.Kudosnäytteiden mikroskooppisten piirteiden visuaalinen tarkastelu on yksi tärkeimmistä määrityksistä syöpäpotilaiden diagnosoinnissa ja hoidon suunnittelussa. Edistyneet kuvantamisteknologiat ovat mahdollistaneet histologisten kasvainkudosnäytteiden digitalisoinnin tarkalla resoluutiolla. Näytteiden digitalisoinnin seurauksena niiden analysointiin voidaan soveltaa edistyneitä koneoppimiseen perustuvia konenäön menetelmiä. Tämä väitöskirja tutkii konenäön menetelmien soveltamista syöpäkudosnäytteiden laskennalliseen analyysiin. Työssä tutkitaan yksittäisten histologisten entiteettien, kuten nekroottisen kudoksen ja immuunisolujen automaattista kvantifiointia. Lisäksi työssä esitellään menetelmä potilaan selviytymisen ennustamiseen pelkkään kudosmorfologiaan perustuen

Artificial intelligence (AI) in medicine, current applications and future role with special emphasis on its potential and promise in pathology: Present and future impact, obstacles including costs and acceptance among pathologists, practical and philosophical considerations. A comprehensive review

Background: The role of Artificial intelligence (AI) which is defined as the ability of computers to perform tasks that normally require human intelligence is constantly expanding. Medicine was slow to embrace AI. However, the role of AI in medicine is rapidly expanding and promises to revolutionize patient care in the coming years. In addition, it has the ability to democratize high level medical care and make it accessible to all parts of the world.Main text: Among specialties of medicine, some like radiology were relatively quick to adopt AI whereas others especially pathology (and surgical pathology in particular) are only just beginning to utilize AI. AI promises to play a major role in accurate diagnosis, prognosis and treatment of cancers. In this paper, the general principles of AI are defined first followed by a detailed discussion of its current role in medicine. In the second half of this comprehensive review, the current and future role of AI in surgical pathology is discussed in detail including an account of the practical difficulties involved and the fear of pathologists of being replaced by computer algorithms. A number of recent studies which demonstrate the usefulness of AI in the practice of surgical pathology are highlighted.Conclusion: AI has the potential to transform the practice of surgical pathology by ensuring rapid and accurate results and enabling pathologists to focus on higher level diagnostic and consultative tasks such as integrating molecular, morphologic and clinical information to make accurate diagnosis in difficult cases, determine prognosis objectively and in this way contribute to personalized care