PathologyGAN: Learning deep representations of cancer tissue

Histopathological images of tumours contain abundant information about how tumours grow and how they interact with their micro-environment. Better understanding of tissue phenotypes in these images could reveal novel determinants of pathological processes underlying cancer, and in turn improve diagnosis and treatment options. Advances of Deep learning makes it ideal to achieve those goals, however, its application is limited by the cost of high quality labels from patients data. Unsupervised learning, in particular, deep generative models with representation learning properties provides an alternative path to further understand cancer tissue phenotypes, capturing tissue morphologies. In this paper, we develop a framework which allows Generative Adversarial Networks (GANs) to capture key tissue features and uses these characteristics to give structure to its latent space. To this end, we trained our model on two different datasets, an H and E colorectal cancer tissue from the National Center for Tumor diseases (NCT, Germany) and an H and E breast cancer tissue from the Netherlands Cancer Institute (NKI, Netherlands) and Vancouver General Hospital (VGH, Canada). Composed of 86 slide images and 576 tissue micro-arrays (TMAs) respectively. We show that our model generates high quality images, with a Frechet Inception Distance (FID) of 16.65 (breast cancer) and 32.05 (colorectal cancer). We further assess the quality of the images with cancer tissue characteristics (e.g. count of cancer, lymphocytes, or stromal cells), using quantitative information to calculate the FID and showing consistent performance of 9.86. Additionally, the latent space of our model shows an interpretable structure and allows semantic vector operations that translate into tissue feature transformations. Furthermore, ratings from two expert pathologists found no significant difference between our generated tissue images from real ones. The code, generated images, and pretrained model are available at https://github.com/AdalbertoCq/Pathology-GA

Learning a Low Dimensional Manifold of Real Cancer Tissue with PathologyGAN

Histopathological images contain information about how a tumor interacts with its micro-environment. Better understanding of such interaction holds the key for improved diagnosis and treatment of cancer. Deep learning shows promise on achieving those goals, however, its application is limited by the cost of high quality labels. Unsupervised learning, in particular, deep generative models with representation learning properties provides an alternative path to further understand cancer tissue phenotypes, capturing tissue morphologies. We present a deep generative model that learns to simulate high-fidelity cancer tissue images while mapping the real images onto an interpretable low dimensional latent space. The key to the model is an encoder trained by a previously developed generative adversarial network, PathologyGAN. Here we provide examples of how the latent space holds morphological characteristics of cancer tissue (e.g. tissue type or cancer, lymphocytes, and stroma cells). We tested the general applicability of our representations in three different settings: latent space visualization, training a tissue type classifier over latent representations, and on multiple instance learning (MIL). Latent visualizations of breast cancer tissue show that distinct regions of the latent space enfold different characteristics (stroma, lymphocytes, and cancer cells). A logistic regression for colorectal tissue type classification trained over latent projections achieves 87% accuracy. Finally, we used the attention-based deep MIL for predicting presence of epithelial cells in colorectal tissue, achieving 90% accuracy. Our results show that PathologyGAN captures distinct phenotype characteristics, paving the way for further understanding of tumor micro-environment and ultimately refining histopathological classification for diagnosis and treatment