Comparison of Different Methods for Tissue Segmentation in Histopathological Whole-Slide Images

Tissue segmentation is an important pre-requisite for efficient and accurate diagnostics in digital pathology. However, it is well known that whole-slide scanners can fail in detecting all tissue regions, for example due to the tissue type, or due to weak staining because their tissue detection algorithms are not robust enough. In this paper, we introduce two different convolutional neural network architectures for whole slide image segmentation to accurately identify the tissue sections. We also compare the algorithms to a published traditional method. We collected 54 whole slide images with differing stains and tissue types from three laboratories to validate our algorithms. We show that while the two methods do not differ significantly they outperform their traditional counterpart (Jaccard index of 0.937 and 0.929 vs. 0.870, p < 0.01).Comment: Accepted for poster presentation at the IEEE International Symposium on Biomedical Imaging (ISBI) 201

A Survey on Deep Learning in Medical Image Analysis

Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, segmentation, registration, and other tasks and provide concise overviews of studies per application area. Open challenges and directions for future research are discussed.Comment: Revised survey includes expanded discussion section and reworked introductory section on common deep architectures. Added missed papers from before Feb 1st 201

Pushing the Boundaries of Biomolecule Characterization through Deep Learning

The importance of studying biological molecules in living organisms can hardly be overstated as they regulate crucial processes in living matter of all kinds.Their ubiquitous nature makes them relevant for disease diagnosis, drug development, and for our fundamental understanding of the complex systems of biology.However, due to their small size, they scatter too little light on their own to be directly visible and available for study.Thus, it is necessary to develop characterization methods which enable their elucidation even in the regime of very faint signals. Optical systems, utilizing the relatively low intrusiveness of visible light, constitute one such approach of characterization. However, the optical systems currently capable of analyzing single molecules in the nano-sized regime today either require the species of interest to be tagged with visible labels like fluorescence or chemically restrained on a surface to be analyzed.Ergo, there exist effectively no methods of characterizing very small biomolecules under naturally relevant conditions through unobtrusive probing. Nanofluidic Scattering Microscopy is a method introduced in this thesis which bridges this gap by enabling the real-time label-free size-and-weight determination of freely diffusing molecules directly in small nano-sized channels. However, the molecule signals are so faint, and the background noise so complex with high spatial and temporal variation, that standard methods of data analysis are incapable of elucidating the molecules\u27 properties of relevance in any but the least challenging conditions.To remedy the weak signal, and realize the method\u27s full potential, this thesis\u27 focus is the development of a versatile deep-learning based computer-vision platform to overcome the bottleneck of data analysis. We find that said platform has considerably increased speed, accuracy, precision and limit of detection compared to standard methods, constituting even a lower detection limit than any other method of label-free optical characterization currently available. In this regime, hitherto elusive species of biomolecules become accessible for study, potentially opening up entirely new avenues of biological research. These results, along with many others in the context of deep learning for optical microscopy in biological applications, suggest that deep learning is likely to be pivotal in solving the complex image analysis problems of the present and enabling new regimes of study within microscopy-based research in the near future

Automated Grading of Bladder Cancer using Deep Learning

PhD thesis in Information technologyUrothelial carcinoma is the most common type of bladder cancer and is among the cancer types with the highest recurrence rate and lifetime treatment cost per patient. Diagnosed patients are stratified into risk groups, mainly based on the histological grade and stage. However, it is well known that correct grading of bladder cancer suffers from intra- and interobserver variability and inconsistent reproducibility between pathologists, potentially leading to under- or overtreatment of the patients. The economic burden, unnecessary patient suffering, and additional load on the health care system illustrate the importance of developing new tools to aid pathologists. With the introduction of digital pathology, large amounts of data have been made available in the form of digital histological whole-slide images (WSI). However, despite the massive amount of data, annotations for the given data are lacking. Another potential problem is that the tissue samples of urothelial carcinoma contain a mixture of damaged tissue, blood, stroma, muscle, and urothelium, where it is mainly the urothelium tissue that is diagnostically relevant for grading. A method for tissue segmentation is investigated, where the aim is to segment WSIs into the six tissue classes: urothelium, stroma, muscle, damaged tissue, blood, and background. Several methods based on convolutional neural networks (CNN) for tile-wise classification are proposed. Both single-scale and multiscale models were explored to see if including more magnification levels would improve the performance. Different techniques, such as unsupervised learning, semi-supervised learning, and domain adaptation techniques, are explored to mitigate the challenge of missing large quantities of annotated data. It is necessary to extract tiles from the WSI since it is intractable to process the entire WSI at full resolution at once. We have proposed a method to parameterize and automate the task of extracting tiles from different scales with a region of interest (ROI) defined at one of the scales. The method is reproducible and easy to describe by reporting the parameters. A pipeline for automated diagnostic grading is proposed, called TRIgrade. First, the tissue segmentation method is utilized to find the diagnostically relevant urothelium tissue. Then, the parameterized tile extraction method is used to extract tiles from the urothelium regions at three magnification levels from 300 WSIs. The extracted tiles form the training, validation, and test data used to train and test a diagnostic model. The final system outputs a segmented tissue image showing all the tissue regions in the WSI, a WHO grade heatmap indicating low- and high-grade carcinoma regions, and finally, a slide-level WHO grade prediction. The proposed TRIgrade pipeline correctly graded 45 of 50 WSIs, achieving an accuracy of 90%

Differentiating Human Embryonic Stem Cells in Micropatterns to Study Cell Fate Specification and Morphogenetic Events During Gastrulation

During mammalian embryogenesis, the first major lineage segregation occurs when embryonic epiblast, and extraembryonic trophectoderm and hypoblast arise in the blastocyst. In the next fundamental and conserved phase of animal embryogenesis known as gastrulation, extraembryonic cells provide signals to epiblast to instruct embryonic patterning, and epiblast gives rise to germ layers ectoderm, mesoderm, and endoderm, that will establish all embryonic tissues. Proper specification and morphogenesis of germ layers during gastrulation is vital for correct embryonic development. Due to ethical and legal restrictions limiting human embryo studies, human gastrulation is poorly understood. Our knowledge of human gastrulation has largely been derived from studies in model organisms, including mouse and more recently, cynomolgus monkey. However, interspecies differences underscore the need for alternative human gastrulation models. In this regard, human and mouse embryonic stem cells have been shown to recapitulate aspects of in vivo gastrulation including germ layer specification, and internalization and elongation morphogenesis. These in vitro systems represent powerful models of gastrulation due to the ease of genetic manipulations and the ability to finely control experimental factors. Human embryonic stem cells, treated with BMP4 for 44 hours in spatially confined micro-discs of extracellular matrix, have been shown to differentiate into 2D micro-colonies termed gastruloids. These gastruloids display highly reproducible differentiation of germ layers and extraembryonic cell types in a radial arrangement. We used combinatorial single-cell RNA sequencing and immunofluorescence imaging to characterize these BMP4-treated 2D gastruloids, and showed the formation in gastruloids of seven cell types, including epiblast, prospective ectoderm, two populations of mesoderm, and endoderm, as well as previously undescribed cell types in 2D gastruloids, primordial germ cell-like cells, and extraembryonic cells that are transcriptionally similar to trophectoderm and amnion. Comparative transcriptomic analyses with human, mouse, and cynomolgus monkey gastrulae support the notion that 2D gastruloid differentiation recapitulates formation of cell types relevant to and models an early-mid stage of in vivo gastrulation. Time course scRNA-seq and immunofluorescence analyses of 2D gastruloid differentiation after 12, 24, and 44 hours of BMP4 treatment showed that germ layer emergence in gastruloids follows the temporal sequence of in vivo gastrulation, with epiblast and ectoderm precursors forming at 12 hour, mesendoderm precursors arising from epiblast at 24 hour to give rise to nascent mesoderm and endoderm at 44 hour, when primordial germ cell-like cells also form. Comparison with human gastrula also showed similarity in transcriptomes and differentiation trajectories of gastruloid cells to their in vivo counterparts. Dynamic changes in transcripts encoding components of key signaling pathways support a BMP, WNT and Nodal hierarchy underlying germ layer specification conserved across mammals, with FGF and HIPPO signaling being active throughout the time course of 2D micropattern gastruloid differentiation. To probe morphogenetic properties of gastruloid cells, differentiated gastruloids treated with BMP4 for 44 hours were dissociated and re-seeded onto extracellular matrix micro-discs. The reseeded cells were highly motile and tended to form aggregates with the same but segregate from or mix with distinct cell types, supporting that 2D gastruloid system exhibits evolutionarily conserved sorting behaviors. In particular, ectodermal cells segregated from endodermal and extraembryonic cells but mixed with mesodermal cells. These results demonstrate that 2D gastruloid system models specification of germ layers and extraembryonic cell types, temporal order and differentiation trajectories of germ layer emergence, and signaling interactions found in early-mid in vivo gastrulation. Dissociated and reseeded gastruloid cells also exhibit conserved cell sorting behaviors. Lastly, this work provides a resource for mining genes and pathways expressed in a stereotyped 2D gastruloid model, common with other species or unique to human gastrulation

Influence de la distribution des classes et évaluation en apprentissage profond‎ - Application à la détection du cancer sur des images histologiques

Le cancer est une maladie mortelle considérée comme la deuxième cause de décès. Toute avancée dans le diagnostic et la détection du cancer est donc cruciale pour sauver des vies. L’analyse d’images histologiques - également appelées Whole Slide Images (WSI) - est considérée comme la référence dans le diagnostic et l’étude du stade du cancer. L’analyse manuelle de ces images par les pathologistes reste le principal processus de diagnostic. Il prend du temps, est laborieux, sujet aux erreurs et difficile à évaluer de manière reproductible. Les techniques de diagnostic assisté par ordinateur peuvent aider les pathologistes dans leur travail. Les techniques d’apprentissage automatique, en particulier les algorithmes d’apprentissage profond, tels que les réseaux de neurones convolutifs (CNN), sont largement utilisés dans divers domaines dont l’analyse d’images. Le succès des modèles CNN dépend cependant de plusieurs hyper-paramètres, tels que l’architecture du réseau, les données utilisées pour entraîner le modèle et la distribution des données d’entraînement. A notre connaissance, parmi les hyper-paramètres, la distribution des données d’entraînement n’est pas encore étudiée dans la littérature pour les données WSI, alors qu’elle pourrait être l’un des critères les plus importants pour réguler les performances du modèle. L’un des objectifs de cette thèse est d’étudier en profondeur l’impact de la répartition des classes tant au stade de l’apprentissage qu’au stade du test ou de la prévision. Un autre objectif de cette thèse est lié à l’évaluation au sens large. Nous avons étudié des moyens d’évaluer les résultats qui correspondent davantage aux objectifs du pathologiste et résolvent les problèmes des métriques actuelles qui souffrent de leur incapacité à distinguer les modèles dans de nombreux cas, manquent d’informations concernant les fausses prédictions et sont optimistes dans le cas de données déséquilibrées. Considérant à la fois la distribution des classes et l’évaluation de la détection du cancer à partir des WSI, les contributions spécifiques de cette thèse sont les suivantes : la première contribution principale de cette thèse est d’étudier l’efficacité de la distribution équilibrée dans la détection automatique du cancer qui est utilisée dans de nombreuses études. Nous proposons une approche systématique pour analyser la distribution des classes des données WSI dans l’ensemble d’apprentissage, pour proposer différentes hypothèses sur la distribution des classes et tester ces hypothèses en utilisant trois ensembles de données et deux architectures CNN, le réseau U-net et le réseau convolutif équivariant de groupe (G-CNN). Nous introduisons également une méthode d’évaluation basée sur les régions de l’image alternative à la méthode habituelle basée sur les pixels. Elle permet d’obtenir une meilleure correspondance par rapport à la façon dont un pathologiste vérifie les images. Nous avons constaté que la distribution équilibrée n’est pas optimale pour l’entrainement d’un CNN, et qu’avec la distribution biaisée par classe, il est possible d’infléchir le modèle vers la précision souhaitée (par exemple, vers le rappel ou la précision). Ces résultats constituent une avancée pour comprendre le comportement du modèle vis-à-vis des différentes distributions de classes dans l’ensemble d’apprentissage. La deuxième contribution principale de cette thèse est de développer une représentation continue basée sur un seuil des courbes de précision et de rappel (PR-T) comme alternative aux courbes de caractéristiques de fonctionnement du récepteur (ROC) et de précision-rappel (PR), les métriques d’évaluation usuelles en classification binaire. De plus, nous avons développé des algorithmes de bout en bout pour calculer la courbe PR moyenne et la moyenne de l’aire sous la courbe (PR-AUC).Cancer is a fatal disease considered the second leading cause of death. Any advances in diagnosis and detection of cancer are thus crucial to save lives. The analysis of histological images -also known as Whole Slide Images (WSIs)-is considered as the gold standard in cancer diagnosis and staging. The pathologists’ manual analysis of WSIs is still the primary diagnosis process. It is time-consuming, laborious, prone to error, and difficult to grade in a reproducible manner. Computer-aided diagnosis techniques can assist pathologists in their workflow. Machine learning techniques, specifically deep learning algorithms, such as Convolutional Neural Networks (CNNs), are widely used in various domains that involve image analysis. The success of CNN models, however, depends on several hyper-parameter settings, such as the network architecture, the data used to train the model, and the class distribution of the training data. To the best of our knowledge, among the hyper-parameters, the class distribution of the training data is not studied yet in the literature for the WSI data, while it could be one of the most important criteria to regulate the model performance. One of the aims of this thesis is to study in-depth the impact of class distribution both at the training stage and at the test or forecasting stage. Another aim of this thesis is related to evaluation in a broader sense. We studied ways of evaluating the results that fit more the pathologist’s goals and solve the issues of current metrics that suffer from their incapacity to distinguish models in many cases, lacking information regarding false predictions and being optimistic in the case of imbalanced data. Considering both the class distribution and the evaluation for cancer detection from WSIs, the specific contributions of this thesis areas follows: The first main contribution of this thesis is to investigate the effectiveness of the balanced distribution in automatic cancer detection which is used in many studies. We propose a systematic approach to analyze the class distribution of the WSI data in the training set; put forward different hypotheses on the class distribution and test those hypotheses using three data sets and two CNN architectures, the U-net and the group equivariant convolutional network (G-CNN). We also introduce a patch-based (i.e., image region-based) evaluation method over the usual pixel-based one to obtain a better match in comparison to how a pathologist checks images. We found that the balanced distribution is not optimal for CNN training for cancer detection from WSI, rather with the class-biased distribution, it is possible to inflect the model toward the desired accuracy (e.g., toward recall or precision). These results are a step forward to understand the model behavior towards the different distributions of classes in the training set. The second main contribution of this thesis is to develop a continuous threshold-based representation of precision and recall (PR-T) curves as an alternative to the Receiver Operating Characteristics (ROC) and Precision-Recall (PR) curves, the state-of-the-art evaluation metrics in binary classification as is cancer detection. Additionally, we developed end-to-end algorithms to compute the mean PR curve and the mean Area Under the Curve (PR-AUC)

Influence de la distribution des classes et évaluation en apprentissage profond‎ - Application à la détection du cancer sur des images histologiques

Le cancer est une maladie mortelle considérée comme la deuxième cause de décès. Toute avancée dans le diagnostic et la détection du cancer est donc cruciale pour sauver des vies. L’analyse d’images histologiques - également appelées Whole Slide Images (WSI) - est considérée comme la référence dans le diagnostic et l’étude du stade du cancer. L’analyse manuelle de ces images par les pathologistes reste le principal processus de diagnostic. Il prend du temps, est laborieux, sujet aux erreurs et difficile à évaluer de manière reproductible. Les techniques de diagnostic assisté par ordinateur peuvent aider les pathologistes dans leur travail. Les techniques d’apprentissage automatique, en particulier les algorithmes d’apprentissage profond, tels que les réseaux de neurones convolutifs (CNN), sont largement utilisés dans divers domaines dont l’analyse d’images. Le succès des modèles CNN dépend cependant de plusieurs hyper-paramètres, tels que l’architecture du réseau, les données utilisées pour entraîner le modèle et la distribution des données d’entraînement. A notre connaissance, parmi les hyper-paramètres, la distribution des données d’entraînement n’est pas encore étudiée dans la littérature pour les données WSI, alors qu’elle pourrait être l’un des critères les plus importants pour réguler les performances du modèle. L’un des objectifs de cette thèse est d’étudier en profondeur l’impact de la répartition des classes tant au stade de l’apprentissage qu’au stade du test ou de la prévision. Un autre objectif de cette thèse est lié à l’évaluation au sens large. Nous avons étudié des moyens d’évaluer les résultats qui correspondent davantage aux objectifs du pathologiste et résolvent les problèmes des métriques actuelles qui souffrent de leur incapacité à distinguer les modèles dans de nombreux cas, manquent d’informations concernant les fausses prédictions et sont optimistes dans le cas de données déséquilibrées. Considérant à la fois la distribution des classes et l’évaluation de la détection du cancer à partir des WSI, les contributions spécifiques de cette thèse sont les suivantes : la première contribution principale de cette thèse est d’étudier l’efficacité de la distribution équilibrée dans la détection automatique du cancer qui est utilisée dans de nombreuses études. Nous proposons une approche systématique pour analyser la distribution des classes des données WSI dans l’ensemble d’apprentissage, pour proposer différentes hypothèses sur la distribution des classes et tester ces hypothèses en utilisant trois ensembles de données et deux architectures CNN, le réseau U-net et le réseau convolutif équivariant de groupe (G-CNN). Nous introduisons également une méthode d’évaluation basée sur les régions de l’image alternative à la méthode habituelle basée sur les pixels. Elle permet d’obtenir une meilleure correspondance par rapport à la façon dont un pathologiste vérifie les images. Nous avons constaté que la distribution équilibrée n’est pas optimale pour l’entrainement d’un CNN, et qu’avec la distribution biaisée par classe, il est possible d’infléchir le modèle vers la précision souhaitée (par exemple, vers le rappel ou la précision). Ces résultats constituent une avancée pour comprendre le comportement du modèle vis-à-vis des différentes distributions de classes dans l’ensemble d’apprentissage. La deuxième contribution principale de cette thèse est de développer une représentation continue basée sur un seuil des courbes de précision et de rappel (PR-T) comme alternative aux courbes de caractéristiques de fonctionnement du récepteur (ROC) et de précision-rappel (PR), les métriques d’évaluation usuelles en classification binaire. De plus, nous avons développé des algorithmes de bout en bout pour calculer la courbe PR moyenne et la moyenne de l’aire sous la courbe (PR-AUC).Cancer is a fatal disease considered the second leading cause of death. Any advances in diagnosis and detection of cancer are thus crucial to save lives. The analysis of histological images -also known as Whole Slide Images (WSIs)-is considered as the gold standard in cancer diagnosis and staging. The pathologists’ manual analysis of WSIs is still the primary diagnosis process. It is time-consuming, laborious, prone to error, and difficult to grade in a reproducible manner. Computer-aided diagnosis techniques can assist pathologists in their workflow. Machine learning techniques, specifically deep learning algorithms, such as Convolutional Neural Networks (CNNs), are widely used in various domains that involve image analysis. The success of CNN models, however, depends on several hyper-parameter settings, such as the network architecture, the data used to train the model, and the class distribution of the training data. To the best of our knowledge, among the hyper-parameters, the class distribution of the training data is not studied yet in the literature for the WSI data, while it could be one of the most important criteria to regulate the model performance. One of the aims of this thesis is to study in-depth the impact of class distribution both at the training stage and at the test or forecasting stage. Another aim of this thesis is related to evaluation in a broader sense. We studied ways of evaluating the results that fit more the pathologist’s goals and solve the issues of current metrics that suffer from their incapacity to distinguish models in many cases, lacking information regarding false predictions and being optimistic in the case of imbalanced data. Considering both the class distribution and the evaluation for cancer detection from WSIs, the specific contributions of this thesis areas follows: The first main contribution of this thesis is to investigate the effectiveness of the balanced distribution in automatic cancer detection which is used in many studies. We propose a systematic approach to analyze the class distribution of the WSI data in the training set; put forward different hypotheses on the class distribution and test those hypotheses using three data sets and two CNN architectures, the U-net and the group equivariant convolutional network (G-CNN). We also introduce a patch-based (i.e., image region-based) evaluation method over the usual pixel-based one to obtain a better match in comparison to how a pathologist checks images. We found that the balanced distribution is not optimal for CNN training for cancer detection from WSI, rather with the class-biased distribution, it is possible to inflect the model toward the desired accuracy (e.g., toward recall or precision). These results are a step forward to understand the model behavior towards the different distributions of classes in the training set. The second main contribution of this thesis is to develop a continuous threshold-based representation of precision and recall (PR-T) curves as an alternative to the Receiver Operating Characteristics (ROC) and Precision-Recall (PR) curves, the state-of-the-art evaluation metrics in binary classification as is cancer detection. Additionally, we developed end-to-end algorithms to compute the mean PR curve and the mean Area Under the Curve (PR-AUC)

The Effectiveness of Transfer Learning Systems on Medical Images

Deep neural networks have revolutionized the performances of many machine learning tasks such as medical image classification and segmentation. Current deep learning (DL) algorithms, specifically convolutional neural networks are increasingly becoming the methodological choice for most medical image analysis. However, training these deep neural networks requires high computational resources and very large amounts of labeled data which is often expensive and laborious. Meanwhile, recent studies have shown the transfer learning (TL) paradigm as an attractive choice in providing promising solutions to challenges of shortage in the availability of labeled medical images. Accordingly, TL enables us to leverage the knowledge learned from related data to solve a new problem. The objective of this dissertation is to examine the effectiveness of TL systems on medical images. First, a comprehensive systematic literature review was performed to provide an up-to-date status of TL systems on medical images. Specifically, we proposed a novel conceptual framework to organize the review. Second, a novel DL network was pretrained on natural images and utilized to evaluate the effectiveness of TL on a very large medical image dataset, specifically Chest X-rays images. Lastly, domain adaptation using an autoencoder was evaluated on the medical image dataset and the results confirmed the effectiveness of TL through fine-tuning strategies. We make several contributions to TL systems on medical image analysis: Firstly, we present a novel survey of TL on medical images and propose a new conceptual framework to organize the findings. Secondly, we propose a novel DL architecture to improve learned representations of medical images while mitigating the problem of vanishing gradients. Additionally, we identified the optimal cut-off layer (OCL) that provided the best model performance. We found that the higher layers in the proposed deep model give a better feature representation of our medical image task. Finally, we analyzed the effect of domain adaptation by fine-tuning an autoencoder on our medical images and provide theoretical contributions on the application of the transductive TL approach. The contributions herein reveal several research gaps to motivate future research and contribute to the body of literature in this active research area of TL systems on medical image analysis