Deep Representation Learning with Limited Data for Biomedical Image Synthesis, Segmentation, and Detection

Biomedical imaging requires accurate expert annotation and interpretation that can aid medical staff and clinicians in automating differential diagnosis and solving underlying health conditions. With the advent of Deep learning, it has become a standard for reaching expert-level performance in non-invasive biomedical imaging tasks by training with large image datasets. However, with the need for large publicly available datasets, training a deep learning model to learn intrinsic representations becomes harder. Representation learning with limited data has introduced new learning techniques, such as Generative Adversarial Networks, Semi-supervised Learning, and Self-supervised Learning, that can be applied to various biomedical applications. For example, ophthalmologists use color funduscopy (CF) and fluorescein angiography (FA) to diagnose retinal degenerative diseases. However, fluorescein angiography requires injecting a dye, which can create adverse reactions in the patients. So, to alleviate this, a non-invasive technique needs to be developed that can translate fluorescein angiography from fundus images. Similarly, color funduscopy and optical coherence tomography (OCT) are also utilized to semantically segment the vasculature and fluid build-up in spatial and volumetric retinal imaging, which can help with the future prognosis of diseases. Although many automated techniques have been proposed for medical image segmentation, the main drawback is the model's precision in pixel-wise predictions. Another critical challenge in the biomedical imaging field is accurately segmenting and quantifying dynamic behaviors of calcium signals in cells. Calcium imaging is a widely utilized approach to studying subcellular calcium activity and cell function; however, large datasets have yielded a profound need for fast, accurate, and standardized analyses of calcium signals. For example, image sequences from calcium signals in colonic pacemaker cells ICC (Interstitial cells of Cajal) suffer from motion artifacts and high periodic and sensor noise, making it difficult to accurately segment and quantify calcium signal events. Moreover, it is time-consuming and tedious to annotate such a large volume of calcium image stacks or videos and extract their associated spatiotemporal maps. To address these problems, we propose various deep representation learning architectures that utilize limited labels and annotations to address the critical challenges in these biomedical applications. To this end, we detail our proposed semi-supervised, generative adversarial networks and transformer-based architectures for individual learning tasks such as retinal image-to-image translation, vessel and fluid segmentation from fundus and OCT images, breast micro-mass segmentation, and sub-cellular calcium events tracking from videos and spatiotemporal map quantification. We also illustrate two multi-modal multi-task learning frameworks with applications that can be extended to other domains of biomedical applications. The main idea is to incorporate each of these as individual modules to our proposed multi-modal frameworks to solve the existing challenges with 1) Fluorescein angiography synthesis, 2) Retinal vessel and fluid segmentation, 3) Breast micro-mass segmentation, and 4) Dynamic quantification of calcium imaging datasets

Deep Learning Methods for Detection and Tracking of Particles in Fluorescence Microscopy Images

Studying the dynamics of sub-cellular structures such as receptors, filaments, and vesicles is a prerequisite for investigating cellular processes at the molecular level. In addition, it is important to characterize the dynamic behavior of virus structures to gain a better understanding of infection mechanisms and to develop novel drugs. To investigate the dynamics of fluorescently labeled sub-cellular and viral structures, time-lapse fluorescence microscopy is the most often used imaging technique. Due to the limited spatial resolution of microscopes caused by diffraction, these very small structures appear as bright, blurred spots, denoted as particles, in microscopy images. To draw statistically meaningful biological conclusions, a large number of such particles need to be analyzed. However, since manual analysis of fluorescent particles is very time consuming, fully automated computer-based methods are indispensable. We introduce novel deep learning methods for detection and tracking of multiple particles in fluorescence microscopy images. We propose a particle detection method based on a convolutional neural network which performs image-to-image mapping by density map regression and uses the adaptive wing loss. For particle tracking, we present a recurrent neural network that exploits past and future information in both forward and backward direction. Assignment probabilities across multiple detections as well as the probabilities for missing detections are computed jointly. To resolve tracking ambiguities using future information, several track hypotheses are propagated to later time points. In addition, we developed a novel probabilistic deep learning method for particle tracking, which is based on a recurrent neural network mimicking classical Bayesian filtering. The method includes both aleatoric and epistemic uncertainty, and provides valuable information about the reliability of the computed trajectories. Short and long-term temporal dependencies of individual object dynamics are exploited for state prediction, and assigned detections are used to update the predicted states. Moreover, we developed a convolutional Long Short-Term Memory neural network for combined particle tracking and colocalization analysis in two-channel microscopy image sequences. The network determines colocalization probabilities, and colocalization information is exploited to improve tracking. Short and long-term temporal dependencies of object motion as well as image intensities are taken into account to compute assignment probabilities jointly across multiple detections. We also introduce a deep learning method for probabilistic particle detection and tracking. For particle detection, temporal information is integrated to regress a density map and determine sub-pixel particle positions. For tracking, a fully Bayesian neural network is presented that mimics classical Bayesian filtering and takes into account both aleatoric and epistemic uncertainty. Uncertainty information of individual particle detections is considered. Network training for the developed deep learning-based particle tracking methods relies only on synthetic data, avoiding the need of time-consuming manual annotation. We performed an extensive evaluation of our methods based on image data of the Particle Tracking Challenge as well as on fluorescence microscopy images displaying virus proteins of HCV and HIV, chromatin structures, and cell-surface receptors. It turned out that the methods outperform previous methods

Efficient Learning Framework for Training Deep Learning Models with Limited Supervision

In recent years, deep learning has shown tremendous success in different applications, however these modes mostly need a large labeled dataset for training their parameters. In this work, we aim to explore the potentials of efficient learning frameworks for training deep models on different problems in the case of limited supervision or noisy labels. For the image clustering problem, we introduce a new deep convolutional autoencoder with an unsupervised learning framework. We employ a relative entropy minimization as the clustering objective regularized by the frequency of cluster assignments and a reconstruction loss. In the case of noisy labels obtained by crowdsourcing platforms, we proposed a novel deep hybrid model for sentiment analysis of text data like tweets based on noisy crowd labels. The proposed model consists of a crowdsourcing aggregation model and a deep text autoencoder. We combine these sub-models based on a probabilistic framework rather than a heuristic way, and derive an efficient optimization algorithm to jointly solve the corresponding problem. In order to improve the performance of unsupervised deep hash functions on image similarity search in big datasets, we adopt generative adversarial networks to propose a new deep image retrieval model, where the adversarial loss is employed as a data-dependent regularization in our objective function. We also introduce a balanced self-paced learning algorithm for training a GAN-based model for image clustering, where the input samples are gradually included into training from easy to difficult, while the diversity of selected samples from all clusters are also considered. In addition, we explore adopting discriminative approaches for unsupervised visual representation learning rather than the generative algorithms, such as maximizing the mutual information between an input image and its representation and a contrastive loss for decreasing the distance between the representations of original and augmented image data

Graph Neural Networks for Molecules

Graph neural networks (GNNs), which are capable of learning representations from graphical data, are naturally suitable for modeling molecular systems. This review introduces GNNs and their various applications for small organic molecules. GNNs rely on message-passing operations, a generic yet powerful framework, to update node features iteratively. Many researches design GNN architectures to effectively learn topological information of 2D molecule graphs as well as geometric information of 3D molecular systems. GNNs have been implemented in a wide variety of molecular applications, including molecular property prediction, molecular scoring and docking, molecular optimization and de novo generation, molecular dynamics simulation, etc. Besides, the review also summarizes the recent development of self-supervised learning for molecules with GNNs.Comment: A chapter for the book "Machine Learning in Molecular Sciences". 31 pages, 4 figure

STED Nanoscopy to Illuminate New Avenues in Cancer Research – From Live Cell Staining and Direct Imaging to Decisive Preclinical Insights for Diagnosis and Therapy

Molecular imaging is established as an indispensable tool in various areas of cancer research, ranging from basic cancer biology and preclinical research to clinical trials and medical practice. In particular, the field of fluorescence imaging has experienced exceptional progress during the last three decades with the development of various in vivo technologies. Within this field, fluorescence microscopy is primarily of experimental use since it is especially qualified for addressing the fundamental questions of molecular oncology. As stimulated emission depletion (STED) nanoscopy combines the highest spatial and temporal resolutions with live specimen compatibility, it is best-suited for real-time investigations of the differences in the molecular machineries of malignant and normal cells to eventually translate the acquired knowledge into increased diagnostic and therapeutic efficacy. This thesis presents the application of STED nanoscopy to two acute topics in cancer research of direct or indirect clinical interest. The first project has investigated the structure of telomeres, the ends of the linear eukaryotic chromosomes, in intact human cells at the nanoscale. To protect genome integrity, a telomere can mask the chromosome end by folding back and sequestering its single-stranded 3’-overhang in an upstream part of the double-stranded DNA repeat region. The formed t-loop structure has so far only been visualized by electron microscopy and fluorescence nanoscopy with cross-linked mammalian telomeric DNA after disruption of cell nuclei and spreading. For the first time, this work demonstrates the existence of t-loops within their endogenous nuclear environment in intact human cells. The identification of further telomere conformations has laid the groundwork for distinguishing cancerous cells that use different telomere maintenance mechanisms based on their individual telomere populations by a combined STED nanoscopy and deep learning approach. The population difference was essentially attributed to the promyelocytic leukemia (PML) protein that significantly perturbs the organization of a subpopulation of telomeres towards an open conformation in cancer cells that employ a telomerase-independent, alternative telomere lengthening mechanism. Elucidating the nanoscale topology of telomeres and associated proteins within the nucleus has provided new insight into telomere structure-function relationships relevant for understanding the deregulation of telomere maintenance in cancer cells. After understanding the molecular foundations, this newly gained knowledge can be exploited to develop novel or refined diagnostic and treatment strategies. The second project has characterized the intracellular distribution of recently developed prostate cancer tracers. These novel prostate-specific membrane antigen (PSMA) inhibitors have revolutionized the treatment regimen of prostate cancer by enabling targeted imaging and therapy approaches. However, the exact internalization mechanism and the subcellular fate of these tracers have remained elusive. By combining STED nanoscopy with a newly developed non-standard live cell staining protocol, this work confirmed cell surface clustering of the targeted membrane antigen upon PSMA inhibitor binding, subsequent clathrin-dependent endocytosis and endosomal trafficking of the antigen-inhibitor complex. PSMA inhibitors accumulate in prostate cancer cells at clinically relevant time points, but strikingly and in contrast to the targeted antigen itself, they eventually distribute homogenously in the cytosol. This project has revealed the subcellular fate of PSMA/PSMA inhibitor complexes for the first time and provides crucial knowledge for the future application of these tracers including the development of new strategies in the field of prostate cancer diagnostics and therapeutics. Relying on the photostability and biocompatibility of the applied fluorophores, the performance of live cell STED nanoscopy in the field of cancer research is boosted by the development of improved fluorophores. The third project in this thesis introduces a biocompatible, small molecule near-infrared dye suitable for live cell STED imaging. By the application of a halogen dance rearrangement, a dihalogenated fluorinatable pyridinyl rhodamine could be synthesized at high yield. The option of subsequent radiolabeling combined with excellent optical properties and a non-toxic profile renders this dye an appropriate candidate for medical and bioimaging applications. Providing an intrinsic and highly specific mitochondrial targeting ability, the radiolabeled analogue is suggested as a vehicle for multimodal (positron emission tomography and optical imaging) medical imaging of mitochondria for cancer diagnosis and therapeutic approaches in patients and biopsy tissue. The absence of cytotoxicity is not only a crucial prerequisite for clinically used fluorophores. To guarantee the generation of meaningful data mirroring biological reality, the absence of cytotoxicity is likewise a decisive property of dyes applied in live cell STED nanoscopy. The fourth project in this thesis proposes a universal approach for cytotoxicity testing based on characterizing the influence of the compound of interest on the proliferation behavior of human cell lines using digital holographic cytometry. By applying this approach to recently developed live cell STED compatible dyes, pronounced cytotoxic effects could be excluded. Looking more closely, some of the tested dyes slightly altered cell proliferation, so this project provides guidance on the right choice of dye for the least invasive live cell STED experiments. Ultimately, live cell STED data should be exploited to extract as much biological information as possible. However, some information might be partially hidden by image degradation due the dynamics of living samples and the deliberate choice of rather conservative imaging parameters in order to preserve sample viability. The fifth project in this thesis presents a novel image restoration method in a Bayesian framework that simultaneously performs deconvolution, denoising as well as super-resolution, to restore images suffering from noise with mixed Poisson-Gaussian statistics. Established deconvolution or denoising methods that consider only one type of noise generally do not perform well on images degraded significantly by mixed noise. The newly introduced method was validated with live cell STED telomere data proving that the method can compete with state-of-the-art approaches. Taken together, this thesis demonstrates the value of an integrated approach for STED nanoscopy imaging studies. A coordinated workflow including sample preparation, image acquisition and data analysis provided a reliable platform for deriving meaningful conclusions for current questions in the field of cancer research. Moreover, this thesis emphasizes the strength of iteratively adapting the individual components in the operational chain and it particularly points towards those components that, if further improved, optimize the significance of the final results rendering live cell STED nanoscopy even more powerful

Mixed Selectivity via Unsupervised Learning in Neural Networks

Mixed selectivity characterises neurons that simultaneously respond to different input stimuli. Neurons with mixed selectivity have been observed in multiple brain regions, and are hypothesised to play important roles in neural computation. Recently, both experimental and theoretical work demonstrated the importance of mixed selectivity in context-dependent decision tasks. This thesis extends existing theoretical work on mixed selectivity, arguing for a general and statistical role of mixed selectivity in learning complex dependencies of input stimuli. This role can be motivated from unsupervised learning of generative models, and is exhibited in increased mutual information between the stimuli and their neural representation. This argument is supported empirically using simulation of a sequence disambiguation task that incorporated key aspects of related behaviour experiments. Mixed selectivity neurons that resembled hippocampal place cells were emerged from models optimised only for behaviour. To understand these results, as well as to generalise the findings to a wider range of computations, I provided a formal connection between learning robust models and mixed selectivity