Perplexity-free Parametric t-SNE

The t-distributed Stochastic Neighbor Embedding (t-SNE) algorithm is a ubiquitously employed dimensionality reduction (DR) method. Its non-parametric nature and impressive efficacy motivated its parametric extension. It is however bounded to a user-defined perplexity parameter, restricting its DR quality compared to recently developed multi-scale perplexity-free approaches. This paper hence proposes a multi-scale parametric t-SNE scheme, relieved from the perplexity tuning and with a deep neural network implementing the mapping. It produces reliable embeddings with out-of-sample extensions, competitive with the best perplexity adjustments in terms of neighborhood preservation on multiple data sets.Comment: ESANN 2020 proceedings, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. Online event, 2-4 October 2020, i6doc.com publ., ISBN 978-2-87587-074-2. Available from http://www.i6doc.com/en

Theoretical Foundations of t-SNE for Visualizing High-Dimensional Clustered Data

This paper investigates the theoretical foundations of the t-distributed stochastic neighbor embedding (t-SNE) algorithm, a popular nonlinear dimension reduction and data visualization method. A novel theoretical framework for the analysis of t-SNE based on the gradient descent approach is presented. For the early exaggeration stage of t-SNE, we show its asymptotic equivalence to power iterations based on the underlying graph Laplacian, characterize its limiting behavior, and uncover its deep connection to Laplacian spectral clustering, and fundamental principles including early stopping as implicit regularization. The results explain the intrinsic mechanism and the empirical benefits of such a computational strategy. For the embedding stage of t-SNE, we characterize the kinematics of the low-dimensional map throughout the iterations, and identify an amplification phase, featuring the intercluster repulsion and the expansive behavior of the low-dimensional map, and a stabilization phase. The general theory explains the fast convergence rate and the exceptional empirical performance of t-SNE for visualizing clustered data, brings forth interpretations of the t-SNE visualizations, and provides theoretical guidance for applying t-SNE and selecting its tuning parameters in various applications.Comment: Accepted by Journal of Machine Learning Researc

Neural recommender models for sparse and skewed behavioral data

Modern online platforms offer recommendations and personalized search and services to a large and diverse user base while still aiming to acquaint users with the broader community on the platform. Prior work backed by large volumes of user data has shown that user retention is reliant on catering to their specific eccentric tastes, in addition to providing them popular services or content on the platform. Long-tailed distributions are a fundamental characteristic of human activity, owing to the bursty nature of human attention. As a result, we often observe skew in data facets that involve human interaction. While there are superficial similarities to Zipf's law in textual data and other domains, the challenges with user data extend further. Individual words may have skewed frequencies in the corpus, but the long-tail words by themselves do not significantly impact downstream text-mining tasks. On the contrary, while sparse users (a majority on most online platforms) contribute little to the training data, they are equally crucial at inference time. Perhaps more so, since they are likely to churn. In this thesis, we study platforms and applications that elicit user participation in rich social settings incorporating user-generated content, user-user interaction, and other modalities of user participation and data generation. For instance, users on the Yelp review platform participate in a follower-followee network and also create and interact with review text (two modalities of user data). Similarly, community question-answer (CQA) platforms incorporate user interaction and collaboratively authored content over diverse domains and discussion threads. Since user participation is multimodal, we develop generalizable abstractions beyond any single data modality. Specifically, we aim to address the distributional mismatch that occurs with user data independent of dataset specifics; While a minority of the users generates most training samples, it is insufficient only to learn the preferences of this subset of users. As a result, the data's overall skew and individual users' sparsity are closely interlinked: sparse users with uncommon preferences are under-represented. Thus, we propose to treat these problems jointly with a skew-aware grouping mechanism that iteratively sharpens the identification of preference groups within the user population. As a result, we improve user characterization; content recommendation and activity prediction (+6-22% AUC, +6-43% AUC, +12-25% RMSE over state-of-the-art baselines), primarily for users with sparse activity. The size of the item or content inventories compounds the skew problem. Recommendation models can achieve very high aggregate performance while recommending only a tiny proportion of the inventory (as little as 5%) to users. We propose a data-driven solution guided by the aggregate co-occurrence information across items in the dataset. We specifically note that different co-occurrences are not equally significant; For example, some co-occurring items are easily substituted while others are not. We develop a self-supervised learning framework where the aggregate co-occurrences guide the recommendation problem while providing room to learn these variations among the item associations. As a result, we improve coverage to ~100% (up from 5%) of the inventory and increase long-tail item recall up to 25%. We also note that the skew and sparsity problems repeat across data modalities. For instance, social interactions and review content both exhibit aggregate skew, although individual users who actively generate reviews may not participate socially and vice-versa. It is necessary to differentially weight and merge different data sources for each user towards inference tasks in such cases. We show that the problem is inherently adversarial since the user participation modalities compete to describe a user accurately. We develop a framework to unify these representations while algorithmically tackling mode collapse, a well-known pitfall with adversarial models. A more challenging but important instantiation of sparsity is the few-shot setting or cross-domain setting. We may only have a single or a few interactions for users or items in the sparse domains or partitions. We show that contextualizing user-item interactions helps us infer behavioral invariants in the dense domain, allowing us to correlate sparse participants to their active counterparts (resulting in 3x faster training, ~19% recall gains in multi-domain settings). Finally, we consider the multi-task setting, where the platform incorporates multiple distinct recommendations and prediction tasks for each user. A single-user representation is insufficient for users who exhibit different preferences along each dimension. At the same time, it is counter-productive to handle correlated prediction or inference tasks in isolation. We develop a multi-faceted representation approach grounded on residual learning with heterogeneous knowledge graph representations, which provides us an expressive data representation for specialized domains and applications with multimodal user data. We achieve knowledge sharing by unifying task-independent and task-specific representations of each entity with a unified knowledge graph framework. In each chapter, we also discuss and demonstrate how the proposed frameworks directly incorporate a wide range of gradient-optimizable recommendation and behavior models, maximizing their applicability and pertinence to user-centered inference tasks and platforms

Graph-based Methods for Visualization and Clustering

The amount of data that we produce and consume is larger than it has been at any point in the history of mankind, and it keeps growing exponentially. All this information, gathered in overwhelming volumes, often comes with two problematic characteristics: it is complex and deprived of semantical context. A common step to address those issues is to embed raw data in lower dimensions, by finding a mapping which preserves the similarity between data points from their original space to a new one. Measuring similarity between large sets of high-dimensional objects is, however, problematic for two main reasons: first, high-dimensional points are subject to the curse of dimensionality and second, the number of pairwise distances between points is quadratic with respect to the amount of data points. Both problems can be addressed by using nearest neighbours graphs to understand the structure in data. As a matter of fact, most dimensionality reduction methods use similarity matrices that can be interpreted as graph adjacency matrices. Yet, despite recent progresses, dimensionality reduction is still very challenging when applied to very large datasets. Indeed, although recent methods specifically address the problem of scaleability, processing datasets of millions of elements remain a very lengthy process. In this thesis, we propose new contributions which address the problem of scaleability using the framework of Graph Signal Processing, which extends traditional signal processing to graphs. We do so motivated by the premise that graphs are well suited to represent the structure of the data. In the first part of this thesis, we look at quantitative measures for the evaluation of dimensionality reduction methods. Using tools from graph theory and Graph Signal Processing, we show that specific characteristics related to quality can be assessed by taking measures on the graph, which indirectly validates the hypothesis relating graph to structure. The second contribution is a new method for a fast eigenspace approximation of the graph Laplacian. Using principles of GSP and random matrices, we show that an approximated eigensubpace can be recovered very efficiently, which be used for fast spectral clustering or visualization. Next, we propose a compressive scheme to accelerate any dimensionality reduction technique. The idea is based on compressive sampling and transductive learning on graphs: after computing the embedding for a small subset of data points, we propagate the information everywhere using transductive inference. The key components of this technique are a good sampling strategy to select the subset and the application of transductive learning on graphs. Finally, we address the problem of over-discriminative feature spaces by proposing a hierarchical clustering structure combined with multi-resolution graphs. Using efficient coarsening and refinement procedures on this structure, we show that dimensionality reduction algorithms can be run on intermediate levels and up-sampled to all points leading to a very fast dimensionality reduction method. For all contributions, we provide extensive experiments on both synthetic and natural datasets, including large-scale problems. This allows us to show the pertinence of our models and the validity of our proposed algorithms. Following reproducible principles, we provide everything needed to repeat the examples and the experiments presented in this work

Training deep retrieval models with noisy datasets

In this thesis we study loss functions that allow to train Convolutional Neural Networks (CNNs) under noisy datasets for the particular task of Content- Based Image Retrieval (CBIR). In particular, we propose two novel losses to fit models that generate global image representations. First, a Soft-Matching (SM) loss, exploiting both image content and meta data, is used to specialized general CNNs to particular cities or regions using weakly annotated datasets. Second, a Bag Exponential (BE) loss inspired by the Multiple Instance Learning (MIL) framework is employed to train CNNs for CBIR under noisy datasets. The first part of the thesis introduces a novel training framework that, relying on image content and meta data, learns location-adapted deep models that provide fine-tuned image descriptors for specific visual contents. Our networks, which start from a baseline model originally learned for a different task, are specialized using a custom pairwise loss function, our proposed SM loss, that uses weak labels based on image content and meta data. The experimental results show that the proposed location-adapted CNNs achieve an improvement of up to a 55% over the baseline networks on a landmark discovery task. This implies that the models successfully learn the visual clues and peculiarities of the region for which they are trained, and generate image descriptors that are better location-adapted. In addition, for those landmarks that are not present on the training set or even other cities, our proposed models perform at least as well as the baseline network, which indicates a good resilience against overfitting. The second part of the thesis introduces the BE Loss function to train CNNs for image retrieval borrowing inspiration from the MIL framework. The loss combines the use of an exponential function acting as a soft margin, and a MILbased mechanism working with bags of positive and negative pairs of images. The method allows to train deep retrieval networks under noisy datasets, by weighing the influence of the different samples at loss level, which increases the performance of the generated global descriptors. The rationale behind the improvement is that we are handling noise in an end-to-end manner and, therefore, avoiding its negative influence as well as the unintentional biases due to fixed pre-processing cleaning procedures. In addition, our method is general enough to suit other scenarios requiring different weights for the training instances (e.g. boosting the influence of hard positives during training). The proposed bag exponential function can bee seen as a back door to guide the learning process according to a certain objective in a end-to-end manner, allowing the model to approach such an objective smoothly and progressively. Our results show that our loss allows CNN-based retrieval systems to be trained with noisy training sets and achieve state-of-the-art performance. Furthermore, we have found that it is better to use training sets that are highly correlated with the final task, even if they are noisy, than training with a clean set that is only weakly related with the topic at hand. From our point of view, this result represents a big leap in the applicability of retrieval systems and help to reduce the effort needed to set-up new CBIR applications: e.g. by allowing a fast automatic generation of noisy training datasets and then using our bag exponential loss to deal with noise. Moreover, we also consider that this result opens a new line of research for CNN-based image retrieval: let the models decide not only on the best features to solve the task but also on the most relevant samples to do it.Programa de Doctorado en Multimedia y Comunicaciones por la Universidad Carlos III de Madrid y la Universidad Rey Juan CarlosPresidente: Luis Salgado Álvarez de Sotomayor.- Secretario: Pablos Martínez Olmos.- Vocal: Ernest Valveny Llobe

Text-basierte Ähnlichkeitssuche zur Treffer- und Leitstruktur-Identifizierung

This work investigated the applicability of global pairwise sequence alignment to the detection of functional analogues in virtual screening. This variant of sequence comparison was developed for the identification of homologue proteins based on amino acid or nucleotide sequences. Because of the significant differences between biopolymers and small molecules several aspects of this approach for sequence comparison had to be adapted. All proposed concepts were implemented as the ‘Pharmacophore Alignment Search Tool’ (PhAST) and evaluated in retrospective experiments on the COBRA dataset in version 6.1. The aim to identify functional analogues raised the necessity for identification and classification of functional properties in molecular structures. This was realized by fragment-based atom-typing, where one out of nine functional properties was assigned to each non-hydrogen atom in a structure. These properties were pre-assigned to atoms in the fragments. Whenever a fragment matched a substructure in a molecule, the assigned properties were transferred from fragment atoms to structure atoms. Each functional property was represented by exactly one symbol. Unlike amino acid or nucleotide sequences, small drug-like molecules contain branches and cycles. This was a major obstacle in the application of sequence alignment to virtual screening, since this technique can only be applied to linear sequences of symbols. The best linearization technique was shown to be Minimum Volume Embedding. To the best of knowledge, this work represents the first application of dimensionality reduction to graph linearization. Sequence alignment relies on a scoring system that rates symbol equivalences (matches) and differences (mismatches) based on functional properties that correspond to rated symbols. Existing scoring schemes are applicable only to amino acids and nucleotides. In this work, scoring schemes for functional properties in drug-like molecules were developed based on property frequencies and isofunctionality judged from chemical experience, pairwise sequence alignments, pairwise kernel-based assignments and stochastic optimization. The scoring system based on property frequencies and isofunctionality proved to be the most powerful (measured in enrichment capability). All developed scoring systems performed superior compared to simple scoring approaches that rate matches and mismatches uniformly. The frameworks proposed for score calculations can be used to guide modifications to the atom-typing in promising directions. The scoring system was further modified to allow for emphasis on particular symbols in a sequence. It was proven that the application of weights to symbols that correspond to key interaction points important to receptor-ligand-interaction significantly improves screening capabilities of PhAST. It was demonstrated that the systematic application of weights to all sequence positions in retrospective experiments can be used for pharmacophore elucidation. A scoring system based on structural instead of functional similarity was investigated and found to be suitable for similarity searches in shape-constrained datasets. Three methods for similarity assessment based on alignments were evaluated: Sequence identity, alignment score and significance. PhAST achieved significantly higher enrichment with alignment scores compared to sequence identity. p-values as significance estimates were calculated in a combination of Marcov Chain Monte Carlo Simulation and Importance Sampling. p-values were adapted to library size in a Bonferroni correction, yielding E-values. A significance threshold of an E-value of 1*10-5 was proposed for the application in prospective screenings. PhAST was compared to state-of-the-art methods for virtual screening. The unweighted version was shown to exhibit comparable enrichment capabilities. Compound rankings obtained with PhAST were proven to be complementary to those of other methods. The application to three-dimensional instead of two-dimensional molecular representations resulted in altered compound rankings without increased enrichment. PhAST was employed in two prospective applications. A screening for non-nucleoside analogue inhibitors of bacterial thymidin kinase yielded a hit with a distinct structural framework but only weak activity. The search for drugs not member of the NSAID (non-steroidal anti-inflammatory drug) class as modulators of gamma-secretase resulted in a potent modulator with clear structural distiction from the reference compound. The calculation of significance estimates, emphasizing on key interactions, the pharmacophore elucidation capabilities and the unique compound rannkings set PhAST apart from other screening techniques.In dieser Arbeit wurde die Anwendbarkeit von paarweisem globalen Sequenzalignment auf das Problem des Molekülsvergleichs im virtuellen Screening untersucht, einem Teilgebiet der computerbasierten Wirkstoffentwicklung. Sequenzalignment wurde zur Identifizierung homologer Proteine entwickelt. Bisher wurde es nur angewendet auf Sequenzen aus Aminosäuren oder Nukleotiden. Aufgrund der Unterschiede zwischen Biopolymeren und wirkstoffartigen Molekülen wurde dieser Ansatz zum Sequenzvergleich modifiziert und auf die konkrete Problemstellung angepasst. Alle vorgestellten und untersuchten Methoden wurden implementiert unter dem Namen ‚Pharmacophore Alignment Search Tool’ (PhAST). Zielsetzung bei der Entwicklung von PhAST war es, die funktionelle Ähnlichkeit zwischen Molekülen zu berechnen. Dafür war es notwendig, einen Ansatz zu implementieren, der den Atomen eines Moleküls funktionelle Eigenschaften zuweist. Dies wurde realisiert durch eine auf Fragmenten basierende Atomtypisierung. Den Atomen einer Sammlung vordefinierter Fragmente wurden nach bestem Wissen und Gewissen Eigenschaften zugewiesen. In jedem Fall, in dem eines der Fragmente als Substruktur eines Moleküls auftrat, wurden die Atomtypisierungen von dem jeweiligen Fragment auf die Atome des Moleküls übertragen. Insgesamt unterscheidet PhAST neun funktionelle Eigenschaften und deren Kombination, wobei jedem Atomtyp genau ein Symbol zugeordnet ist. Im Gegensatz zu Sequenzen von Aminosäuren und Nukleotiden sind wirkstoffartige Moleküle verzweigt, ungerichtet und enthalten Ringeschlüsse. Sequenzalignment ist aber ausschließlich auf lineare Sequenzen anwendbar. Folglich mussten Moleküle mit ihren funktionellen Eigenschaften zunächst in einer linearisierten Form gespeichert werden. Es wurde gezeigt, dass Minimum Volume Embedding die performanteste der getesteten Linearisierungsmethoden war. Nach bestem Wissen und Gewissen wurden in dieser Arbeit zum ersten mal Methoden zur Dimensionsreduktion auf das Problem der kanonischen Indizierung von Graphen angewendet. Zur Berechnung von Sequenzalignments ist ein Bewertungssystem von Equivalenzen und Unterschieden von Symbolen notwendig. Die bestehenden Systeme sind nur anwendbar auf Aminosäuren und Nukleotide. Daher wurde ein Bewertungssystem für Atomeigenschaften nach chemischer Intuition entwickelt, sowie drei automatisierte Methoden, solche Systeme zu berechnen. Das nach chemischer Intuition erstellte Bewertungsschema wurde als den anderen signifikant überlegen identifiziert. Die Flexibilität des Bewertungssystems in globalem Sequenzalignment machte es möglich, Symbole die berechneten Alignments stärker beeinflussen zu lassen, von denen bekannt war, dass sie für essentielle Wechselwirkungen in der Rezeptor-Ligand-Interaktion stehen. Es wurde gezeigt, dass diese Gewichtung die Screening Fähigkeiten von PhAST signifikant steigerte. Weiterhin konnte gezeigt werden, dass PhAST mit der systematischen Anwendung von Gewichten auf alle Sequenzpositionen in der Lage war, essentielle Wechselwirkungen für die Rezeptor-Ligand-Interaktion zu identifizieren. Bedingung hierfür war jedoch, dass ein geeigneter Datensatz mit aktiven und inaktiven Substanzen zur Verfügung stand. In dieser Arbeit wurden verschiedene Methoden evaluiert, mit denen aus Alignments Ähnlichkeiten berechnet werden können: Sequenzidentität, Alignment Score und p-Werte. Es wurde gezeigt, dass der Alignmentscore der Sequenzidentität für die Verwendung in PhAST signifikant überlegen ist. Für die Berechnung von p-Werten zur Bestimmung der Signfifikanz von Alignments musste zunächst die Verteilung von Alignment Scores für zufällige Sequenzen bestimmter Längen bestimmt werden. Dies geschah mit einer Kombination aus ‚Marcov Chain Monte Carlo Simulation’ und ‚Importance Sampling’. Die berechneten p-Werte wurden einer Bonferroni Korrektur unterzogen, und so unter Berücksichtigung der Gesamtzahl von im virtuellen Screening verglichenen Molekülen zu E-Werten. Als Ergebnis dieser Arbeit wird ein E-Wert von 1*10-5 als Grenzwert vorgeschlagen, wobei Alignments mit geringeren E-Werten als signifikant anzuerkennen sind. PhAST wurde in retrospektiven Screening mit anderen Methoden zum virtuellen Screening verglichen. Es konnte gezeigt werden, dass seine Fähigkeiten zur Identifizierung funktioneller Analoga mit denen der besten anderen Methoden vergleichbar oder ihnen sogar überlegen ist. Es konnte gezeigt werden, dass nach von PhAST berechneten Ähnlichkeiten sortierte Molekülsammlungen von den Sortierungen anderer Methoden abweichen. Im Rahmen dieser Arbeit wurde PhAST erfolgreich in zwei prospektiven Anwendungen eingesetzt. So wurde ein schwacher Inhibitor der bakteriellen Thymidinkinase identifiziert, der kein Nukleosid Analogon ist. In einem Screening nach Modulatoren der Gamma-Sekretase wurde ein potentes Molekül identifiziert, das deutliche strukturelle Unterschiede zur verwendeten Referenz aufwies

Dimension-reduction and discrimination of neuronal multi-channel signals

Dimensionsreduktion und Trennung neuronaler Multikanal-Signale

Use of Whole Genome Shotgun Sequencing for the Analysis of Microbial Communities in Arabidopsis thaliana Leaves

Microorganisms, such as all Bacteria, Archaeae, and some Eukaryotes, inhabit all imaginable habitats in the planet, from water vents in the deep ocean to extreme environments of high temperature and salinity. Microbes also constitute the most diverse group of organisms in terms if genetic information, metabolic function, and taxonomy. Furthermore, many of these microbes establish complex interactions with each others and with many other multicellular organisms. The collection of microbes that share a body space with a plant or animal is called the microbiota, and their genetic information is called the microbiome. The microbiota has emerged as a crucial determinant of a host’s overall health and understanding it has become crucial in many biological fields. In mammals, the gut microbiota has been linked to important diseases such as diabetes, inflammatory bowel disease, and dementia. In plants, the microbiota can provide protection against certain pathogens or confer resistance against harsh environmental conditions such as drought. Furthermore, the leaves of plants represent one of the largest surface areas that can potentially be colonized by microbes. The advent of sequencing technologies has let researchers to study microbial communities at unprecedented resolution and scale. By targeting individual loci such as the 16S rDNA locus in bacteria, many species can be studied simultaneously, as well as their properties such as relative abundance without the need of individual isolation of target taxa. Decreasing costs of DNA sequencing has also led to whole shotgun sequencing where instead of targeting a single or a number of loci, random fragments of DNA are sequenced. This effectively renders the entire microbiome accessible to study, referred to as metagenomics. Consequently many more areas of investigation are open, such as the exploration of within host genetic diversity, functional analysis, or assembly of individual genomes from metagenomes. In this study, I described the analysis of metagenomic sequencing data from microbial 11 communities in leaves of wild Arabidopsis thaliana individuals from southwest Germany. As a model organisms, A. thaliana not only is accessible in the wild but also has a rich body of previous research in plant-microbe interactions. In the first section, I describe how whole shotgun sequencing of leaf DNA extracts can be used to accurately describe the taxonomic composition of the microbial community of individual hosts. The nature of whole shotgun sequencing is used to estimate true microbial abundances which can not be done with amplicons sequencing. I show how this community varies across hosts, but some trends are seen, such as the dominance of the bacterial genera Pseudomonas and Sphingomonas . Moreover, even though there is variation between individuals, I explore the influence of site of origin and host genotype. Finally, metagenomic assembly is applied to individual samples, showing the limitations of WGS in plant leaves. In the second section, I explore the genomic diversity of the most abundant genera: Pseudomonas and Sphingomonas . I use a core genome approach where a set of common genes is obtained from previously sequenced and assembled genomes. Thereafter, the gene sequences of the core genome is used as a reference for short genome mapping. Based on these mappings, individual strain mixtures are inferred based on the frequency distribution of non reference bases at each detected single nucleotide polymorphism (SNP). Finally, SNP’s are then used to derive population structure of strain mixtures across samples and with known reference genomes. In conclusion, this thesis provides insights into the use of metagenomic sequencing to study microbial populations in wild plants. I identify the strengths and weaknesses of using whole genome sequencing for this purpose. As well as a way to study strain level dynamics of prevalent taxa within a single host

On Improving Generalization of CNN-Based Image Classification with Delineation Maps Using the CORF Push-Pull Inhibition Operator

Deployed image classification pipelines are typically dependent on the images captured in real-world environments. This means that images might be affected by different sources of perturbations (e.g. sensor noise in low-light environments). The main challenge arises by the fact that image quality directly impacts the reliability and consistency of classification tasks. This challenge has, hence, attracted wide interest within the computer vision communities. We propose a transformation step that attempts to enhance the generalization ability of CNN models in the presence of unseen noise in the test set. Concretely, the delineation maps of given images are determined using the CORF push-pull inhibition operator. Such an operation transforms an input image into a space that is more robust to noise before being processed by a CNN. We evaluated our approach on the Fashion MNIST data set with an AlexNet model. It turned out that the proposed CORF-augmented pipeline achieved comparable results on noise-free images to those of a conventional AlexNet classification model without CORF delineation maps, but it consistently achieved significantly superior performance on test images perturbed with different levels of Gaussian and uniform noise