5 research outputs found
Intrinsic Universal Measurements of Non-linear Embeddings
A basic problem in machine learning is to find a mapping from a low
dimensional latent space to a high dimensional observation space. Equipped with
the representation power of non-linearity, a learner can easily find a mapping
which perfectly fits all the observations. However such a mapping is often not
considered as good as it is not simple enough and over-fits. How to define
simplicity? This paper tries to make such a formal definition of the amount of
information imposed by a non-linear mapping. This definition is based on
information geometry and is independent of observations, nor specific
parametrizations. We prove these basic properties and discuss relationships
with parametric and non-parametric embeddings.Comment: work in progres
Parametric t-Distributed Stochastic Exemplar-centered Embedding
Parametric embedding methods such as parametric t-SNE (pt-SNE) have been
widely adopted for data visualization and out-of-sample data embedding without
further computationally expensive optimization or approximation. However, the
performance of pt-SNE is highly sensitive to the hyper-parameter batch size due
to conflicting optimization goals, and often produces dramatically different
embeddings with different choices of user-defined perplexities. To effectively
solve these issues, we present parametric t-distributed stochastic
exemplar-centered embedding methods. Our strategy learns embedding parameters
by comparing given data only with precomputed exemplars, resulting in a cost
function with linear computational and memory complexity, which is further
reduced by noise contrastive samples. Moreover, we propose a shallow embedding
network with high-order feature interactions for data visualization, which is
much easier to tune but produces comparable performance in contrast to a deep
neural network employed by pt-SNE. We empirically demonstrate, using several
benchmark datasets, that our proposed methods significantly outperform pt-SNE
in terms of robustness, visual effects, and quantitative evaluations.Comment: fixed typo