795 research outputs found
GLSR-VAE: Geodesic Latent Space Regularization for Variational AutoEncoder Architectures
VAEs (Variational AutoEncoders) have proved to be powerful in the context of
density modeling and have been used in a variety of contexts for creative
purposes. In many settings, the data we model possesses continuous attributes
that we would like to take into account at generation time. We propose in this
paper GLSR-VAE, a Geodesic Latent Space Regularization for the Variational
AutoEncoder architecture and its generalizations which allows a fine control on
the embedding of the data into the latent space. When augmenting the VAE loss
with this regularization, changes in the learned latent space reflects changes
of the attributes of the data. This deeper understanding of the VAE latent
space structure offers the possibility to modulate the attributes of the
generated data in a continuous way. We demonstrate its efficiency on a
monophonic music generation task where we manage to generate variations of
discrete sequences in an intended and playful way.Comment: 11 page
Manifold Regularized Discriminative Neural Networks
Unregularized deep neural networks (DNNs) can be easily overfit with a
limited sample size. We argue that this is mostly due to the disriminative
nature of DNNs which directly model the conditional probability (or score) of
labels given the input. The ignorance of input distribution makes DNNs
difficult to generalize to unseen data. Recent advances in regularization
techniques, such as pretraining and dropout, indicate that modeling input data
distribution (either explicitly or implicitly) greatly improves the
generalization ability of a DNN. In this work, we explore the manifold
hypothesis which assumes that instances within the same class lie in a smooth
manifold. We accordingly propose two simple regularizers to a standard
discriminative DNN. The first one, named Label-Aware Manifold Regularization,
assumes the availability of labels and penalizes large norms of the loss
function w.r.t. data points. The second one, named Label-Independent Manifold
Regularization, does not use label information and instead penalizes the
Frobenius norm of the Jacobian matrix of prediction scores w.r.t. data points,
which makes semi-supervised learning possible. We perform extensive control
experiments on fully supervised and semi-supervised tasks using the MNIST,
CIFAR10 and SVHN datasets and achieve excellent results.Comment: In submission to ICLR 201
Adversarially Approximated Autoencoder for Image Generation and Manipulation
Regularized autoencoders learn the latent codes, a structure with the
regularization under the distribution, which enables them the capability to
infer the latent codes given observations and generate new samples given the
codes. However, they are sometimes ambiguous as they tend to produce
reconstructions that are not necessarily faithful reproduction of the inputs.
The main reason is to enforce the learned latent code distribution to match a
prior distribution while the true distribution remains unknown. To improve the
reconstruction quality and learn the latent space a manifold structure, this
work present a novel approach using the adversarially approximated autoencoder
(AAAE) to investigate the latent codes with adversarial approximation. Instead
of regularizing the latent codes by penalizing on the distance between the
distributions of the model and the target, AAAE learns the autoencoder flexibly
and approximates the latent space with a simpler generator. The ratio is
estimated using generative adversarial network (GAN) to enforce the similarity
of the distributions. Additionally, the image space is regularized with an
additional adversarial regularizer. The proposed approach unifies two deep
generative models for both latent space inference and diverse generation. The
learning scheme is realized without regularization on the latent codes, which
also encourages faithful reconstruction. Extensive validation experiments on
four real-world datasets demonstrate the superior performance of AAAE. In
comparison to the state-of-the-art approaches, AAAE generates samples with
better quality and shares the properties of regularized autoencoder with a nice
latent manifold structure
Deep Clustering via Joint Convolutional Autoencoder Embedding and Relative Entropy Minimization
Image clustering is one of the most important computer vision applications,
which has been extensively studied in literature. However, current clustering
methods mostly suffer from lack of efficiency and scalability when dealing with
large-scale and high-dimensional data. In this paper, we propose a new
clustering model, called DEeP Embedded RegularIzed ClusTering (DEPICT), which
efficiently maps data into a discriminative embedding subspace and precisely
predicts cluster assignments. DEPICT generally consists of a multinomial
logistic regression function stacked on top of a multi-layer convolutional
autoencoder. We define a clustering objective function using relative entropy
(KL divergence) minimization, regularized by a prior for the frequency of
cluster assignments. An alternating strategy is then derived to optimize the
objective by updating parameters and estimating cluster assignments.
Furthermore, we employ the reconstruction loss functions in our autoencoder, as
a data-dependent regularization term, to prevent the deep embedding function
from overfitting. In order to benefit from end-to-end optimization and
eliminate the necessity for layer-wise pretraining, we introduce a joint
learning framework to minimize the unified clustering and reconstruction loss
functions together and train all network layers simultaneously. Experimental
results indicate the superiority and faster running time of DEPICT in
real-world clustering tasks, where no labeled data is available for
hyper-parameter tuning
PILAE: A Non-gradient Descent Learning Scheme for Deep Feedforward Neural Networks
In this work, a non-gradient descent learning scheme is proposed for deep
feedforward neural networks (DNN). As we known, autoencoder can be used as the
building blocks of the multi-layer perceptron (MLP) deep neural network. So,
the MLP will be taken as an example to illustrate the proposed scheme of
pseudoinverse learning algorithm for autoencoder (PILAE) training. The PILAE
with low rank approximation is a non-gradient based learning algorithm, and the
encoder weight matrix is set to be the low rank approximation of the
pseudoinverse of the input matrix, while the decoder weight matrix is
calculated by the pseudoinverse learning algorithm. It is worth to note that
only few network structure hyperparameters need to be tuned. Hence, the
proposed algorithm can be regarded as a quasi-automated training algorithm
which can be utilized in autonomous machine learning research field. The
experimental results show that the proposed learning scheme for DNN can achieve
better performance on considering the tradeoff between training efficiency and
classification accuracy.Comment: This work is our effort toward to realize AutoM
Learning invariant features through local space contraction
We present in this paper a novel approach for training deterministic
auto-encoders. We show that by adding a well chosen penalty term to the
classical reconstruction cost function, we can achieve results that equal or
surpass those attained by other regularized auto-encoders as well as denoising
auto-encoders on a range of datasets. This penalty term corresponds to the
Frobenius norm of the Jacobian matrix of the encoder activations with respect
to the input. We show that this penalty term results in a localized space
contraction which in turn yields robust features on the activation layer.
Furthermore, we show how this penalty term is related to both regularized
auto-encoders and denoising encoders and how it can be seen as a link between
deterministic and non-deterministic auto-encoders. We find empirically that
this penalty helps to carve a representation that better captures the local
directions of variation dictated by the data, corresponding to a
lower-dimensional non-linear manifold, while being more invariant to the vast
majority of directions orthogonal to the manifold. Finally, we show that by
using the learned features to initialize a MLP, we achieve state of the art
classification error on a range of datasets, surpassing other methods of
pre-training
Stacked Wasserstein Autoencoder
Approximating distributions over complicated manifolds, such as natural
images, are conceptually attractive. The deep latent variable model, trained
using variational autoencoders and generative adversarial networks, is now a
key technique for representation learning. However, it is difficult to unify
these two models for exact latent-variable inference and parallelize both
reconstruction and sampling, partly due to the regularization under the latent
variables, to match a simple explicit prior distribution. These approaches are
prone to be oversimplified, and can only characterize a few modes of the true
distribution. Based on the recently proposed Wasserstein autoencoder (WAE) with
a new regularization as an optimal transport. The paper proposes a stacked
Wasserstein autoencoder (SWAE) to learn a deep latent variable model. SWAE is a
hierarchical model, which relaxes the optimal transport constraints at two
stages. At the first stage, the SWAE flexibly learns a representation
distribution, i.e., the encoded prior; and at the second stage, the encoded
representation distribution is approximated with a latent variable model under
the regularization encouraging the latent distribution to match the explicit
prior. This model allows us to generate natural textual outputs as well as
perform manipulations in the latent space to induce changes in the output
space. Both quantitative and qualitative results demonstrate the superior
performance of SWAE compared with the state-of-the-art approaches in terms of
faithful reconstruction and generation quality.Comment: arXiv admin note: text overlap with arXiv:1902.0558
Generative Class-conditional Autoencoders
Recent work by Bengio et al. (2013) proposes a sampling procedure for
denoising autoencoders which involves learning the transition operator of a
Markov chain. The transition operator is typically unimodal, which limits its
capacity to model complex data. In order to perform efficient sampling from
conditional distributions, we extend this work, both theoretically and
algorithmically, to gated autoencoders (Memisevic, 2013), The proposed model is
able to generate convincing class-conditional samples when trained on both the
MNIST and TFD datasets
Representation Learning with Autoencoders for Electronic Health Records: A Comparative Study
Increasing volume of Electronic Health Records (EHR) in recent years provides
great opportunities for data scientists to collaborate on different aspects of
healthcare research by applying advanced analytics to these EHR clinical data.
A key requirement however is obtaining meaningful insights from high
dimensional, sparse and complex clinical data. Data science approaches
typically address this challenge by performing feature learning in order to
build more reliable and informative feature representations from clinical data
followed by supervised learning. In this paper, we propose a predictive
modeling approach based on deep learning based feature representations and word
embedding techniques. Our method uses different deep architectures (stacked
sparse autoencoders, deep belief network, adversarial autoencoders and
variational autoencoders) for feature representation in higher-level
abstraction to obtain effective and robust features from EHRs, and then build
prediction models on top of them. Our approach is particularly useful when the
unlabeled data is abundant whereas labeled data is scarce. We investigate the
performance of representation learning through a supervised learning approach.
Our focus is to present a comparative study to evaluate the performance of
different deep architectures through supervised learning and provide insights
in the choice of deep feature representation techniques. Our experiments
demonstrate that for small data sets, stacked sparse autoencoder demonstrates a
superior generality performance in prediction due to sparsity regularization
whereas variational autoencoders outperform the competing approaches for large
data sets due to its capability of learning the representation distribution
Transfer Adaptation Learning: A Decade Survey
The world we see is ever-changing and it always changes with people, things,
and the environment. Domain is referred to as the state of the world at a
certain moment. A research problem is characterized as transfer adaptation
learning (TAL) when it needs knowledge correspondence between different
moments/domains. Conventional machine learning aims to find a model with the
minimum expected risk on test data by minimizing the regularized empirical risk
on the training data, which, however, supposes that the training and test data
share similar joint probability distribution. TAL aims to build models that can
perform tasks of target domain by learning knowledge from a semantic related
but distribution different source domain. It is an energetic research filed of
increasing influence and importance, which is presenting a blowout publication
trend. This paper surveys the advances of TAL methodologies in the past decade,
and the technical challenges and essential problems of TAL have been observed
and discussed with deep insights and new perspectives. Broader solutions of
transfer adaptation learning being created by researchers are identified, i.e.,
instance re-weighting adaptation, feature adaptation, classifier adaptation,
deep network adaptation and adversarial adaptation, which are beyond the early
semi-supervised and unsupervised split. The survey helps researchers rapidly
but comprehensively understand and identify the research foundation, research
status, theoretical limitations, future challenges and under-studied issues
(universality, interpretability, and credibility) to be broken in the field
toward universal representation and safe applications in open-world scenarios.Comment: 26 pages, 4 figure
- …