384 research outputs found
Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer
Autoencoders provide a powerful framework for learning compressed
representations by encoding all of the information needed to reconstruct a data
point in a latent code. In some cases, autoencoders can "interpolate": By
decoding the convex combination of the latent codes for two datapoints, the
autoencoder can produce an output which semantically mixes characteristics from
the datapoints. In this paper, we propose a regularization procedure which
encourages interpolated outputs to appear more realistic by fooling a critic
network which has been trained to recover the mixing coefficient from
interpolated data. We then develop a simple benchmark task where we can
quantitatively measure the extent to which various autoencoders can interpolate
and show that our regularizer dramatically improves interpolation in this
setting. We also demonstrate empirically that our regularizer produces latent
codes which are more effective on downstream tasks, suggesting a possible link
between interpolation abilities and learning useful representations
Generative Model without Prior Distribution Matching
Variational Autoencoder (VAE) and its variations are classic generative
models by learning a low-dimensional latent representation to satisfy some
prior distribution (e.g., Gaussian distribution). Their advantages over GAN are
that they can simultaneously generate high dimensional data and learn latent
representations to reconstruct the inputs. However, it has been observed that a
trade-off exists between reconstruction and generation since matching prior
distribution may destroy the geometric structure of data manifold. To mitigate
this problem, we propose to let the prior match the embedding distribution
rather than imposing the latent variables to fit the prior. The embedding
distribution is trained using a simple regularized autoencoder architecture
which preserves the geometric structure to the maximum. Then an adversarial
strategy is employed to achieve a latent mapping. We provide both theoretical
and experimental support for the effectiveness of our method, which alleviates
the contradiction between topological properties' preserving of data manifold
and distribution matching in latent space.Comment: 8 pages, 8 figure
Quantization-Based Regularization for Autoencoders
Autoencoders and their variations provide unsupervised models for learning
low-dimensional representations for downstream tasks. Without proper
regularization, autoencoder models are susceptible to the overfitting problem
and the so-called posterior collapse phenomenon. In this paper, we introduce a
quantization-based regularizer in the bottleneck stage of autoencoder models to
learn meaningful latent representations. We combine both perspectives of Vector
Quantized-Variational AutoEncoders (VQ-VAE) and classical denoising
regularization methods of neural networks. We interpret quantizers as
regularizers that constrain latent representations while fostering a
similarity-preserving mapping at the encoder. Before quantization, we impose
noise on the latent codes and use a Bayesian estimator to optimize the
quantizer-based representation. The introduced bottleneck Bayesian estimator
outputs the posterior mean of the centroids to the decoder, and thus, is
performing soft quantization of the noisy latent codes. We show that our
proposed regularization method results in improved latent representations for
both supervised learning and clustering downstream tasks when compared to
autoencoders using other bottleneck structures.Comment: AAAI 202
HRINet: Alternative Supervision Network for High-resolution CT image Interpolation
Image interpolation in medical area is of high importance as most 3D
biomedical volume images are sampled where the distance between consecutive
slices significantly greater than the in-plane pixel size due to radiation dose
or scanning time. Image interpolation creates a number of new slices between
known slices in order to obtain an isotropic volume image. The results can be
used for the higher quality of 3D reconstruction and visualization of human
body structures. Semantic interpolation on the manifold has been proved to be
very useful for smoothing image interpolation. Nevertheless, all previous
methods focused on low-resolution image interpolation, and most of them work
poorly on high-resolution image. We propose a novel network, High Resolution
Interpolation Network (HRINet), aiming at producing high-resolution CT image
interpolations. We combine the idea of ACAI and GANs, and propose a novel idea
of alternative supervision method by applying supervised and unsupervised
training alternatively to raise the accuracy of human organ structures in CT
while keeping high quality. We compare an MSE based and a perceptual based loss
optimizing methods for high quality interpolation, and show the tradeoff
between the structural correctness and sharpness. Our experiments show the
great improvement on 256 2 and 5122 images quantitatively and qualitatively
Adversarially Approximated Autoencoder for Image Generation and Manipulation
Regularized autoencoders learn the latent codes, a structure with the
regularization under the distribution, which enables them the capability to
infer the latent codes given observations and generate new samples given the
codes. However, they are sometimes ambiguous as they tend to produce
reconstructions that are not necessarily faithful reproduction of the inputs.
The main reason is to enforce the learned latent code distribution to match a
prior distribution while the true distribution remains unknown. To improve the
reconstruction quality and learn the latent space a manifold structure, this
work present a novel approach using the adversarially approximated autoencoder
(AAAE) to investigate the latent codes with adversarial approximation. Instead
of regularizing the latent codes by penalizing on the distance between the
distributions of the model and the target, AAAE learns the autoencoder flexibly
and approximates the latent space with a simpler generator. The ratio is
estimated using generative adversarial network (GAN) to enforce the similarity
of the distributions. Additionally, the image space is regularized with an
additional adversarial regularizer. The proposed approach unifies two deep
generative models for both latent space inference and diverse generation. The
learning scheme is realized without regularization on the latent codes, which
also encourages faithful reconstruction. Extensive validation experiments on
four real-world datasets demonstrate the superior performance of AAAE. In
comparison to the state-of-the-art approaches, AAAE generates samples with
better quality and shares the properties of regularized autoencoder with a nice
latent manifold structure
Faithful Autoencoder Interpolation by Shaping the Latent Space
One of the fascinating properties of deep learning is the ability of the
network to reveal the underlying factors characterizing elements in datasets of
different types. Autoencoders represent an effective approach for computing
these factors. Autoencoders have been studied in the context of their ability
to interpolate between data points by decoding mixed latent vectors. However,
this interpolation often incorporates disrupting artifacts or produces
unrealistic images during reconstruction. We argue that these incongruities are
due to the manifold structure of the latent space where interpolated latent
vectors deviate from the data manifold. In this paper, we propose a
regularization technique that shapes the latent space following the manifold
assumption while enforcing the manifold to be smooth and convex. This
regularization enables faithful interpolation between data points and can be
used as a general regularization as well for avoiding overfitting and
constraining the model complexity
Recent Advances in Autoencoder-Based Representation Learning
Learning useful representations with little or no supervision is a key
challenge in artificial intelligence. We provide an in-depth review of recent
advances in representation learning with a focus on autoencoder-based models.
To organize these results we make use of meta-priors believed useful for
downstream tasks, such as disentanglement and hierarchical organization of
features. In particular, we uncover three main mechanisms to enforce such
properties, namely (i) regularizing the (approximate or aggregate) posterior
distribution, (ii) factorizing the encoding and decoding distribution, or (iii)
introducing a structured prior distribution. While there are some promising
results, implicit or explicit supervision remains a key enabler and all current
methods use strong inductive biases and modeling assumptions. Finally, we
provide an analysis of autoencoder-based representation learning through the
lens of rate-distortion theory and identify a clear tradeoff between the amount
of prior knowledge available about the downstream tasks, and how useful the
representation is for this task.Comment: Presented at the third workshop on Bayesian Deep Learning (NeurIPS
2018
Concept-Oriented Deep Learning: Generative Concept Representations
Generative concept representations have three major advantages over
discriminative ones: they can represent uncertainty, they support integration
of learning and reasoning, and they are good for unsupervised and
semi-supervised learning. We discuss probabilistic and generative deep
learning, which generative concept representations are based on, and the use of
variational autoencoders and generative adversarial networks for learning
generative concept representations, particularly for concepts whose data are
sequences, structured data or graphs
Transfer Adaptation Learning: A Decade Survey
The world we see is ever-changing and it always changes with people, things,
and the environment. Domain is referred to as the state of the world at a
certain moment. A research problem is characterized as transfer adaptation
learning (TAL) when it needs knowledge correspondence between different
moments/domains. Conventional machine learning aims to find a model with the
minimum expected risk on test data by minimizing the regularized empirical risk
on the training data, which, however, supposes that the training and test data
share similar joint probability distribution. TAL aims to build models that can
perform tasks of target domain by learning knowledge from a semantic related
but distribution different source domain. It is an energetic research filed of
increasing influence and importance, which is presenting a blowout publication
trend. This paper surveys the advances of TAL methodologies in the past decade,
and the technical challenges and essential problems of TAL have been observed
and discussed with deep insights and new perspectives. Broader solutions of
transfer adaptation learning being created by researchers are identified, i.e.,
instance re-weighting adaptation, feature adaptation, classifier adaptation,
deep network adaptation and adversarial adaptation, which are beyond the early
semi-supervised and unsupervised split. The survey helps researchers rapidly
but comprehensively understand and identify the research foundation, research
status, theoretical limitations, future challenges and under-studied issues
(universality, interpretability, and credibility) to be broken in the field
toward universal representation and safe applications in open-world scenarios.Comment: 26 pages, 4 figure
Augmentation-Interpolative AutoEncoders for Unsupervised Few-Shot Image Generation
We aim to build image generation models that generalize to new domains from
few examples. To this end, we first investigate the generalization properties
of classic image generators, and discover that autoencoders generalize
extremely well to new domains, even when trained on highly constrained data. We
leverage this insight to produce a robust, unsupervised few-shot image
generation algorithm, and introduce a novel training procedure based on
recovering an image from data augmentations. Our Augmentation-Interpolative
AutoEncoders synthesize realistic images of novel objects from only a few
reference images, and outperform both prior interpolative models and supervised
few-shot image generators. Our procedure is simple and lightweight, generalizes
broadly, and requires no category labels or other supervision during training
- …