357 research outputs found
Max-Sliced Wasserstein Distance and its use for GANs
Generative adversarial nets (GANs) and variational auto-encoders have
significantly improved our distribution modeling capabilities, showing promise
for dataset augmentation, image-to-image translation and feature learning.
However, to model high-dimensional distributions, sequential training and
stacked architectures are common, increasing the number of tunable
hyper-parameters as well as the training time. Nonetheless, the sample
complexity of the distance metrics remains one of the factors affecting GAN
training. We first show that the recently proposed sliced Wasserstein distance
has compelling sample complexity properties when compared to the Wasserstein
distance. To further improve the sliced Wasserstein distance we then analyze
its `projection complexity' and develop the max-sliced Wasserstein distance
which enjoys compelling sample complexity while reducing projection complexity,
albeit necessitating a max estimation. We finally illustrate that the proposed
distance trains GANs on high-dimensional images up to a resolution of 256x256
easily.Comment: Accepted to CVPR 201
Sliced Wasserstein Generative Models
In generative modeling, the Wasserstein distance (WD) has emerged as a useful
metric to measure the discrepancy between generated and real data
distributions. Unfortunately, it is challenging to approximate the WD of
high-dimensional distributions. In contrast, the sliced Wasserstein distance
(SWD) factorizes high-dimensional distributions into their multiple
one-dimensional marginal distributions and is thus easier to approximate. In
this paper, we introduce novel approximations of the primal and dual SWD.
Instead of using a large number of random projections, as it is done by
conventional SWD approximation methods, we propose to approximate SWDs with a
small number of parameterized orthogonal projections in an end-to-end deep
learning fashion. As concrete applications of our SWD approximations, we design
two types of differentiable SWD blocks to equip modern generative
frameworks---Auto-Encoders (AE) and Generative Adversarial Networks (GAN). In
the experiments, we not only show the superiority of the proposed generative
models on standard image synthesis benchmarks, but also demonstrate the
state-of-the-art performance on challenging high resolution image and video
generation in an unsupervised manner.Comment: This paper is accepted by CVPR 2019, accidentally uploaded as a new
submission (arXiv:1904.05408, which has been withdrawn). The code is
available at this https URL https:// github.com/musikisomorphie/swd.gi
Amortized Projection Optimization for Sliced Wasserstein Generative Models
Seeking informative projecting directions has been an important task in
utilizing sliced Wasserstein distance in applications. However, finding these
directions usually requires an iterative optimization procedure over the space
of projecting directions, which is computationally expensive. Moreover, the
computational issue is even more severe in deep learning applications, where
computing the distance between two mini-batch probability measures is repeated
several times. This nested loop has been one of the main challenges that
prevent the usage of sliced Wasserstein distances based on good projections in
practice. To address this challenge, we propose to utilize the
learning-to-optimize technique or amortized optimization to predict the
informative direction of any given two mini-batch probability measures. To the
best of our knowledge, this is the first work that bridges amortized
optimization and sliced Wasserstein generative models. In particular, we derive
linear amortized models, generalized linear amortized models, and non-linear
amortized models which are corresponding to three types of novel mini-batch
losses, named amortized sliced Wasserstein. We demonstrate the favorable
performance of the proposed sliced losses in deep generative modeling on
standard benchmark datasets.Comment: Accepted to NeurIPS 2022, 22 pages, 6 figures, 8 table
SAN: Inducing Metrizability of GAN with Discriminative Normalized Linear Layer
Generative adversarial networks (GANs) learn a target probability
distribution by optimizing a generator and a discriminator with minimax
objectives. This paper addresses the question of whether such optimization
actually provides the generator with gradients that make its distribution close
to the target distribution. We derive metrizable conditions, sufficient
conditions for the discriminator to serve as the distance between the
distributions by connecting the GAN formulation with the concept of sliced
optimal transport. Furthermore, by leveraging these theoretical results, we
propose a novel GAN training scheme, called slicing adversarial network (SAN).
With only simple modifications, a broad class of existing GANs can be converted
to SANs. Experiments on synthetic and image datasets support our theoretical
results and the SAN's effectiveness as compared to usual GANs. Furthermore, we
also apply SAN to StyleGAN-XL, which leads to state-of-the-art FID score
amongst GANs for class conditional generation on ImageNet 256256.Comment: 24 pages with 12 figure
- …