8,156 research outputs found
Diffusion-Model-Assisted Supervised Learning of Generative Models for Density Estimation
We present a supervised learning framework of training generative models for
density estimation. Generative models, including generative adversarial
networks, normalizing flows, variational auto-encoders, are usually considered
as unsupervised learning models, because labeled data are usually unavailable
for training. Despite the success of the generative models, there are several
issues with the unsupervised training, e.g., requirement of reversible
architectures, vanishing gradients, and training instability. To enable
supervised learning in generative models, we utilize the score-based diffusion
model to generate labeled data. Unlike existing diffusion models that train
neural networks to learn the score function, we develop a training-free score
estimation method. This approach uses mini-batch-based Monte Carlo estimators
to directly approximate the score function at any spatial-temporal location in
solving an ordinary differential equation (ODE), corresponding to the
reverse-time stochastic differential equation (SDE). This approach can offer
both high accuracy and substantial time savings in neural network training.
Once the labeled data are generated, we can train a simple fully connected
neural network to learn the generative model in the supervised manner. Compared
with existing normalizing flow models, our method does not require to use
reversible neural networks and avoids the computation of the Jacobian matrix.
Compared with existing diffusion models, our method does not need to solve the
reverse-time SDE to generate new samples. As a result, the sampling efficiency
is significantly improved. We demonstrate the performance of our method by
applying it to a set of 2D datasets as well as real data from the UCI
repository
Reversible GANs for Memory-efficient Image-to-Image Translation
The Pix2pix and CycleGAN losses have vastly improved the qualitative and
quantitative visual quality of results in image-to-image translation tasks. We
extend this framework by exploring approximately invertible architectures which
are well suited to these losses. These architectures are approximately
invertible by design and thus partially satisfy cycle-consistency before
training even begins. Furthermore, since invertible architectures have constant
memory complexity in depth, these models can be built arbitrarily deep. We are
able to demonstrate superior quantitative output on the Cityscapes and Maps
datasets at near constant memory budget
- …