24 research outputs found
Generative Adversarial Networks via a Composite Annealing of Noise and Diffusion
Generative adversarial network (GAN) is a framework for generating fake data
using a set of real examples. However, GAN is unstable in the training stage.
In order to stabilize GANs, the noise injection has been used to enlarge the
overlap of the real and fake distributions at the cost of increasing variance.
The diffusion (or smoothing) may reduce the intrinsic underlying dimensionality
of data but it suppresses the capability of GANs to learn high-frequency
information in the training procedure. Based on these observations, we propose
a data representation for the GAN training, called noisy scale-space (NSS),
that recursively applies the smoothing with a balanced noise to data in order
to replace the high-frequency information by random data, leading to a
coarse-to-fine training of GANs. We experiment with NSS using DCGAN and
StyleGAN2 based on benchmark datasets in which the NSS-based GANs outperforms
the state-of-the-arts in most cases
Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processes
Deep Gaussian Process (DGP) models offer a powerful nonparametric approach
for Bayesian inference, but exact inference is typically intractable,
motivating the use of various approximations. However, existing approaches,
such as mean-field Gaussian assumptions, limit the expressiveness and efficacy
of DGP models, while stochastic approximation can be computationally expensive.
To tackle these challenges, we introduce Neural Operator Variational Inference
(NOVI) for Deep Gaussian Processes. NOVI uses a neural generator to obtain a
sampler and minimizes the Regularized Stein Discrepancy in L2 space between the
generated distribution and true posterior. We solve the minimax problem using
Monte Carlo estimation and subsampling stochastic optimization techniques. We
demonstrate that the bias introduced by our method can be controlled by
multiplying the Fisher divergence with a constant, which leads to robust error
control and ensures the stability and precision of the algorithm. Our
experiments on datasets ranging from hundreds to tens of thousands demonstrate
the effectiveness and the faster convergence rate of the proposed method. We
achieve a classification accuracy of 93.56 on the CIFAR10 dataset,
outperforming SOTA Gaussian process methods. Furthermore, our method guarantees
theoretically controlled prediction error for DGP models and demonstrates
remarkable performance on various datasets. We are optimistic that NOVI has the
potential to enhance the performance of deep Bayesian nonparametric models and
could have significant implications for various practical application
Accelerating optimization over the space of probability measures
Acceleration of gradient-based optimization methods is an issue of
significant practical and theoretical interest, particularly in machine
learning applications. Most research has focused on optimization over Euclidean
spaces, but given the need to optimize over spaces of probability measures in
many machine learning problems, it is of interest to investigate accelerated
gradient methods in this context too. To this end, we introduce a
Hamiltonian-flow approach that is analogous to moment-based approaches in
Euclidean space. We demonstrate that algorithms based on this approach can
achieve convergence rates of arbitrarily high order. Numerical examples
illustrate our claim
OPT-GAN: Black-Box Global Optimization via Generative Adversarial Nets
Black-box optimization (BBO) algorithms are concerned with finding the best
solutions for problems with missing analytical details. Most classical methods
for such problems are based on strong and fixed a priori assumptions, such as
Gaussianity. However, the complex real-world problems, especially when the
global optimum is desired, could be very far from the a priori assumptions
because of their diversities, causing unexpected obstacles to these methods. In
this study, we propose a generative adversarial net-based broad-spectrum global
optimizer (OPT-GAN) which estimates the distribution of optimum gradually, with
strategies to balance exploration-exploitation trade-off. It has potential to
better adapt to the regularity and structure of diversified landscapes than
other methods with fixed prior, e.g. Gaussian assumption or separability.
Experiments conducted on BBO benchmarking problems and several other benchmarks
with diversified landscapes exhibit that OPT-GAN outperforms other traditional
and neural net-based BBO algorithms.Comment: M. Lu and S. Ning contribute equally. Submitted to IEEE transactions
on Neural Networks and Learning System
NF-ULA: Langevin Monte Carlo with Normalizing Flow Prior for Imaging Inverse Problems
Bayesian methods for solving inverse problems are a powerful alternative to
classical methods since the Bayesian approach offers the ability to quantify
the uncertainty in the solution. In recent years, data-driven techniques for
solving inverse problems have also been remarkably successful, due to their
superior representation ability. In this work, we incorporate data-based models
into a class of Langevin-based sampling algorithms for Bayesian inference in
imaging inverse problems. In particular, we introduce NF-ULA (Normalizing
Flow-based Unadjusted Langevin algorithm), which involves learning a
normalizing flow (NF) as the image prior. We use NF to learn the prior because
a tractable closed-form expression for the log prior enables the
differentiation of it using autograd libraries. Our algorithm only requires a
normalizing flow-based generative network, which can be pre-trained
independently of the considered inverse problem and the forward operator. We
perform theoretical analysis by investigating the well-posedness and
non-asymptotic convergence of the resulting NF-ULA algorithm. The efficacy of
the proposed NF-ULA algorithm is demonstrated in various image restoration
problems such as image deblurring, image inpainting, and limited-angle X-ray
computed tomography (CT) reconstruction. NF-ULA is found to perform better than
competing methods for severely ill-posed inverse problems
NF-ULA: Normalizing Flow-Based Unadjusted Langevin Algorithm for Imaging Inverse Problems
Bayesian methods for solving inverse problems are a powerful alternative to classical methods since the Bayesian approach offers the ability to quantify the uncertainty in the solution. In recent years, data-driven techniques for solving inverse problems have also been remarkably successful, due to their superior representation ability. In this work, we incorporate data-based models into a class of Langevin-based sampling algorithms for Bayesian inference in imaging inverse problems. In particular, we introduce NF-ULA (Normalizing Flow-based Unadjusted Langevin algorithm), which involves learning a normalizing flow (NF) as the image prior. We use NF to learn the prior because a tractable closed-form expression for the log prior enables the differentiation of it using autograd libraries. Our algorithm only requires a normalizing flow-based generative network, which can be pre-trained independently of the considered inverse problem and the forward operator. We perform theoretical analysis by investigating the well-posedness and non-asymptotic convergence of the resulting NF-ULA algorithm. The efficacy of the proposed NF-ULA algorithm is demonstrated in various image restoration problems such as image deblurring, image inpainting, and limited-angle X-ray computed tomography (CT) reconstruction. NF-ULA is found to perform better than competing methods for severely ill-posed inverse problems
NF-ULA: Normalizing Flow-Based Unadjusted Langevin Algorithm for Imaging Inverse Problems
Bayesian methods for solving inverse problems are a powerful alternative to classical methods since the Bayesian approach offers the ability to quantify the uncertainty in the solution. In recent years, data-driven techniques for solving inverse problems have also been remarkably successful, due to their superior representation ability. In this work, we incorporate data-based models into a class of Langevin-based sampling algorithms for Bayesian inference in imaging inverse problems. In particular, we introduce NF-ULA (Normalizing Flow-based Unadjusted Langevin algorithm), which involves learning a normalizing flow (NF) as the image prior. We use NF to learn the prior because a tractable closed-form expression for the log prior enables the differentiation of it using autograd libraries. Our algorithm only requires a normalizing flow-based generative network, which can be pre-trained independently of the considered inverse problem and the forward operator. We perform theoretical analysis by investigating the well-posedness and non-asymptotic convergence of the resulting NF-ULA algorithm. The efficacy of the proposed NF-ULA algorithm is demonstrated in various image restoration problems such as image deblurring, image inpainting, and limited-angle X-ray computed tomography (CT) reconstruction. NF-ULA is found to perform better than competing methods for severely ill-posed inverse problems
Hybrid Energy-based Models for Image Generation and Classification
In recent years, deep neural networks (DNNs) have achieved state-of-the-art performance on a wide range of learning tasks. Among those tasks, two fundamental tasks are discriminative models and generative models. However, they are largely separated although prior works have shown that generative training is beneficial to classifiers to alleviate several notorious issues. Energy-based Model (EBM) especially the Joint Energy-based Model(JEM) only needs to train a single network with shared features for discriminative and generative tasks. However, EBMs are expensive to train and very unstable. It is crucial to understand the behavior of EBM training and thus improve the stability, speed, accuracy, and generative quality altogether. This dissertation mainly summarizes my research on EBMs for Hybrid Image Discriminative-Generative Models. We first proposed GMMC which models the joint density p(x, y). As an alternative to the SoftMax classifier utilized in JEM, GMMC has a well-formulated latent feature distribution, which fits well with the generative process of image synthesis. Then we came up with a variety of new training techniques to improve JEM\u27s accuracy, training stability, and speed altogether, and we named it JEM++. Based on JEM++, we analyzed and improved it from three different aspects, 1) the manifold, 2) the data augmentation, 3) the energy landscape. Hence, we propose Manifold-Aware EBM/JEM and Sharpness-Aware JEM to further improve the speed, generation quality, stability, and classification significantly. Beyond MCMC-based EBM, we found we can combine two recent emergent approaches Vision Transformer (ViT) and Denoising Diffusion Probabilistic Model (DDPM) to learn a simple but powerful model for image classification and generation. The new direction can get rid of most disadvantages of EBM, such as the expensive MCMC sampling and instability. Finally, we discuss future research topics including the speed, generation quality, and applications of hybrid models