573 research outputs found
Enhancing Low-Precision Sampling via Stochastic Gradient Hamiltonian Monte Carlo
Low-precision training has emerged as a promising low-cost technique to
enhance the training efficiency of deep neural networks without sacrificing
much accuracy. Its Bayesian counterpart can further provide uncertainty
quantification and improved generalization accuracy. This paper investigates
low-precision sampling via Stochastic Gradient Hamiltonian Monte Carlo (SGHMC)
with low-precision and full-precision gradient accumulators for both strongly
log-concave and non-log-concave distributions. Theoretically, our results show
that, to achieve -error in the 2-Wasserstein distance for
non-log-concave distributions, low-precision SGHMC achieves quadratic
improvement
()
compared to the state-of-the-art low-precision sampler, Stochastic Gradient
Langevin Dynamics (SGLD)
().
Moreover, we prove that low-precision SGHMC is more robust to the quantization
error compared to low-precision SGLD due to the robustness of the
momentum-based update w.r.t. gradient noise. Empirically, we conduct
experiments on synthetic data, and {MNIST, CIFAR-10 \& CIFAR-100} datasets,
which validate our theoretical findings. Our study highlights the potential of
low-precision SGHMC as an efficient and accurate sampling method for
large-scale and resource-limited machine learning
Langevin and Hamiltonian based Sequential MCMC for Efficient Bayesian Filtering in High-dimensional Spaces
Nonlinear non-Gaussian state-space models arise in numerous applications in
statistics and signal processing. In this context, one of the most successful
and popular approximation techniques is the Sequential Monte Carlo (SMC)
algorithm, also known as particle filtering. Nevertheless, this method tends to
be inefficient when applied to high dimensional problems. In this paper, we
focus on another class of sequential inference methods, namely the Sequential
Markov Chain Monte Carlo (SMCMC) techniques, which represent a promising
alternative to SMC methods. After providing a unifying framework for the class
of SMCMC approaches, we propose novel efficient strategies based on the
principle of Langevin diffusion and Hamiltonian dynamics in order to cope with
the increasing number of high-dimensional applications. Simulation results show
that the proposed algorithms achieve significantly better performance compared
to existing algorithms
- …