4 research outputs found
Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference
Understanding the gradient variance of black-box variational inference (BBVI)
is a crucial step for establishing its convergence and developing algorithmic
improvements. However, existing studies have yet to show that the gradient
variance of BBVI satisfies the conditions used to study the convergence of
stochastic gradient descent (SGD), the workhorse of BBVI. In this work, we show
that BBVI satisfies a matching bound corresponding to the condition used
in the SGD literature when applied to smooth and quadratically-growing
log-likelihoods. Our results generalize to nonlinear covariance
parameterizations widely used in the practice of BBVI. Furthermore, we show
that the variance of the mean-field parameterization has provably superior
dimensional dependence.Comment: Accepted to ICML'23 for live oral presentatio
Provable convergence guarantees for black-box variational inference
While black-box variational inference is widely used, there is no proof that
its stochastic optimization succeeds. We suggest this is due to a theoretical
gap in existing stochastic optimization proofs-namely the challenge of gradient
estimators with unusual noise bounds, and a composite non-smooth objective. For
dense Gaussian variational families, we observe that existing gradient
estimators based on reparameterization satisfy a quadratic noise bound and give
novel convergence guarantees for proximal and projected stochastic gradient
descent using this bound. This provides the first rigorous guarantee that
black-box variational inference converges for realistic inference problems.Comment: 32 page