Semi-Implicit Variational Inference via Score Matching

Semi-implicit variational inference (SIVI) greatly enriches the expressiveness of variational families by considering implicit variational distributions defined in a hierarchical manner. However, due to the intractable densities of variational distributions, current SIVI approaches often use surrogate evidence lower bounds (ELBOs) or employ expensive inner-loop MCMC runs for unbiased ELBOs for training. In this paper, we propose SIVI-SM, a new method for SIVI based on an alternative training objective via score matching. Leveraging the hierarchical structure of semi-implicit variational families, the score matching objective allows a minimax formulation where the intractable variational densities can be naturally handled with denoising score matching. We show that SIVI-SM closely matches the accuracy of MCMC and outperforms ELBO-based SIVI methods in a variety of Bayesian inference tasks.Comment: 17 pages, 8 figures; ICLR 202

An Improved Nearest Neighbor Based Entropy Estimator with Local Ellipsoid Correction and its Application to Evaluation of MCMC Posterior Samples

Entropy estimation is an important technique to summarize the uncertainty of a distribution underlying a set of samples. It ties to important research problems in fields such as statistics, machine learning and so on. The k-nearest neighbor (kNN) estimator is one widely used classical nonparametric method although it suffers bias issue especially when the dimensionality of the data is high. In this thesis, an improved kNN entropy estimator is developed. The proposed method has the advantage of a learning a local ellipsoid to be used in the estimation, in order to mitigate the bias issue which results from the local uniformity. Several numerical experiments have been conducted and the results have shown that the proposed approach can efficiently reduce the bias especially in when the dimension is high. Another studied topic in this thesis is the evaluation of the correctness of the posterior samples when conducting Bayesian inferences. This thesis demonstrates that the proposed estimator can be applied to such a task. We show that the simulation-based approach is more efficient and discriminative than a lower bound based method by one simple experiment, and the proposed kNN estimation can improve the accuracy of the state-of-the-art simulation-based approach