45 research outputs found
Advances in Bayesian Inference for Binary and Categorical Data
No abstract availableBayesian binary probit regression and its extensions to time-dependent observations and multi-class responses are popular tools in binary and categorical data regression due to their high interpretability and non-restrictive assumptions.
Although the theory is well established in the frequentist literature, such models still face a florid research in the Bayesian framework.This is mostly due to the fact that state-of-the-art methods for Bayesian inference in such settings are either computationally impractical or inaccurate in high dimensions and in many cases a closed-form expression for the posterior distribution of the model parameters is, apparently, lacking.The development of improved computational methods and theoretical results to perform inference with this vast class of models is then of utmost importance.
In order to overcome the above-mentioned computational issues, we develop a novel variational approximation for the posterior of the coefficients in high-dimensional probit regression with binary responses and Gaussian priors, resulting in a unified skew-normal (SUN) approximating distribution that converges to the exact posterior as the number of predictors p increases.
Moreover, we show that closed-form expressions are actually available for posterior distributions arising from models that account for correlated binary time-series and multi-class responses.
In the former case, we prove that the filtering, predictive and smoothing distributions in dynamic probit models with Gaussian state variables are, in fact, available and belong to a class of SUN distributions whose parameters can be updated recursively in time via analytical expressions, allowing to develop an i.i.d. sampler together with an optimal sequential Monte Carlo procedure.
As for the latter case, i.e. multi-class probit models, we show that many different formulations developed in the literature in separate ways admit a unified view and a closed-form SUN posterior distribution under a SUN prior distribution (thus including the Gaussian case).
This allows to implement computational methods which outperform state-of-the-art routines in high-dimensional settings by leveraging SUN properties and the variational methods introduced for the binary probit.
Finally, motivated also by the possible linkage of some of the above-mentioned models to the Bayesian nonparametrics literature, a novel species-sampling model for partially-exchangeable observations is introduced, with the double goal of both predicting the class (or species) of the future observations and testing for homogeneity among the different available populations.
Such model arises from a combination of Pitman-Yor processes and leverages on the appealing features of both hierarchical and nested structures developed in the Bayesian nonparametrics literature.
Posterior inference is feasible thanks to the implementation of a marginal Gibbs sampler, whose pseudo-code is given in full detail
A Class of Conjugate Priors for Multinomial Probit Models which Includes the Multivariate Normal One
Multinomial probit models are widely-implemented representations which allow
both classification and inference by learning changes in vectors of class
probabilities with a set of p observed predictors. Although various frequentist
methods have been developed for estimation, inference and classification within
such a class of models, Bayesian inference is still lagging behind. This is due
to the apparent absence of a tractable class of conjugate priors, that may
facilitate posterior inference on the multinomial probit coefficients. Such an
issue has motivated increasing efforts toward the development of effective
Markov chain Monte Carlo methods, but state-of-the-art solutions still face
severe computational bottlenecks, especially in large p settings. In this
article, we prove that the entire class of unified skew-normal (SUN)
distributions is conjugate to a wide variety of multinomial probit models, and
we exploit the SUN properties to improve upon state-of-art-solutions for
posterior inference and classification both in terms of closed-form results for
key functionals of interest, and also by developing novel computational methods
relying either on independent and identically distributed samples from the
exact posterior or on scalable and accurate variational approximations based on
blocked partially-factorized representations. As illustrated in a
gastrointestinal lesions application, the magnitude of the improvements
relative to current methods is particularly evident, in practice, when the
focus is on large p applications
Variational inference for the smoothing distribution in dynamic probit models
Recently, Fasano, Rebaudo, Durante and Petrone (2019) provided closed-form
expressions for the filtering, predictive and smoothing distributions of
multivariate dynamic probit models, leveraging on unified skew-normal
distribution properties. This allows to develop algorithms to draw independent
and identically distributed samples from such distributions, as well as
sequential Monte Carlo procedures for the filtering and predictive
distributions, allowing to overcome computational bottlenecks that may arise
for large sample sizes. In this paper, we briefly review the above-mentioned
closed-form expressions, mainly focusing on the smoothing distribution of the
univariate dynamic probit. We develop a variational Bayes approach, extending
the partially factorized mean-field variational approximation introduced by
Fasano, Durante and Zanella (2019) for the static binary probit model to the
dynamic setting. Results are shown for a financial application
Expectation propagation for the smoothing distribution in dynamic probit
The smoothing distribution of dynamic probit models with Gaussian state
dynamics was recently proved to belong to the unified skew-normal family.
Although this is computationally tractable in small-to-moderate settings, it
may become computationally impractical in higher dimensions. In this work,
adapting a recent more general class of expectation propagation (EP)
algorithms, we derive an efficient EP routine to perform inference for such a
distribution. We show that the proposed approximation leads to accuracy gains
over available approximate algorithms in a financial illustration
Scalable and Accurate Variational Bayes for High-Dimensional Binary Regression Models
Modern methods for Bayesian regression beyond the Gaussian response setting
are computationally impractical or inaccurate in high dimensions. As discussed
in recent literature, bypassing this trade-off is still an open problem even in
basic binary regression models, and there is limited theory on the quality of
variational approximations in high-dimensional settings. To address this gap,
we study the approximation accuracy of routine-use mean-field variational Bayes
in high-dimensional probit regression with Gaussian priors, obtaining new and
practically relevant results on the pathological behavior of this strategy in
uncertainty quantification, estimation and prediction, that also suggest
caution against maximum a posteriori estimates when p>n. Motivated by these
results, we develop a new partially-factorized variational approximation for
the posterior distribution of the probit coefficients that leverages a
representation with global and local variables but, unlike for classical
mean-field assumptions, it avoids a fully factorized approximation, and instead
assumes a factorization only for local variables. We prove that the resulting
approximation belongs to a tractable class of unified skew-normal densities
that incorporates skewness and, unlike for state-of-the-art mean-field
solutions, converges to the exact posterior density as p goes to infinity. To
solve the variational optimization problem, we derive a tractable CAVI
algorithm that easily scales to p in tens of thousands, and provably requires a
number of iterations converging to 1 as p goes to infinity. Such findings are
also illustrated in extensive empirical studies where our new solution is shown
to improve the accuracy of mean-field variational Bayes for any n and p, with
the magnitude of these gains being remarkable in those high-dimensional p>n
settings where state-of-the-art methods are computationally impractical
Efficient expectation propagation for posterior approximation in high-dimensional probit models
Bayesian binary regression is a prosperous area of research due to the
computational challenges encountered by currently available methods either for
high-dimensional settings or large datasets, or both. In the present work, we
focus on the expectation propagation (EP) approximation of the posterior
distribution in Bayesian probit regression under a multivariate Gaussian prior
distribution. Adapting more general derivations in Anceschi et al. (2023), we
show how to leverage results on the extended multivariate skew-normal
distribution to derive an efficient implementation of the EP routine having a
per-iteration cost that scales linearly in the number of covariates. This makes
EP computationally feasible also in challenging high-dimensional settings, as
shown in a detailed simulation study