1,684 research outputs found
Robust Bayesian Regression with Synthetic Posterior
Although linear regression models are fundamental tools in statistical
science, the estimation results can be sensitive to outliers. While several
robust methods have been proposed in frequentist frameworks, statistical
inference is not necessarily straightforward. We here propose a Bayesian
approach to robust inference on linear regression models using synthetic
posterior distributions based on -divergence, which enables us to
naturally assess the uncertainty of the estimation through the posterior
distribution. We also consider the use of shrinkage priors for the regression
coefficients to carry out robust Bayesian variable selection and estimation
simultaneously. We develop an efficient posterior computation algorithm by
adopting the Bayesian bootstrap within Gibbs sampling. The performance of the
proposed method is illustrated through simulation studies and applications to
famous datasets.Comment: 23 pages, 5 figure
On the asymptotic rate of convergence of Stochastic Newton algorithms and their Weighted Averaged versions
The majority of machine learning methods can be regarded as the minimization
of an unavailable risk function. To optimize the latter, given samples provided
in a streaming fashion, we define a general stochastic Newton algorithm and its
weighted average version. In several use cases, both implementations will be
shown not to require the inversion of a Hessian estimate at each iteration, but
a direct update of the estimate of the inverse Hessian instead will be favored.
This generalizes a trick introduced in [2] for the specific case of logistic
regression, by directly updating the estimate of the inverse Hessian. Under
mild assumptions such as local strong convexity at the optimum, we establish
almost sure convergences and rates of convergence of the algorithms, as well as
central limit theorems for the constructed parameter estimates. The unified
framework considered in this paper covers the case of linear, logistic or
softmax regressions to name a few. Numerical experiments on simulated data give
the empirical evidence of the pertinence of the proposed methods, which
outperform popular competitors particularly in case of bad initializa-tions.Comment: Computational Optimization and Applications, 202
Conditionally conjugate mean-field variational Bayes for logistic models
Variational Bayes (VB) is a common strategy for approximate Bayesian
inference, but simple methods are only available for specific classes of models
including, in particular, representations having conditionally conjugate
constructions within an exponential family. Models with logit components are an
apparently notable exception to this class, due to the absence of conjugacy
between the logistic likelihood and the Gaussian priors for the coefficients in
the linear predictor. To facilitate approximate inference within this widely
used class of models, Jaakkola and Jordan (2000) proposed a simple variational
approach which relies on a family of tangent quadratic lower bounds of logistic
log-likelihoods, thus restoring conjugacy between these approximate bounds and
the Gaussian priors. This strategy is still implemented successfully, but less
attempts have been made to formally understand the reasons underlying its
excellent performance. To cover this key gap, we provide a formal connection
between the above bound and a recent P\'olya-gamma data augmentation for
logistic regression. Such a result places the computational methods associated
with the aforementioned bounds within the framework of variational inference
for conditionally conjugate exponential family models, thereby allowing recent
advances for this class to be inherited also by the methods relying on Jaakkola
and Jordan (2000)
Targeted Maximum Likelihood Estimation using Exponential Families
Targeted maximum likelihood estimation (TMLE) is a general method for
estimating parameters in semiparametric and nonparametric models. Each
iteration of TMLE involves fitting a parametric submodel that targets the
parameter of interest. We investigate the use of exponential families to define
the parametric submodel. This implementation of TMLE gives a general approach
for estimating any smooth parameter in the nonparametric model. A computational
advantage of this approach is that each iteration of TMLE involves estimation
of a parameter in an exponential family, which is a convex optimization problem
for which software implementing reliable and computationally efficient methods
exists. We illustrate the method in three estimation problems, involving the
mean of an outcome missing at random, the parameter of a median regression
model, and the causal effect of a continuous exposure, respectively. We conduct
a simulation study comparing different choices for the parametric submodel,
focusing on the first of these problems. To the best of our knowledge, this is
the first study investigating robustness of TMLE to different specifications of
the parametric submodel. We find that the choice of submodel can have an
important impact on the behavior of the estimator in finite samples
Model based clustering of multinomial count data
We consider the problem of inferring an unknown number of clusters in
replicated multinomial data. Under a model based clustering point of view, this
task can be treated by estimating finite mixtures of multinomial distributions
with or without covariates. Both Maximum Likelihood (ML) as well as Bayesian
estimation are taken into account. Under a Maximum Likelihood approach, we
provide an Expectation--Maximization (EM) algorithm which exploits a careful
initialization procedure combined with a ridge--stabilized implementation of
the Newton--Raphson method in the M--step. Under a Bayesian setup, a stochastic
gradient Markov chain Monte Carlo (MCMC) algorithm embedded within a prior
parallel tempering scheme is devised. The number of clusters is selected
according to the Integrated Completed Likelihood criterion in the ML approach
and estimating the number of non-empty components in overfitting mixture models
in the Bayesian case. Our method is illustrated in simulated data and applied
to two real datasets. An R package is available at
https://github.com/mqbssppe/multinomialLogitMix.Comment: to appear in ADA
Particle algorithms for optimization on binary spaces
We discuss a unified approach to stochastic optimization of pseudo-Boolean
objective functions based on particle methods, including the cross-entropy
method and simulated annealing as special cases. We point out the need for
auxiliary sampling distributions, that is parametric families on binary spaces,
which are able to reproduce complex dependency structures, and illustrate their
usefulness in our numerical experiments. We provide numerical evidence that
particle-driven optimization algorithms based on parametric families yield
superior results on strongly multi-modal optimization problems while local
search heuristics outperform them on easier problems
Smooth Transition Regression Models in UK Stock Returns
This paper models UK stock market returns in a smooth transition regression (STR) framework. We employ a variety of financial and macroeconomic series that are assumed to influence UK stock returns, namely GDP, interest rates, inflation, money supply and US stock prices. We estimate STR models where the linearity hypothesis is strongly rejected for at least one transition variable. These non-linear models describe the in-sample movements of the stock returns series better than the corresponding linear model. Moreover, the US stock market appears to play an important role in determining the UK stock market returns regime.
- …