17,828 research outputs found
Regularization and Bayesian Learning in Dynamical Systems: Past, Present and Future
Regularization and Bayesian methods for system identification have been
repopularized in the recent years, and proved to be competitive w.r.t.
classical parametric approaches. In this paper we shall make an attempt to
illustrate how the use of regularization in system identification has evolved
over the years, starting from the early contributions both in the Automatic
Control as well as Econometrics and Statistics literature. In particular we
shall discuss some fundamental issues such as compound estimation problems and
exchangeability which play and important role in regularization and Bayesian
approaches, as also illustrated in early publications in Statistics. The
historical and foundational issues will be given more emphasis (and space), at
the expense of the more recent developments which are only briefly discussed.
The main reason for such a choice is that, while the recent literature is
readily available, and surveys have already been published on the subject, in
the author's opinion a clear link with past work had not been completely
clarified.Comment: Plenary Presentation at the IFAC SYSID 2015. Submitted to Annual
Reviews in Contro
Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem
This paper studies the multiplicity-correction effect of standard Bayesian
variable-selection priors in linear regression. Our first goal is to clarify
when, and how, multiplicity correction happens automatically in Bayesian
analysis, and to distinguish this correction from the Bayesian Ockham's-razor
effect. Our second goal is to contrast empirical-Bayes and fully Bayesian
approaches to variable selection through examples, theoretical results and
simulations. Considerable differences between the two approaches are found. In
particular, we prove a theorem that characterizes a surprising aymptotic
discrepancy between fully Bayes and empirical Bayes. This discrepancy arises
from a different source than the failure to account for hyperparameter
uncertainty in the empirical-Bayes estimate. Indeed, even at the extreme, when
the empirical-Bayes estimate converges asymptotically to the true
variable-inclusion probability, the potential for a serious difference remains.Comment: Published in at http://dx.doi.org/10.1214/10-AOS792 the Annals of
Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Post-selection point and interval estimation of signal sizes in Gaussian samples
We tackle the problem of the estimation of a vector of means from a single
vector-valued observation . Whereas previous work reduces the size of the
estimates for the largest (absolute) sample elements via shrinkage (like
James-Stein) or biases estimated via empirical Bayes methodology, we take a
novel approach. We adapt recent developments by Lee et al (2013) in post
selection inference for the Lasso to the orthogonal setting, where sample
elements have different underlying signal sizes. This is exactly the setup
encountered when estimating many means. It is shown that other selection
procedures, like selecting the largest (absolute) sample elements and the
Benjamini-Hochberg procedure, can be cast into their framework, allowing us to
leverage their results. Point and interval estimates for signal sizes are
proposed. These seem to perform quite well against competitors, both recent and
more tenured.
Furthermore, we prove an upper bound to the worst case risk of our estimator,
when combined with the Benjamini-Hochberg procedure, and show that it is within
a constant multiple of the minimax risk over a rich set of parameter spaces
meant to evoke sparsity.Comment: 27 pages, 13 figure
Marginal likelihoods in phylogenetics: a review of methods and applications
By providing a framework of accounting for the shared ancestry inherent to
all life, phylogenetics is becoming the statistical foundation of biology. The
importance of model choice continues to grow as phylogenetic models continue to
increase in complexity to better capture micro and macroevolutionary processes.
In a Bayesian framework, the marginal likelihood is how data update our prior
beliefs about models, which gives us an intuitive measure of comparing model
fit that is grounded in probability theory. Given the rapid increase in the
number and complexity of phylogenetic models, methods for approximating
marginal likelihoods are increasingly important. Here we try to provide an
intuitive description of marginal likelihoods and why they are important in
Bayesian model testing. We also categorize and review methods for estimating
marginal likelihoods of phylogenetic models, highlighting several recent
methods that provide well-behaved estimates. Furthermore, we review some
empirical studies that demonstrate how marginal likelihoods can be used to
learn about models of evolution from biological data. We discuss promising
alternatives that can complement marginal likelihoods for Bayesian model
choice, including posterior-predictive methods. Using simulations, we find one
alternative method based on approximate-Bayesian computation (ABC) to be
biased. We conclude by discussing the challenges of Bayesian model choice and
future directions that promise to improve the approximation of marginal
likelihoods and Bayesian phylogenetics as a whole.Comment: 33 pages, 3 figure
Fast and scalable non-parametric Bayesian inference for Poisson point processes
We study the problem of non-parametric Bayesian estimation of the intensity
function of a Poisson point process. The observations are independent
realisations of a Poisson point process on the interval . We propose two
related approaches. In both approaches we model the intensity function as
piecewise constant on bins forming a partition of the interval . In
the first approach the coefficients of the intensity function are assigned
independent gamma priors, leading to a closed form posterior distribution. On
the theoretical side, we prove that as the posterior
asymptotically concentrates around the "true", data-generating intensity
function at an optimal rate for -H\"older regular intensity functions (). In the second approach we employ a gamma Markov chain prior on the
coefficients of the intensity function. The posterior distribution is no longer
available in closed form, but inference can be performed using a
straightforward version of the Gibbs sampler. Both approaches scale well with
sample size, but the second is much less sensitive to the choice of .
Practical performance of our methods is first demonstrated via synthetic data
examples. We compare our second method with other existing approaches on the UK
coal mining disasters data. Furthermore, we apply it to the US mass shootings
data and Donald Trump's Twitter data.Comment: 45 pages, 22 figure
- …