1,409 research outputs found
Variational Analysis of Constrained M-Estimators
We propose a unified framework for establishing existence of nonparametric
M-estimators, computing the corresponding estimates, and proving their strong
consistency when the class of functions is exceptionally rich. In particular,
the framework addresses situations where the class of functions is complex
involving information and assumptions about shape, pointwise bounds, location
of modes, height at modes, location of level-sets, values of moments, size of
subgradients, continuity, distance to a "prior" function, multivariate total
positivity, and any combination of the above. The class might be engineered to
perform well in a specific setting even in the presence of little data. The
framework views the class of functions as a subset of a particular metric space
of upper semicontinuous functions under the Attouch-Wets distance. In addition
to allowing a systematic treatment of numerous M-estimators, the framework
yields consistency of plug-in estimators of modes of densities, maximizers of
regression functions, level-sets of classifiers, and related quantities, and
also enables computation by means of approximating parametric classes. We
establish consistency through a one-sided law of large numbers, here extended
to sieves, that relaxes assumptions of uniform laws, while ensuring global
approximations even under model misspecification
Learning mixtures of structured distributions over discrete domains
Let be a class of probability distributions over the discrete
domain We show that if satisfies a rather
general condition -- essentially, that each distribution in can
be well-approximated by a variable-width histogram with few bins -- then there
is a highly efficient (both in terms of running time and sample complexity)
algorithm that can learn any mixture of unknown distributions from
We analyze several natural types of distributions over , including
log-concave, monotone hazard rate and unimodal distributions, and show that
they have the required structural property of being well-approximated by a
histogram with few bins. Applying our general algorithm, we obtain
near-optimally efficient algorithms for all these mixture learning problems.Comment: preliminary full version of soda'13 pape
On the Computational Complexity of MCMC-based Estimators in Large Samples
In this paper we examine the implications of the statistical large sample
theory for the computational complexity of Bayesian and quasi-Bayesian
estimation carried out using Metropolis random walks. Our analysis is motivated
by the Laplace-Bernstein-Von Mises central limit theorem, which states that in
large samples the posterior or quasi-posterior approaches a normal density.
Using the conditions required for the central limit theorem to hold, we
establish polynomial bounds on the computational complexity of general
Metropolis random walks methods in large samples. Our analysis covers cases
where the underlying log-likelihood or extremum criterion function is possibly
non-concave, discontinuous, and with increasing parameter dimension. However,
the central limit theorem restricts the deviations from continuity and
log-concavity of the log-likelihood or extremum criterion function in a very
specific manner.
Under minimal assumptions required for the central limit theorem to hold
under the increasing parameter dimension, we show that the Metropolis algorithm
is theoretically efficient even for the canonical Gaussian walk which is
studied in detail. Specifically, we show that the running time of the algorithm
in large samples is bounded in probability by a polynomial in the parameter
dimension , and, in particular, is of stochastic order in the leading
cases after the burn-in period. We then give applications to exponential
families, curved exponential families, and Z-estimation of increasing
dimension.Comment: 36 pages, 2 figure
- …