36 research outputs found
Sparse Regression Learning by Aggregation and Langevin Monte-Carlo
We consider the problem of regression learning for deterministic design and
independent random errors. We start by proving a sharp PAC-Bayesian type bound
for the exponentially weighted aggregate (EWA) under the expected squared
empirical loss. For a broad class of noise distributions the presented bound is
valid whenever the temperature parameter of the EWA is larger than or
equal to , where is the noise variance. A remarkable
feature of this result is that it is valid even for unbounded regression
functions and the choice of the temperature parameter depends exclusively on
the noise level. Next, we apply this general bound to the problem of
aggregating the elements of a finite-dimensional linear space spanned by a
dictionary of functions . We allow to be much larger
than the sample size but we assume that the true regression function can be
well approximated by a sparse linear combination of functions . Under
this sparsity scenario, we propose an EWA with a heavy tailed prior and we show
that it satisfies a sparsity oracle inequality with leading constant one.
Finally, we propose several Langevin Monte-Carlo algorithms to approximately
compute such an EWA when the number of aggregated functions can be large.
We discuss in some detail the convergence of these algorithms and present
numerical experiments that confirm our theoretical findings.Comment: Short version published in COLT 200
Entropic optimal transport is maximum-likelihood deconvolution
We give a statistical interpretation of entropic optimal transport by showing
that performing maximum-likelihood estimation for Gaussian deconvolution
corresponds to calculating a projection with respect to the entropic optimal
transport distance. This structural result gives theoretical support for the
wide adoption of these tools in the machine learning community
Pac-bayesian bounds for sparse regression estimation with exponential weights
We consider the sparse regression model where the number of parameters is
larger than the sample size . The difficulty when considering
high-dimensional problems is to propose estimators achieving a good compromise
between statistical and computational performances. The BIC estimator for
instance performs well from the statistical point of view \cite{BTW07} but can
only be computed for values of of at most a few tens. The Lasso estimator
is solution of a convex minimization problem, hence computable for large value
of . However stringent conditions on the design are required to establish
fast rates of convergence for this estimator. Dalalyan and Tsybakov
\cite{arnak} propose a method achieving a good compromise between the
statistical and computational aspects of the problem. Their estimator can be
computed for reasonably large and satisfies nice statistical properties
under weak assumptions on the design. However, \cite{arnak} proposes sparsity
oracle inequalities in expectation for the empirical excess risk only. In this
paper, we propose an aggregation procedure similar to that of \cite{arnak} but
with improved statistical performances. Our main theoretical result is a
sparsity oracle inequality in probability for the true excess risk for a
version of exponential weight estimator. We also propose a MCMC method to
compute our estimator for reasonably large values of .Comment: 19 page
Statistical inference in compound functional models
We consider a general nonparametric regression model called the compound
model. It includes, as special cases, sparse additive regression and
nonparametric (or linear) regression with many covariates but possibly a small
number of relevant covariates. The compound model is characterized by three
main parameters: the structure parameter describing the "macroscopic" form of
the compound function, the "microscopic" sparsity parameter indicating the
maximal number of relevant covariates in each component and the usual
smoothness parameter corresponding to the complexity of the members of the
compound. We find non-asymptotic minimax rate of convergence of estimators in
such a model as a function of these three parameters. We also show that this
rate can be attained in an adaptive way
PAC-Bayesian High Dimensional Bipartite Ranking
This paper is devoted to the bipartite ranking problem, a classical
statistical learning task, in a high dimensional setting. We propose a scoring
and ranking strategy based on the PAC-Bayesian approach. We consider nonlinear
additive scoring functions, and we derive non-asymptotic risk bounds under a
sparsity assumption. In particular, oracle inequalities in probability holding
under a margin condition assess the performance of our procedure, and prove its
minimax optimality. An MCMC-flavored algorithm is proposed to implement our
method, along with its behavior on synthetic and real-life datasets