2,361 research outputs found
Regularization and Bayesian Learning in Dynamical Systems: Past, Present and Future
Regularization and Bayesian methods for system identification have been
repopularized in the recent years, and proved to be competitive w.r.t.
classical parametric approaches. In this paper we shall make an attempt to
illustrate how the use of regularization in system identification has evolved
over the years, starting from the early contributions both in the Automatic
Control as well as Econometrics and Statistics literature. In particular we
shall discuss some fundamental issues such as compound estimation problems and
exchangeability which play and important role in regularization and Bayesian
approaches, as also illustrated in early publications in Statistics. The
historical and foundational issues will be given more emphasis (and space), at
the expense of the more recent developments which are only briefly discussed.
The main reason for such a choice is that, while the recent literature is
readily available, and surveys have already been published on the subject, in
the author's opinion a clear link with past work had not been completely
clarified.Comment: Plenary Presentation at the IFAC SYSID 2015. Submitted to Annual
Reviews in Contro
Tight conditions for consistency of variable selection in the context of high dimensionality
We address the issue of variable selection in the regression model with very
high ambient dimension, that is, when the number of variables is very large.
The main focus is on the situation where the number of relevant variables,
called intrinsic dimension, is much smaller than the ambient dimension d.
Without assuming any parametric form of the underlying regression function, we
get tight conditions making it possible to consistently estimate the set of
relevant variables. These conditions relate the intrinsic dimension to the
ambient dimension and to the sample size. The procedure that is provably
consistent under these tight conditions is based on comparing quadratic
functionals of the empirical Fourier coefficients with appropriately chosen
threshold values. The asymptotic analysis reveals the presence of two quite
different re gimes. The first regime is when the intrinsic dimension is fixed.
In this case the situation in nonparametric regression is the same as in linear
regression, that is, consistent variable selection is possible if and only if
log d is small compared to the sample size n. The picture is different in the
second regime, that is, when the number of relevant variables denoted by s
tends to infinity as . Then we prove that consistent variable
selection in nonparametric set-up is possible only if s+loglog d is small
compared to log n. We apply these results to derive minimax separation rates
for the problem of variableComment: arXiv admin note: text overlap with arXiv:1102.3616; Published in at
http://dx.doi.org/10.1214/12-AOS1046 the Annals of Statistics
(http://www.imstat.org/aos/) by the Institute of Mathematical Statistics
(http://www.imstat.org
Advances in Feature Selection with Mutual Information
The selection of features that are relevant for a prediction or
classification problem is an important problem in many domains involving
high-dimensional data. Selecting features helps fighting the curse of
dimensionality, improving the performances of prediction or classification
methods, and interpreting the application. In a nonlinear context, the mutual
information is widely used as relevance criterion for features and sets of
features. Nevertheless, it suffers from at least three major limitations:
mutual information estimators depend on smoothing parameters, there is no
theoretically justified stopping criterion in the feature selection greedy
procedure, and the estimation itself suffers from the curse of dimensionality.
This chapter shows how to deal with these problems. The two first ones are
addressed by using resampling techniques that provide a statistical basis to
select the estimator parameters and to stop the search procedure. The third one
is addressed by modifying the mutual information criterion into a measure of
how features are complementary (and not only informative) for the problem at
hand
- âŠ