979 research outputs found
Regularization and Bayesian Learning in Dynamical Systems: Past, Present and Future
Regularization and Bayesian methods for system identification have been
repopularized in the recent years, and proved to be competitive w.r.t.
classical parametric approaches. In this paper we shall make an attempt to
illustrate how the use of regularization in system identification has evolved
over the years, starting from the early contributions both in the Automatic
Control as well as Econometrics and Statistics literature. In particular we
shall discuss some fundamental issues such as compound estimation problems and
exchangeability which play and important role in regularization and Bayesian
approaches, as also illustrated in early publications in Statistics. The
historical and foundational issues will be given more emphasis (and space), at
the expense of the more recent developments which are only briefly discussed.
The main reason for such a choice is that, while the recent literature is
readily available, and surveys have already been published on the subject, in
the author's opinion a clear link with past work had not been completely
clarified.Comment: Plenary Presentation at the IFAC SYSID 2015. Submitted to Annual
Reviews in Contro
Regularized linear system identification using atomic, nuclear and kernel-based norms: the role of the stability constraint
Inspired by ideas taken from the machine learning literature, new
regularization techniques have been recently introduced in linear system
identification. In particular, all the adopted estimators solve a regularized
least squares problem, differing in the nature of the penalty term assigned to
the impulse response. Popular choices include atomic and nuclear norms (applied
to Hankel matrices) as well as norms induced by the so called stable spline
kernels. In this paper, a comparative study of estimators based on these
different types of regularizers is reported. Our findings reveal that stable
spline kernels outperform approaches based on atomic and nuclear norms since
they suitably embed information on impulse response stability and smoothness.
This point is illustrated using the Bayesian interpretation of regularization.
We also design a new class of regularizers defined by "integral" versions of
stable spline/TC kernels. Under quite realistic experimental conditions, the
new estimators outperform classical prediction error methods also when the
latter are equipped with an oracle for model order selection
Augmented balancing weights as linear regression
We provide a novel characterization of augmented balancing weights, also
known as Automatic Debiased Machine Learning (AutoDML). These estimators
combine outcome modeling with balancing weights, which estimate inverse
propensity score weights directly. When the outcome and weighting models are
both linear in some (possibly infinite) basis, we show that the augmented
estimator is equivalent to a single linear model with coefficients that combine
the original outcome model coefficients and OLS; in many settings, the
augmented estimator collapses to OLS alone. We then extend these results to
specific choices of outcome and weighting models. We first show that the
combined estimator that uses (kernel) ridge regression for both outcome and
weighting models is equivalent to a single, undersmoothed (kernel) ridge
regression; this also holds when considering asymptotic rates. When the
weighting model is instead lasso regression, we give closed-form expressions
for special cases and demonstrate a ``double selection'' property. Finally, we
generalize these results to linear estimands via the Riesz representer. Our
framework ``opens the black box'' on these increasingly popular estimators and
provides important insights into estimation choices for augmented balancing
weights
Bayesian and regularization approaches to multivariable linear system identification: the role of rank penalties
Recent developments in linear system identification have proposed the use of
non-parameteric methods, relying on regularization strategies, to handle the
so-called bias/variance trade-off. This paper introduces an impulse response
estimator which relies on an -type regularization including a
rank-penalty derived using the log-det heuristic as a smooth approximation to
the rank function. This allows to account for different properties of the
estimated impulse response (e.g. smoothness and stability) while also
penalizing high-complexity models. This also allows to account and enforce
coupling between different input-output channels in MIMO systems. According to
the Bayesian paradigm, the parameters defining the relative weight of the two
regularization terms as well as the structure of the rank penalty are estimated
optimizing the marginal likelihood. Once these hyperameters have been
estimated, the impulse response estimate is available in closed form.
Experiments show that the proposed method is superior to the estimator relying
on the "classic" -regularization alone as well as those based in atomic
and nuclear norm.Comment: to appear in IEEE Conference on Decision and Control, 201
Distributed multi-agent Gaussian regression via finite-dimensional approximations
We consider the problem of distributedly estimating Gaussian processes in
multi-agent frameworks. Each agent collects few measurements and aims to
collaboratively reconstruct a common estimate based on all data. Agents are
assumed with limited computational and communication capabilities and to gather
noisy measurements in total on input locations independently drawn from a
known common probability density. The optimal solution would require agents to
exchange all the input locations and measurements and then invert an matrix, a non-scalable task. Differently, we propose two suboptimal
approaches using the first orthonormal eigenfunctions obtained from the
\ac{KL} expansion of the chosen kernel, where typically . The benefits
are that the computation and communication complexities scale with and not
with , and computing the required statistics can be performed via standard
average consensus algorithms. We obtain probabilistic non-asymptotic bounds
that determine a priori the desired level of estimation accuracy, and new
distributed strategies relying on Stein's unbiased risk estimate (SURE)
paradigms for tuning the regularization parameters and applicable to generic
basis functions (thus not necessarily kernel eigenfunctions) and that can again
be implemented via average consensus. The proposed estimators and bounds are
finally tested on both synthetic and real field data
Functional Generalized Empirical Likelihood Estimation for Conditional Moment Restrictions
Important problems in causal inference, economics, and, more generally,
robust machine learning can be expressed as conditional moment restrictions,
but estimation becomes challenging as it requires solving a continuum of
unconditional moment restrictions. Previous works addressed this problem by
extending the generalized method of moments (GMM) to continuum moment
restrictions. In contrast, generalized empirical likelihood (GEL) provides a
more general framework and has been shown to enjoy favorable small-sample
properties compared to GMM-based estimators. To benefit from recent
developments in machine learning, we provide a functional reformulation of GEL
in which arbitrary models can be leveraged. Motivated by a dual formulation of
the resulting infinite dimensional optimization problem, we devise a practical
method and explore its asymptotic properties. Finally, we provide kernel- and
neural network-based implementations of the estimator, which achieve
state-of-the-art empirical performance on two conditional moment restriction
problems
- …