283 research outputs found
Regularization and Bayesian Learning in Dynamical Systems: Past, Present and Future
Regularization and Bayesian methods for system identification have been
repopularized in the recent years, and proved to be competitive w.r.t.
classical parametric approaches. In this paper we shall make an attempt to
illustrate how the use of regularization in system identification has evolved
over the years, starting from the early contributions both in the Automatic
Control as well as Econometrics and Statistics literature. In particular we
shall discuss some fundamental issues such as compound estimation problems and
exchangeability which play and important role in regularization and Bayesian
approaches, as also illustrated in early publications in Statistics. The
historical and foundational issues will be given more emphasis (and space), at
the expense of the more recent developments which are only briefly discussed.
The main reason for such a choice is that, while the recent literature is
readily available, and surveys have already been published on the subject, in
the author's opinion a clear link with past work had not been completely
clarified.Comment: Plenary Presentation at the IFAC SYSID 2015. Submitted to Annual
Reviews in Contro
Maximum Entropy Kernels for System Identification
A new nonparametric approach for system identification has been recently
proposed where the impulse response is modeled as the realization of a
zero-mean Gaussian process whose covariance (kernel) has to be estimated from
data. In this scheme, quality of the estimates crucially depends on the
parametrization of the covariance of the Gaussian process. A family of kernels
that have been shown to be particularly effective in the system identification
framework is the family of Diagonal/Correlated (DC) kernels. Maximum entropy
properties of a related family of kernels, the Tuned/Correlated (TC) kernels,
have been recently pointed out in the literature. In this paper we show that
maximum entropy properties indeed extend to the whole family of DC kernels. The
maximum entropy interpretation can be exploited in conjunction with results on
matrix completion problems in the graphical models literature to shed light on
the structure of the DC kernel. In particular, we prove that the DC kernel
admits a closed-form factorization, inverse and determinant. These results can
be exploited both to improve the numerical stability and to reduce the
computational complexity associated with the computation of the DC estimator.Comment: Extends results of 2014 IEEE MSC Conference Proceedings
(arXiv:1406.5706
Efficient Multidimensional Regularization for Volterra Series Estimation
This paper presents an efficient nonparametric time domain nonlinear system
identification method. It is shown how truncated Volterra series models can be
efficiently estimated without the need of long, transient-free measurements.
The method is a novel extension of the regularization methods that have been
developed for impulse response estimates of linear time invariant systems. To
avoid the excessive memory needs in case of long measurements or large number
of estimated parameters, a practical gradient-based estimation method is also
provided, leading to the same numerical results as the proposed Volterra
estimation method. Moreover, the transient effects in the simulated output are
removed by a special regularization method based on the novel ideas of
transient removal for Linear Time-Varying (LTV) systems. Combining the proposed
methodologies, the nonparametric Volterra models of the cascaded water tanks
benchmark are presented in this paper. The results for different scenarios
varying from a simple Finite Impulse Response (FIR) model to a 3rd degree
Volterra series with and without transient removal are compared and studied. It
is clear that the obtained models capture the system dynamics when tested on a
validation dataset, and their performance is comparable with the white-box
(physical) models
Regularized Nonparametric Volterra Kernel Estimation
In this paper, the regularization approach introduced recently for
nonparametric estimation of linear systems is extended to the estimation of
nonlinear systems modelled as Volterra series. The kernels of order higher than
one, representing higher dimensional impulse responses in the series, are
considered to be realizations of multidimensional Gaussian processes. Based on
this, prior information about the structure of the Volterra kernel is
introduced via an appropriate penalization term in the least squares cost
function. It is shown that the proposed method is able to deliver accurate
estimates of the Volterra kernels even in the case of a small amount of data
points
The Harmonic Analysis of Kernel Functions
Kernel-based methods have been recently introduced for linear system
identification as an alternative to parametric prediction error methods.
Adopting the Bayesian perspective, the impulse response is modeled as a
non-stationary Gaussian process with zero mean and with a certain kernel (i.e.
covariance) function. Choosing the kernel is one of the most challenging and
important issues. In the present paper we introduce the harmonic analysis of
this non-stationary process, and argue that this is an important tool which
helps in designing such kernel. Furthermore, this analysis suggests also an
effective way to approximate the kernel, which allows to reduce the
computational burden of the identification procedure
Regularized linear system identification using atomic, nuclear and kernel-based norms: the role of the stability constraint
Inspired by ideas taken from the machine learning literature, new
regularization techniques have been recently introduced in linear system
identification. In particular, all the adopted estimators solve a regularized
least squares problem, differing in the nature of the penalty term assigned to
the impulse response. Popular choices include atomic and nuclear norms (applied
to Hankel matrices) as well as norms induced by the so called stable spline
kernels. In this paper, a comparative study of estimators based on these
different types of regularizers is reported. Our findings reveal that stable
spline kernels outperform approaches based on atomic and nuclear norms since
they suitably embed information on impulse response stability and smoothness.
This point is illustrated using the Bayesian interpretation of regularization.
We also design a new class of regularizers defined by "integral" versions of
stable spline/TC kernels. Under quite realistic experimental conditions, the
new estimators outperform classical prediction error methods also when the
latter are equipped with an oracle for model order selection
Kernel-based methods for Volterra series identification
Volterra series approximate a broad range of nonlinear systems. Their identification is challenging due to the curse of dimensionality: the number of model parameters grows exponentially with the complexity of the input-output response. This fact limits the applicability of such models and has stimulated recently much research on regularized solutions. Along this line, we propose two new strategies that use kernel-based methods. First, we introduce the multiplicative polynomial kernel (MPK). Compared to the standard polynomial kernel, the MPK is equipped with a richer set of hyperparameters, increasing flexibility in selecting the monomials that really influence the system output. Second, we introduce the smooth exponentially decaying multiplicative polynomial kernel (SEDMPK), that is a regularized version of MPK which requires less hyperparameters, allowing to handle also high-order Volterra series. Numerical results show the effectiveness of the two approaches. (C) 2021 Elsevier Ltd. All rights reserved
Bayesian and regularization approaches to multivariable linear system identification: the role of rank penalties
Recent developments in linear system identification have proposed the use of
non-parameteric methods, relying on regularization strategies, to handle the
so-called bias/variance trade-off. This paper introduces an impulse response
estimator which relies on an -type regularization including a
rank-penalty derived using the log-det heuristic as a smooth approximation to
the rank function. This allows to account for different properties of the
estimated impulse response (e.g. smoothness and stability) while also
penalizing high-complexity models. This also allows to account and enforce
coupling between different input-output channels in MIMO systems. According to
the Bayesian paradigm, the parameters defining the relative weight of the two
regularization terms as well as the structure of the rank penalty are estimated
optimizing the marginal likelihood. Once these hyperameters have been
estimated, the impulse response estimate is available in closed form.
Experiments show that the proposed method is superior to the estimator relying
on the "classic" -regularization alone as well as those based in atomic
and nuclear norm.Comment: to appear in IEEE Conference on Decision and Control, 201
- …