13,700 research outputs found
Maximum Entropy Kernels for System Identification
A new nonparametric approach for system identification has been recently
proposed where the impulse response is modeled as the realization of a
zero-mean Gaussian process whose covariance (kernel) has to be estimated from
data. In this scheme, quality of the estimates crucially depends on the
parametrization of the covariance of the Gaussian process. A family of kernels
that have been shown to be particularly effective in the system identification
framework is the family of Diagonal/Correlated (DC) kernels. Maximum entropy
properties of a related family of kernels, the Tuned/Correlated (TC) kernels,
have been recently pointed out in the literature. In this paper we show that
maximum entropy properties indeed extend to the whole family of DC kernels. The
maximum entropy interpretation can be exploited in conjunction with results on
matrix completion problems in the graphical models literature to shed light on
the structure of the DC kernel. In particular, we prove that the DC kernel
admits a closed-form factorization, inverse and determinant. These results can
be exploited both to improve the numerical stability and to reduce the
computational complexity associated with the computation of the DC estimator.Comment: Extends results of 2014 IEEE MSC Conference Proceedings
(arXiv:1406.5706
Maximum Entropy Vector Kernels for MIMO system identification
Recent contributions have framed linear system identification as a
nonparametric regularized inverse problem. Relying on -type
regularization which accounts for the stability and smoothness of the impulse
response to be estimated, these approaches have been shown to be competitive
w.r.t classical parametric methods. In this paper, adopting Maximum Entropy
arguments, we derive a new penalty deriving from a vector-valued
kernel; to do so we exploit the structure of the Hankel matrix, thus
controlling at the same time complexity, measured by the McMillan degree,
stability and smoothness of the identified models. As a special case we recover
the nuclear norm penalty on the squared block Hankel matrix. In contrast with
previous literature on reweighted nuclear norm penalties, our kernel is
described by a small number of hyper-parameters, which are iteratively updated
through marginal likelihood maximization; constraining the structure of the
kernel acts as a (hyper)regularizer which helps controlling the effective
degrees of freedom of our estimator. To optimize the marginal likelihood we
adapt a Scaled Gradient Projection (SGP) algorithm which is proved to be
significantly computationally cheaper than other first and second order
off-the-shelf optimization methods. The paper also contains an extensive
comparison with many state-of-the-art methods on several Monte-Carlo studies,
which confirms the effectiveness of our procedure
Maximum entropy properties of discrete-time first-order stable spline kernel
The first order stable spline (SS-1) kernel is used extensively in
regularized system identification. In particular, the stable spline estimator
models the impulse response as a zero-mean Gaussian process whose covariance is
given by the SS-1 kernel. In this paper, we discuss the maximum entropy
properties of this prior. In particular, we formulate the exact maximum entropy
problem solved by the SS-1 kernel without Gaussian and uniform sampling
assumptions. Under general sampling schemes, we also explicitly derive the
special structure underlying the SS-1 kernel (e.g. characterizing the
tridiagonal nature of its inverse), also giving to it a maximum entropy
covariance completion interpretation. Along the way similar maximum entropy
properties of the Wiener kernel are also given
Entropy of Overcomplete Kernel Dictionaries
In signal analysis and synthesis, linear approximation theory considers a
linear decomposition of any given signal in a set of atoms, collected into a
so-called dictionary. Relevant sparse representations are obtained by relaxing
the orthogonality condition of the atoms, yielding overcomplete dictionaries
with an extended number of atoms. More generally than the linear decomposition,
overcomplete kernel dictionaries provide an elegant nonlinear extension by
defining the atoms through a mapping kernel function (e.g., the gaussian
kernel). Models based on such kernel dictionaries are used in neural networks,
gaussian processes and online learning with kernels.
The quality of an overcomplete dictionary is evaluated with a diversity
measure the distance, the approximation, the coherence and the Babel measures.
In this paper, we develop a framework to examine overcomplete kernel
dictionaries with the entropy from information theory. Indeed, a higher value
of the entropy is associated to a further uniform spread of the atoms over the
space. For each of the aforementioned diversity measures, we derive lower
bounds on the entropy. Several definitions of the entropy are examined, with an
extensive analysis in both the input space and the mapped feature space.Comment: 10 page
Regularization and Bayesian Learning in Dynamical Systems: Past, Present and Future
Regularization and Bayesian methods for system identification have been
repopularized in the recent years, and proved to be competitive w.r.t.
classical parametric approaches. In this paper we shall make an attempt to
illustrate how the use of regularization in system identification has evolved
over the years, starting from the early contributions both in the Automatic
Control as well as Econometrics and Statistics literature. In particular we
shall discuss some fundamental issues such as compound estimation problems and
exchangeability which play and important role in regularization and Bayesian
approaches, as also illustrated in early publications in Statistics. The
historical and foundational issues will be given more emphasis (and space), at
the expense of the more recent developments which are only briefly discussed.
The main reason for such a choice is that, while the recent literature is
readily available, and surveys have already been published on the subject, in
the author's opinion a clear link with past work had not been completely
clarified.Comment: Plenary Presentation at the IFAC SYSID 2015. Submitted to Annual
Reviews in Contro
- …