Regularized System Identification

This open access book provides a comprehensive treatment of recent developments in kernel-based identification that are of interest to anyone engaged in learning dynamic systems from data. The reader is led step by step into understanding of a novel paradigm that leverages the power of machine learning without losing sight of the system-theoretical principles of black-box identification. The authors’ reformulation of the identification problem in the light of regularization theory not only offers new insight on classical questions, but paves the way to new and powerful algorithms for a variety of linear and nonlinear problems. Regression methods such as regularization networks and support vector machines are the basis of techniques that extend the function-estimation problem to the estimation of dynamic models. Many examples, also from real-world applications, illustrate the comparative advantages of the new nonparametric approach with respect to classic parametric prediction error methods. The challenges it addresses lie at the intersection of several disciplines so Regularized System Identification will be of interest to a variety of researchers and practitioners in the areas of control systems, machine learning, statistics, and data science. This is an open access book

Functional Regression

Functional data analysis (FDA) involves the analysis of data whose ideal units of observation are functions defined on some continuous domain, and the observed data consist of a sample of functions taken from some population, sampled on a discrete grid. Ramsay and Silverman's 1997 textbook sparked the development of this field, which has accelerated in the past 10 years to become one of the fastest growing areas of statistics, fueled by the growing number of applications yielding this type of data. One unique characteristic of FDA is the need to combine information both across and within functions, which Ramsay and Silverman called replication and regularization, respectively. This article will focus on functional regression, the area of FDA that has received the most attention in applications and methodological development. First will be an introduction to basis functions, key building blocks for regularization in functional regression methods, followed by an overview of functional regression methods, split into three types: [1] functional predictor regression (scalar-on-function), [2] functional response regression (function-on-scalar) and [3] function-on-function regression. For each, the role of replication and regularization will be discussed and the methodological development described in a roughly chronological manner, at times deviating from the historical timeline to group together similar methods. The primary focus is on modeling and methodology, highlighting the modeling structures that have been developed and the various regularization approaches employed. At the end is a brief discussion describing potential areas of future development in this field

Representing functional data in reproducing Kernel Hilbert Spaces with applications to clustering and classification

Functional data are difficult to manage for many traditional statistical techniques given their very high (or intrinsically infinite) dimensionality. The reason is that functional data are essentially functions and most algorithms are designed to work with (low) finite-dimensional vectors. Within this context we propose techniques to obtain finitedimensional representations of functional data. The key idea is to consider each functional curve as a point in a general function space and then project these points onto a Reproducing Kernel Hilbert Space with the aid of Regularization theory. In this work we describe the projection method, analyze its theoretical properties and propose a model selection procedure to select appropriate Reproducing Kernel Hilbert spaces to project the functional data.Functional data, Reproducing, Kernel Hilbert Spaces, Regularization theory

Robust EM kernel-based methods for linear system identification

Recent developments in system identification have brought attention to regularized kernel-based methods. This type of approach has been proven to compare favorably with classic parametric methods. However, current formulations are not robust with respect to outliers. In this paper, we introduce a novel method to robustify kernel-based system identification methods. To this end, we model the output measurement noise using random variables with heavy-tailed probability density functions (pdfs), focusing on the Laplacian and the Student's t distributions. Exploiting the representation of these pdfs as scale mixtures of Gaussians, we cast our system identification problem into a Gaussian process regression framework, which requires estimating a number of hyperparameters of the data size order. To overcome this difficulty, we design a new maximum a posteriori (MAP) estimator of the hyperparameters, and solve the related optimization problem with a novel iterative scheme based on the Expectation-Maximization (EM) method. In presence of outliers, tests on simulated data and on a real system show a substantial performance improvement compared to currently used kernel-based methods for linear system identification.Comment: Accepted for publication in Automatic

Gravity Field Refinement by Radial Basis Functions from In-situ Satellite Data

In this thesis, an integrated approach is developed for the regional refinement of global gravity field solutions. The analysis concepts are tailored to the in-situ type character of the observations provided by the new satellite missions CHAMP, GRACE, and GOCE. They are able to evaluate data derived from short arcs of the satellite's orbit and, therefore, offer the opportunity to use regional satellite data for the calculation of regional gravity field solutions. The regional character of the approach will be realized at various stages of the analysis procedure. The first step is the design of specifically tailored space localizing basis functions. In order to adapt the basis functions to the signal content to be expected in the gravity field solution, they will be derived from the covariance function of the gravitational potential. To use the basis functions in gravity field modeling, they have to be located at the nodal points of a spherical grid; therefore investigations will be performed regarding a suitable choice of such a nodal point distribution. Another important aspect in the regional gravity field analysis approach is the downward continuation process. In this context, a regionally adapted regularization will be introduced which assigns different regularization matrices to geographical areas with varying signal content. Regularization parameters individually determined for each region take into account the varying frequency behavior, allowing to extract additional information out of a given data set. To conclude the analysis chain, an approach will be described that combines regional solutions with global coverage to obtain a global solution and to derive the corresponding spherical harmonic coefficients by means of the Gauss-Legendre quadrature method. The capability of the method will be demonstrated by its successful application to real data provided by CHAMP and GRACE and to a simulation scenario based on a combination of GRACE and GOCE observations.Verfeinerungen des Gravitationsfeldes mit radialen Basisfunktionen aus in-situ Satellitendaten In der vorliegenden Arbeit wird ein ganzheitliches Konzept für die regionale Verfeinerung globaler Gravitationsfeldmodelle entwickelt. Die dazu verwendeten Analyseverfahren sind dem in-situ Charakter der Beobachtungen der neuen Satellitenmissionen CHAMP, GRACE und GOCE angepasst. Sie beruhen auf kurzen Bahnbögen und ermöglichen somit die Berechnung regionaler Gravitationsfeldmodelle aus regional begrenzten Satellitendaten. Der regionale Charakter des Ansatzes wird dabei auf verschiedenen Ebenen des Analyseprozesses realisiert. Der erste Schritt ist die Entwicklung angepasster orts-lokalisierender Basisfunktionen. Diese sollen das Frequenzverhalten des zu bestimmenden Gravitationsfeldes widerspiegeln; sie werden daher aus der Kovarianzfunktion des Gravitationspotentials abgeleitet. Um die Basisfunktionen für die Schwerefeldmodellierung zu verwenden, müssen sie an den Knotenpunkten eines sphärischen Gitters angeordnet werden. Daher werden Untersuchungen durchgeführt, welche Punktverteilung für diese Aufgabe besonders geeignet ist. Einen wichtigen Aspekt bei der regionalen Gravi-tationsfeldanalyse stellt der Fortsetzungsprozess nach unten dar. In diesem Zusammenhang wird ein regional angepasstes Regularisierungsverfahren entwickelt, das verschiedene Regularisierungsmatrizen für regionale Gebiete mit unterschiedlichem Schwerefeldsignal ermöglicht. Individuell angepasste Regularisierungsparameter berücksichtigen den variierenden Signalinhalt, wodurch erreicht wird, dass zusätzliche Informationen aus einem gegebenen Datensatz extrahiert werden können. Schließlich wird ein Ansatz vorgestellt, der regionale Lösungen mit globaler Überdeckung zu einer globalen Lösung zusammenfügt und die zugehörigen sphärischen harmonischen Koeffizienten mit Hilfe der Gauss-Legendre-Quadratur berechnet. Die Leistungsfähigkeit des beschriebenen Ansatzes wird durch eine erfolgreiche Anwendung auf die Echtdatenanalyse aus Daten der Satellitenmissionen CHAMP und GRACE und auf ein Simulationsszenario aus einer Kombination simulierter GRACE- und GOCE-Beobachtungen verdeutlicht

Representing functional data in reproducing Kernel Hilbert Spaces with applications to clustering and classification

Functional data are difficult to manage for many traditional statistical techniques given their very high (or intrinsically infinite) dimensionality. The reason is that functional data are essentially functions and most algorithms are designed to work with (low) finite-dimensional vectors. Within this context we propose techniques to obtain finitedimensional representations of functional data. The key idea is to consider each functional curve as a point in a general function space and then project these points onto a Reproducing Kernel Hilbert Space with the aid of Regularization theory. In this work we describe the projection method, analyze its theoretical properties and propose a model selection procedure to select appropriate Reproducing Kernel Hilbert spaces to project the functional data