Gaussian processes for state space models and change point detection

This thesis details several applications of Gaussian processes (GPs) for enhanced time series modeling. We first cover different approaches for using Gaussian processes in time series problems. These are extended to the state space approach to time series in two different problems. We also combine Gaussian processes and Bayesian online change point detection (BOCPD) to increase the generality of the Gaussian process time series methods. These methodologies are evaluated on predictive performance on six real world data sets, which include three environmental data sets, one financial, one biological, and one from industrial well drilling. Gaussian processes are capable of generalizing standard linear time series models. We cover two approaches: the Gaussian process time series model (GPTS) and the autoregressive Gaussian process (ARGP). We cover a variety of methods that greatly reduce the computational and memory complexity of Gaussian process approaches, which are generally cubic in computational complexity. Two different improvements to state space based approaches are covered. First, Gaussian process inference and learning (GPIL) generalizes linear dynamical systems (LDS), for which the Kalman filter is based, to general nonlinear systems for nonparametric system identification. Second, we address pathologies in the unscented Kalman filter (UKF). We use Gaussian process optimization (GPO) to learn UKF settings that minimize the potential for sigma point collapse. We show how to embed mentioned Gaussian process approaches to time series into a change point framework. Old data, from an old regime, that hinders predictive performance is automatically and elegantly phased out. The computational improvements for Gaussian process time series approaches are of even greater use in the change point framework. We also present a supervised framework learning a change point model when change point labels are available in training.I would like to thank Rockwell Collins, formerly DataPath, Inc., which funded my studentship

Safe model-based design of experiments using Gaussian processes

The construction of kinetic models has become an indispensable step in developing and scale-up of processes in the industry. Model-based design of experiments (MBDoE) has been widely used to improve parameter precision in nonlinear dynamic systems. Such a framework needs to account for both parametric and structural uncertainty, as the physical or safety constraints imposed on the system may well turn out to be violated, leading to unsafe experimental conditions when an optimally designed experiment is performed. In this work, Gaussian processes are utilized in a two-fold manner: 1) to quantify the uncertainty realization of the physical system and calculate the plant-model mismatch, 2) to compute the optimal experimental design while accounting for the parametric uncertainty. TheOur proposed method, Gaussian process-based MBDoE (GP-MBDoE), guarantees the probabilistic satisfaction of the constraints in the context of the model-based design of experiments. GP-MBDoE is assisted with the use of adaptive trust regions to facilitate a satisfactory local approximation. The proposed method can allow the design of optimal experiments starting from limited preliminary knowledge of the parameter set, leading to a safe exploration of the parameter space. This method’s performance is demonstrated through illustrative case studies regarding the parameter identification of kinetic models in flow reactors

Learning nonparametric Volterra kernels with Gaussian processes

This paper introduces a method for the nonparametric Bayesian learning of nonlinear operators, through the use of the Volterra series with kernels represented using Gaussian processes (GPs), which we term the nonparametric Volterra kernels model (NVKM). When the input function to the operator is unobserved and has a GP prior, the NVKM constitutes a powerful method for both single and multiple output regression, and can be viewed as a nonlinear and nonparametric latent force model. When the input function is observed, the NVKM can be used to perform Bayesian system identification. We use recent advances in efficient sampling of explicit functions from GPs to map process realisations through the Volterra series without resorting to numerical integration, allowing scalability through doubly stochastic variational inference, and avoiding the need for Gaussian approximations of the output processes. We demonstrate the performance of the model for both multiple output regression and system identification using standard benchmarks

Stochastic response determination and spectral identification of complex dynamic structural systems

Uncertainty propagation in engineering mechanics and dynamics is a highly challenging problem that requires development of analytical/numerical techniques for determining the stochastic response of complex engineering systems. In this regard, although Monte Carlo simulation (MCS) has been the most versatile technique for addressing the above problem, it can become computationally daunting when faced with high-dimensional systems or with computing very low probability events. Thus, there is a demand for pursuing more computationally efficient methodologies. Further, most structural systems are likely to exhibit nonlinear and time-varying behavior when subjected to extreme events such as severe earthquake, wind and sea wave excitations. In such cases, a reliable identification approach is behavior and for assessing its reliability. Current work addresses two research themes in the field of stochastic engineering dynamics related to the above challenges. In the first part of the dissertation, the recently developedWiener Path Integral (WPI) technique for determining the joint response probability density function (PDF) of nonlinear systems subject to Gaussian white noise excitation is generalized herein to account for non-white, non-Gaussian, and non-stationary excitation processes. Specifically, modeling the excitation process as the output of a filter equation with Gaussian white noise as its input, it is possible to define an augmented response vector process to be considered in the WPI solution technique. A significant advantage relates to the fact that the technique is still applicable even for arbitrary excitation power spectrum forms. In such cases, it is shown that the use of a filter approximation facilitates the implementation of the WPI technique in a straightforward manner, without compromising its accuracy necessarily. Further, in addition to dynamical systems subject to stochastic excitation, the technique can also account for a special class of engineering mechanics problems where the media properties are modeled as non-Gaussian and non-homogeneous stochastic fields. Several numerical examples pertaining to both single- and multi-degree-of freedom systems are considered, including a marine structural system exposed to flow-induced non-white excitation, as well as a beam with a non-Gaussian and non-homogeneous Young’s modulus. Comparisons with MCS data demonstrate the accuracy of the technique. In the second part of the dissertation, a novel multiple-input/single-output (MISO) system identification technique is developed for parameter identification of nonlinear time-variant multi-degree-of-freedom oscillators with fractional derivative terms subject to incomplete non-stationary data. The technique utilizes a representation of the nonlinear restoring forces as a set of parallel linear subsystems. In this regard, the oscillator is transformed into an equivalent MISO system in the wavelet domain. Next, a recently developed L1-norm minimization procedure based on compressive sampling theory is applied for determining the wavelet coefficients of the available incomplete non-stationary input-output (excitation-response) data. Finally, these wavelet coefficients are utilized to determine appropriately defined time- and frequency-dependent wavelet based frequency response functions and related oscillator parameters. A nonlinear time-variant system with fractional derivative elements is used as a numerical example to demonstrate the reliability of the technique even in cases of noise corrupted and incomplete data

Integrated Pre-Processing for Bayesian Nonlinear System Identification with Gaussian Processes

We introduce GP-FNARX: a new model for nonlinear system identification based on a nonlinear autoregressive exogenous model (NARX) with filtered regressors (F) where the nonlinear regression problem is tackled using sparse Gaussian processes (GP). We integrate data pre-processing with system identification into a fully automated procedure that goes from raw data to an identified model. Both pre-processing parameters and GP hyper-parameters are tuned by maximizing the marginal likelihood of the probabilistic model. We obtain a Bayesian model of the system's dynamics which is able to report its uncertainty in regions where the data is scarce. The automated approach, the modeling of uncertainty and its relatively low computational cost make of GP-FNARX a good candidate for applications in robotics and adaptive control.Comment: Proceedings of the 52th IEEE International Conference on Decision and Control (CDC), Firenze, Italy, December 201

Gaussian process model based predictive control

Gaussian process models provide a probabilistic non-parametric modelling approach for black-box identification of non-linear dynamic systems. The Gaussian processes can highlight areas of the input space where prediction quality is poor, due to the lack of data or its complexity, by indicating the higher variance around the predicted mean. Gaussian process models contain noticeably less coefficients to be optimized. This paper illustrates possible application of Gaussian process models within model-based predictive control. The extra information provided within Gaussian process model is used in predictive control, where optimization of control signal takes the variance information into account. The predictive control principle is demonstrated on control of pH process benchmark