Gaussian processes for state space models and change point detection

This thesis details several applications of Gaussian processes (GPs) for enhanced time series modeling. We first cover different approaches for using Gaussian processes in time series problems. These are extended to the state space approach to time series in two different problems. We also combine Gaussian processes and Bayesian online change point detection (BOCPD) to increase the generality of the Gaussian process time series methods. These methodologies are evaluated on predictive performance on six real world data sets, which include three environmental data sets, one financial, one biological, and one from industrial well drilling. Gaussian processes are capable of generalizing standard linear time series models. We cover two approaches: the Gaussian process time series model (GPTS) and the autoregressive Gaussian process (ARGP). We cover a variety of methods that greatly reduce the computational and memory complexity of Gaussian process approaches, which are generally cubic in computational complexity. Two different improvements to state space based approaches are covered. First, Gaussian process inference and learning (GPIL) generalizes linear dynamical systems (LDS), for which the Kalman filter is based, to general nonlinear systems for nonparametric system identification. Second, we address pathologies in the unscented Kalman filter (UKF). We use Gaussian process optimization (GPO) to learn UKF settings that minimize the potential for sigma point collapse. We show how to embed mentioned Gaussian process approaches to time series into a change point framework. Old data, from an old regime, that hinders predictive performance is automatically and elegantly phased out. The computational improvements for Gaussian process time series approaches are of even greater use in the change point framework. We also present a supervised framework learning a change point model when change point labels are available in training.I would like to thank Rockwell Collins, formerly DataPath, Inc., which funded my studentship

Phenotype Extraction: Estimation and Biometrical Genetic Analysis of Individual Dynamics

Within-person data can exhibit a virtually limitless variety of statistical patterns, but it can be difficult to distinguish meaningful features from statistical artifacts. Studies of complex traits have previously used genetic signals like twin-based heritability to distinguish between the two. This dissertation is a collection of studies applying state-space modeling to conceptualize and estimate novel phenotypic constructs for use in psychiatric research and further biometrical genetic analysis. The aims are to: (1) relate control theoretic concepts to health-related phenotypes; (2) design statistical models that formally define those phenotypes; (3) estimate individual phenotypic values from time series data; (4) consider hierarchical methods for biometrical genetic analysis of individual phenotypic variation

Time-varying Autoregressive Modeling of Nonstationary Signals

Nonstationary signal modeling is a research topic of practical interest. In this thesis, we adopt a time-varying (TV) autoregressive (AR) model using the basis function (BF) parameter estimation method for nonstationary process identification and instantaneous frequency (IF) estimation. The current TVAR model in direct form (DF) with the blockwise least-squares and recursive weighted-least-squares BF methods perform equivalently well in signal modeling, but the large estimation error may cause temporary instabilities of the estimated model. To achieve convenient model stability monitoring and pole tracking, the TVAR model in cascade form (CF) was proposed through the parameterization in terms of TV poles (represented by second order section coefficients, Cartesian coordinates, Polar coordinates), where the time variation of each pole parameter is assumed to be the linear combination of BFs. The nonlinear system equations for the TVAR model in CF are solved iteratively using the Gauss-Newton algorithm. Using the CF, the model stability is easily controlled by constraining the estimated TV poles within the unit circle. The CF model shows similar performance trends to the DF model using the recursive BF method, and the TV pole representation in Cartesian coordinates outperforms all other representations. The individual frequency variation can be finely tracked using the CF model, when several frequency components are present in the signal. Simulations were carried on synthetic sinusoidal signals with different frequency variations for IF estimation. For the TVAR model in DF (blockwise), the basis dimension (BD) is an important factor on frequency estimation accuracy. For the TVAR model in DF (recursive) and CF (Cartesian), the influences of BD are negligible. The additive white noise in the observed signal degrades the estimation performance, and the the noise effects can be reduce by using higher model order. Experiments were carried on the real electromyography (EMG) data for frequency estimation in the analysis of muscle fatigue. The TVAR modeling methods show equivalent performance to the conventional Fourier transform method

Human Motion Trajectory Prediction: A Survey

With growing numbers of intelligent autonomous systems in human environments, the ability of such systems to perceive, understand and anticipate human behavior becomes increasingly important. Specifically, predicting future positions of dynamic agents and planning considering such predictions are key tasks for self-driving vehicles, service robots and advanced surveillance systems. This paper provides a survey of human motion trajectory prediction. We review, analyze and structure a large selection of work from different communities and propose a taxonomy that categorizes existing methods based on the motion modeling approach and level of contextual information used. We provide an overview of the existing datasets and performance metrics. We discuss limitations of the state of the art and outline directions for further research.Comment: Submitted to the International Journal of Robotics Research (IJRR), 37 page

Sensor fusion with Gaussian processes

This thesis presents a new approach to multi-rate sensor fusion for (1) user matching and (2) position stabilisation and lag reduction. The Microsoft Kinect sensor and the inertial sensors in a mobile device are fused with a Gaussian Process (GP) prior method. We present a Gaussian Process prior model-based framework for multisensor data fusion and explore the use of this model for fusing mobile inertial sensors and an external position sensing device. The Gaussian Process prior model provides a principled mechanism for incorporating the low-sampling-rate position measurements and the high-sampling-rate derivatives in multi-rate sensor fusion, which takes account of the uncertainty of each sensor type. We explore the complementary properties of the Kinect sensor and the built-in inertial sensors in a mobile device and apply the GP framework for sensor fusion in the mobile human-computer interaction area. The Gaussian Process prior model-based sensor fusion is presented as a principled probabilistic approach to dealing with position uncertainty and the lag of the system, which are critical for indoor augmented reality (AR) and other location-aware sensing applications. The sensor fusion helps increase the stability of the position and reduce the lag. This is of great benefit for improving the usability of a human-computer interaction system. We develop two applications using the novel and improved GP prior model. (1) User matching and identification. We apply the GP model to identify individual users, by matching the observed Kinect skeletons with the sensed inertial data from their mobile devices. (2) Position stabilisation and lag reduction in a spatially aware display application for user performance improvement. We conduct a user study. Experimental results show the improved accuracy of target selection, and reduced delay from the sensor fusion system, allowing the users to acquire the target more rapidly, and with fewer errors in comparison with the Kinect filtered system. They also reported improved performance in subjective questions. The two applications can be combined seamlessly in a proxemic interaction system as identification of people and their positions in a room-sized environment plays a key role in proxemic interactions

Multi-scale Attention Flow for Probabilistic Time Series Forecasting

The probability prediction of multivariate time series is a notoriously challenging but practical task. On the one hand, the challenge is how to effectively capture the cross-series correlations between interacting time series, to achieve accurate distribution modeling. On the other hand, we should consider how to capture the contextual information within time series more accurately to model multivariate temporal dynamics of time series. In this work, we proposed a novel non-autoregressive deep learning model, called Multi-scale Attention Normalizing Flow(MANF), where we integrate multi-scale attention and relative position information and the multivariate data distribution is represented by the conditioned normalizing flow. Additionally, compared with autoregressive modeling methods, our model avoids the influence of cumulative error and does not increase the time complexity. Extensive experiments demonstrate that our model achieves state-of-the-art performance on many popular multivariate datasets

Data cleaning and knowledge discovery in process data

This dissertation presents several methods for overcoming the Big Data challenges, with an emphasis on data cleaning and knowledge discovery in process data. Data cleaning and knowledge discovery is chosen as a main research area here due to its importance from both theoretical and practical points of view. Theoretical background and recent developments of data cleaning methods are reviewed from four aspects: missing data imputation, outlier detection, noise removal and time delay estimation. Moreover, the impact of contaminated data on model performance and corresponding improvement obtained by data cleaning methods are analyzed through both simulated and industrial case studies. The results provide a starting point for further advanced methodology development. It is hard to find a universally applicable method for data cleaning since every data set may have its own distinctive features. Thus, we have to customize available methods so that the quality of the data set is guaranteed. An integrated data cleaning scheme is proposed, which incorporates model building and performance evaluation, to provide guidance in tuning the parameters of data cleaning methods and prevent over-cleaning. A case study based on industrial data has been used to verify the feasibility and effectiveness of the proposed new method, during which a partial least squares (PLS) model was built and three univariate data cleaning procedures is tested. A time series Kalman filter (TSKF) is proposed that successfully handles outlier detection in dynamic systems, where normal process changes often mask the existence of outliers. The TSKF method combines a time series model fitting procedure with a modified Kalman filter to deal with additive outlier (AO) and innovational outlier (IO) detection problems in dynamic process data set. A comparative analysis of TSKF and available methods is performed on simulated and real chemical plant data. Root cause diagnosis of plant-wide oscillations, as a concrete example of data cleaning and knowledge discovery in the process data, is provided. Plant-wide oscillations can negatively influence the overall control performance of the process and the detection results are often affected by noise at different frequency ranges. To address such a problem, an information transfer method combining spectral envelope algorithm with spectral transfer entropy is proposed to detect and diagnose such oscillations within a specific frequency range, mitigating the effects from measurement noise. The feasibility and effectiveness of the proposed method are verified and compared with available methods through both simulated and industrial case studies.Chemical Engineerin

PREDICTION OF RESPIRATORY MOTION

Radiation therapy is a cancer treatment method that employs high-energy radiation beams to destroy cancer cells by damaging the ability of these cells to reproduce. Thoracic and abdominal tumors may change their positions during respiration by as much as three centimeters during radiation treatment. The prediction of respiratory motion has become an important research area because respiratory motion severely affects precise radiation dose delivery. This study describes recent radiotherapy technologies including tools for measuring target position during radiotherapy and tracking-based delivery systems. In the first part of our study we review three prediction approaches of respiratory motion, i.e., model-based methods, model-free heuristic learning algorithms, and hybrid methods. In the second part of our work we propose respiratory motion estimation with hybrid implementation of extended Kalman filter. The proposed method uses the recurrent neural network as the role of the predictor and the extended Kalman filter as the role of the corrector. In the third part of our work we further extend our research work to present customized prediction of respiratory motion with clustering from multiple patient interactions. For the customized prediction we construct the clustering based on breathing patterns of multiple patients using the feature selection metrics that are composed of a variety of breathing features. In the fourth part of our work we retrospectively categorize breathing data into several classes and propose a new approach to detect irregular breathing patterns using neural networks. We have evaluated the proposed new algorithm by comparing the prediction overshoot and the tracking estimation value. The experimental results of 448 patients’ breathing patterns validated the proposed irregular breathing classifier