Efficient state-space inference of periodic latent force models

Latent force models (LFM) are principled approaches to incorporating solutions to differen-tial equations within non-parametric inference methods. Unfortunately, the developmentand application of LFMs can be inhibited by their computational cost, especially whenclosed-form solutions for the LFM are unavailable, as is the case in many real world prob-lems where these latent forces exhibit periodic behaviour. Given this, we develop a newsparse representation of LFMs which considerably improves their computational efficiency,as well as broadening their applicability, in a principled way, to domains with periodic ornear periodic latent forces. Our approach uses a linear basis model to approximate onegenerative model for each periodic force. We assume that the latent forces are generatedfrom Gaussian process priors and develop a linear basis model which fully expresses thesepriors. We apply our approach to model the thermal dynamics of domestic buildings andshow that it is effective at predicting day-ahead temperatures within the homes. We alsoapply our approach within queueing theory in which quasi-periodic arrival rates are mod-elled as latent forces. In both cases, we demonstrate that our approach can be implemented efficiently using state-space methods which encode the linear dynamic systems via LFMs.Further, we show that state estimates obtained using periodic latent force models can re-duce the root mean squared error to 17% of that from non-periodic models and 27% of thenearest rival approach which is the resonator model (S ̈arkk ̈a et al., 2012; Hartikainen et al.,2012.

Efficient State-Space Inference of Periodic Latent Force Models

Latent force models (LFM) are principled approaches to incorporating solutions to differential equations within non-parametric inference methods. Unfortunately, the development and application of LFMs can be inhibited by their computational cost, especially when closed-form solutions for the LFM are unavailable, as is the case in many real world problems where these latent forces exhibit periodic behaviour. Given this, we develop a new sparse representation of LFMs which considerably improves their computational efficiency, as well as broadening their applicability, in a principled way, to domains with periodic or near periodic latent forces. Our approach uses a linear basis model to approximate one generative model for each periodic force. We assume that the latent forces are generated from Gaussian process priors and develop a linear basis model which fully expresses these priors. We apply our approach to model the thermal dynamics of domestic buildings and show that it is effective at predicting day-ahead temperatures within the homes. We also apply our approach within queueing theory in which quasi-periodic arrival rates are modelled as latent forces. In both cases, we demonstrate that our approach can be implemented efficiently using state-space methods which encode the linear dynamic systems via LFMs. Further, we show that state estimates obtained using periodic latent force models can reduce the root mean squared error to 17% of that from non-periodic models and 27% of the nearest rival approach which is the resonator model.Comment: 61 pages, 13 figures, accepted for publication in JMLR. Updates from earlier version occur throughout article in response to JMLR review

Linear latent force models using Gaussian processes.

Purely data-driven approaches for machine learning present difficulties when data are scarce relative to the complexity of the model or when the model is forced to extrapolate. On the other hand, purely mechanistic approaches need to identify and specify all the interactions in the problem at hand (which may not be feasible) and still leave the issue of how to parameterize the system. In this paper, we present a hybrid approach using Gaussian processes and differential equations to combine data-driven modeling with a physical model of the system. We show how different, physically inspired, kernel functions can be developed through sensible, simple, mechanistic assumptions about the underlying system. The versatility of our approach is illustrated with three case studies from motion capture, computational biology, and geostatistics

Multiplicative latent force models

Latent force models (LFM) are a class of flexible models of dynamic systems, combining a simple mechanistic model with the flexibility of an additive inhomogeneous Gaussian process (GP) forcing term. These hybrid models achieve the dual goal of being flexible enough to be broadly applied, even for complex dynamic systems where a full mechanistic model may be hard to motivate, but by also encoding relevant properties of dynamic systems they are better able to model the underlying dynamics and so demonstrate superior generalisation. In this thesis, we consider an extension of this framework which keeps the same general form, a linear ordinary di↵erential equation with time-varying behaviour arising from a set of smooth GPs, but now we allow for multiplicative interactions between the state variables and the GP terms. The result is a semi-parametric modelling framework that allows for the embedding of rich topological structure. Following a brief review of the latent force model, which we note is a particular case of the GP regression model, we introduce our extension with multiplicative interactions which we refer to as the multiplicative latent force model (MLFM). We demonstrate that this class of models allows for the possibility of strong geometric constraints on the pathwise trajectories. This will enable the modelling of systems for which the GP trajectories of the LFM are unsatisfactory. Unfortunately, and as a direct consequence of the strong geometric constraints we have introduced, it is no longer straightforward to carry out inference in these models; therefore the remainder of this thesis is primarily devoted to constructing two methods for carrying out approximate inference for this class of models. The first is referred to as the Bayesian adaptive gradient matching method, and the second is a novel construction based on the method of successive approximations; a theoretical construct used in the standard classical existence and uniqueness theorems for ODEs. After introducing these methods, we demonstrate their accuracy on simulated data, which also allows for an investigation into the regimes in which each of the respective methods can be expected to perform well. Finally, we demonstrate the utility of the MLFM on motion capture data and show that, by using the framework developed in this thesis to allow for the sharing of a smaller number of latent forces between distinct trajectories with specific geometric constraints, we can achieve superior predictive performance than by the modelling of a single trajectory