The kernel Kalman rule: efficient nonparametric inference with recursive least squares

Nonparametric inference techniques provide promising tools for probabilistic reasoning in high-dimensional nonlinear systems. Most of these techniques embed distributions into reproducing kernel Hilbert spaces (RKHS) and rely on the kernel Bayes’ rule (KBR) to manipulate the embeddings. However, the computational demands of the KBR scale poorly with the number of samples and the KBR often suffers from numerical instabilities. In this paper, we present the kernel Kalman rule (KKR) as an alternative to the KBR. The derivation of the KKR is based on recursive least squares, inspired by the derivation of the Kalman innovation update. We apply the KKR to filtering tasks where we use RKHS embeddings to represent the belief state, resulting in the kernel Kalman filter (KKF). We show on a nonlinear state estimation task with high dimensional observations that our approach provides a significantly improved estimation accuracy while the computational demands are significantly decreased

Using Mean Embeddings for State Estimation and Reinforcement Learning

To act in complex, high-dimensional environments, autonomous systems require versatile state estimation techniques and compact state representations. State estimation is crucial when the system only has access to stochastic measurements or partial observations. Furthermore, in combination with models of the system such techniques allow to predict the future which enables the system to asses the outcome of possible decisions. Compact state representations alleviate the curse of dimensionality by distilling the important information from high-dimensional observations. Due to noisy sensory information and non-perfect models of the system, estimates of the state never reflect the true state perfectly but are always subject to errors. The natural choice to incorporate the uncertainty about the state estimate is to use a probability distribution as representation. This results in the so called belief state. High-dimensional observations, for example images, often contain much less information than conveyed by their dimensionality. But also if all the information is necessary to describe the state of the system—for example, think of the state of a swarm with the positions of all agents—a less complex description might be a sufficient representation. In such situations, finding the generative distribution that explains the state would give a much more compact while informative representation. Traditionally, parametric distributions have been used as state representations such as most prevalently the Gaussian distribution. However, in many cases a unimodal distribution might not be sufficient to represent the belief state. Using multi-modal probability distributions, instead, requires more advanced approaches such as mixture models or particle-based Monte Carlo methods. Learning mixture models is however not straight-forward and often results in locally optimal solutions. Similarly, maintaining a good population of particles during inference is a complicated and cumbersome process. A third approach is kernel density estimation which is located at the intersection of mixture models and particle-based approaches. Still, performing inference with any of these approaches requires heuristics that lead to poor performance and a limited scalability to higher dimensional spaces. A recent technique that alleviates this problem are the embeddings of probability distributions into reproducing kernel Hilbert spaces (RKHS). Conditional distributions can be embedded as operators based on which a framework for inference has been presented that allows to apply the sum rule, the product rule and Bayes’ rule entirely in Hilbert space. Using sample based estimators and the kernel-trick of the representer theorem allows to represent the operations as vector-matrix manipulations. The contributions of this thesis are based on or inspired by the embeddings of distributions into reproducing kernel Hilbert spaces. In the first part of this thesis, I propose additions to the framework for nonparametric inference that allow the inference operators to scale more gracefully with the number of samples in the training set. The first contribution is an alternative approach to the conditional embedding operator formulated as a least-squares problem i which allows to use only a subset of the data as representation while using the full data set to learn the conditional operator. I call this operator the subspace conditional embedding operator. Inspired by the least-squares derivations of the Kalman filter, I furthermore propose an alternative operator for Bayesian updates in Hilbert space, the kernel Kalman rule. This alternative approach is numerically more robust than the kernel Bayes rule presented in the framework for non-parametric inference and scales better with the number of samples. Based on the kernel Kalman rule, I derive the kernel Kalman filter and the kernel forward-backward smoother to perform state estimation, prediction and smoothing based on Hilbert space embeddings of the belief state. This representation is able to capture multi-modal distributions and inference resolves--due to the kernel trick--into easy matrix manipulations. In the second part of this thesis, I propose a representation for large sets of homogeneous observations. Specifically, I consider the problem of learning a controller for object assembly and object manipulation with a robotic swarm. I assume a swarm of homogeneous robots that are controlled by a common input signal, e.g., the gradient of a light source or a magnetic field. Learning policies for swarms is a challenging problem since the state space grows with the number of agents and becomes quickly very high dimensional. Furthermore, the exact number of agents and the order of the agents in the observation is not important to solve the task. To approach this issue, I propose the swarm kernel which uses a Hilbert space embedding to represent the swarm. Instead of the exact positions of the agents in the swarm, the embedding estimates the generative distribution behind the swarm configuration. The specific agent positions are regarded as samples of this distribution. Since the swarm kernel compares the embeddings of distributions, it can compare swarm configurations with varying numbers of individuals and is invariant to the permutation of the agents. I present a hierarchical approach for solving the object manipulation task where I assume a high-level object assembly policy as given. To learn the low-level object pushing policy, I use the swarm kernel with an actor-critic policy search method. The policies which I learn in simulation can be directly transferred to a real robotic system. In the last part of this thesis, I investigate how we can employ the idea of kernel mean embeddings to deep reinforcement learning. As in the previous part, I consider a variable number of homogeneous observations—such as robot swarms where the number of agents can change. Another example is the representation of 3D structures as point clouds. The number of points in such clouds can vary strongly and the order of the points in a vectorized representation is arbitrary. The common architectures for neural networks have a fixed structure that requires that the dimensionality of inputs and outputs is known in advance. A variable number of inputs can only be processed by applying tricks. To approach this problem, I propose the deep M-embeddings which are inspired by the kernel mean embeddings. The deep M-embeddings provide a network structure to compute a fixed length representation from a variable number of inputs. Additionally, the deep M-embeddings exploit the homogeneous nature of the inputs to reduce the number of parameters in the network and, thus, make the learning easier. Similar to the swarm kernel, the policies learned with the deep M-embeddings can be transferred to different swarm sizes and different number of objects in the environment without further learning

Interactions between gaussian processes and bayesian estimation

L’apprentissage (machine) de modèle et l’estimation d’état sont cruciaux pour interpréter les phénomènes sous-jacents à de nombreuses applications du monde réel. Toutefois, il est souvent difficile d’apprendre le modèle d’un système et de capturer les états latents, efficacement et avec précision, en raison du fait que la connaissance du monde est généralement incertaine. Au cours des dernières années, les approches d’estimation et de modélisation bayésiennes ont été extensivement étudiées afin que l’incertain soit réduit élégamment et de manière flexible. Dans la pratique cependant, différentes limitations au niveau de la modélisation et de l’estimation bayésiennes peuvent détériorer le pouvoir d’interprétation bayésienne. Ainsi, la performance de l’estimation est souvent limitée lorsque le modèle de système manque de souplesse ou/et est partiellement inconnu. De même, la performance de la modélisation est souvent restreinte lorsque l’estimateur Bayésien est inefficace. Inspiré par ces faits, nous proposons d’étudier dans cette thèse, les connections possibles entre modélisation bayésienne (via le processus gaussien) et l’estimation bayésienne (via le filtre de Kalman et les méthodes de Monte Carlo) et comment on pourrait améliorer l’une en utilisant l’autre. À cet effet, nous avons d’abord vu de plus près comment utiliser les processus gaussiens pour l’estimation bayésienne. Dans ce contexte, nous avons utilisé le processus gaussien comme un prior non-paramétrique des modèles et nous avons montré comment cela permettait d’améliorer l’efficacité et la précision de l’estimation bayésienne. Ensuite, nous nous somme intéressé au fait de savoir comment utiliser l’estimation bayésienne pour le processus gaussien. Dans ce cadre, nous avons utilisé différentes estimations bayésiennes comme le filtre de Kalman et les filtres particulaires en vue d’améliorer l’inférence au niveau du processus gaussien. Ceci nous a aussi permis de capturer différentes propriétés au niveau des données d’entrée. Finalement, on s’est intéressé aux interactions dynamiques entre estimation bayésienne et processus gaussien. On s’est en particulier penché sur comment l’estimation bayésienne et le processus gaussien peuvent ”travailler” de manière interactive et complémentaire de façon à améliorer à la fois le modèle et l’estimation. L’efficacité de nos approches, qui contribuent à la fois au processus gaussien et à l’estimation bayésienne, est montrée au travers d’une analyse mathématique rigoureuse et validée au moyen de différentes expérimentations reflétant des applications réelles.Model learning and state estimation are crucial to interpret the underlying phenomena in many real-world applications. However, it is often challenging to learn the system model and capture the latent states accurately and efficiently due to the fact that the knowledge of the world is highly uncertain. During the past years, Bayesian modeling and estimation approaches have been significantly investigated so that the uncertainty can be elegantly reduced in a flexible probabilistic manner. In practice, however, several drawbacks in both Bayesian modeling and estimation approaches deteriorate the power of Bayesian interpretation. On one hand, the estimation performance is often limited when the system model lacks in flexibility and/or is partially unknown. On the other hand, the modeling performance is often restricted when a Bayesian estimator is not efficient and/or accurate. Inspired by these facts, we propose Interactions Between Gaussian Processes and Bayesian Estimation where we investigate the novel connections between Bayesian model (Gaussian processes) and Bayesian estimator (Kalman filter and Monte Carlo methods) in different directions to address a number of potential difficulties in modeling and estimation tasks. Concretely, we first pay our attention to Gaussian Processes for Bayesian Estimation where a Gaussian process (GP) is used as an expressive nonparametric prior for system models to improve the accuracy and efficiency of Bayesian estimation. Then, we work on Bayesian Estimation for Gaussian Processes where a number of Bayesian estimation approaches, especially Kalman filter and particle filters, are used to speed up the inference efficiency of GP and also capture the distinct input-dependent data properties. Finally, we investigate Dynamical Interaction Between Gaussian Processes and Bayesian Estimation where GP modeling and Bayesian estimation work in a dynamically interactive manner so that GP learner and Bayesian estimator are positively complementary to improve the performance of both modeling and estimation. Through a number of mathematical analysis and experimental demonstrations, we show the effectiveness of our approaches which contribute to both GP and Bayesian estimation

Computing Arrival Cost Parameters in Moving Horizon Estimation Using Sampling Based Filters

Moving horizon estimation (MHE) is a numerical optimization based approach to state estimation, where the joint probability density function (pdf) of a finite state trajectory is sought, which is conditioned on a moving horizon of measurements. The joint conditional pdf depends on the a priori state pdf at the start of the horizon, which is a prediction pdf based on historical data outside the horizon. When the joint pdf is maximized, the arrival cost is a penalty term based on the a priori pdf in the MHE objective function. Traditionally, the a priori pdf is assumed as a multivariate Gaussian pdf and the extended Kalman filter (EKF) and smoother are used to recursively update the mean and covariance. However, transformation of moments through nonlinearity is poorly approximated by linearization, which can result in poor initialization of MHE. Sampling based nonlinear filters completely avoid Taylor series approximations of nonlinearities and attempt to approximate the non-Gaussian state pdf using samples and associated weights or probability mass points. The performance gains of sampling based filters over EKF motivate their use to formulate the arrival cost in MHE. The a priori mean and covariance are more effectively propagated through nonlinearities and the resulting arrival cost term can help to keep the horizon small. It is also possible to find closed-form approximations to the non-Gaussian a priori pdf from the sampling based filters. Thus, more realistic nonparametric arrival cost terms can be included by avoiding the Gaussian assumption. In this paper the use of the deterministic sampling based unscented Kalman filter, the class of random sampling based particle filter and the aggregate Markov chain based cell filter are discussed for initializing MHE. Two simulation examples are included to demonstrate the benefits of these methods over the traditional EKF approach

A semiparametric state space model

This paper considers the problem of estimating a linear univariate Time Series State Space model for which the shape of the distribution of the observation noise is not specified a priori. Although somewhat challenging computationally, the simultaneous estimation of the parameters of the model and the unknown observation noise density is made feasible through a combination of Gaussian-sum Filtering and Smoothing algorithms and Kernel Density Estimation methods. The bottleneck in these calculations consists in avoiding the geometric increase, with time, of the number of simultaneous Kalman filter components. It is the aim of this paper to show that this can be achieved by the use of standard techniques from Cluster Analysis and unsupervised Classification. An empirical illustration of this new methodology is included; this consists in the application of a semiparametric version of the Local Level model to the analysis of the wellknown river Nile data series

Does money matter in inflation forecasting?.

This paper provides the most fully comprehensive evidence to date on whether or not monetary aggregates are valuable for forecasting US inflation in the early to mid 2000s. We explore a wide range of different definitions of money, including different methods of aggregation and different collections of included monetary assets. In our forecasting experiment we use two non-linear techniques, namely, recurrent neural networks and kernel recursive least squares regression - techniques that are new to macroeconomics. Recurrent neural networks operate with potentially unbounded input memory, while the kernel regression technique is a finite memory predictor. The two methodologies compete to find the best fitting US inflation forecasting models and are then compared to forecasts from a naive random walk model. The best models were non-linear autoregressive models based on kernel methods. Our findings do not provide much support for the usefulness of monetary aggregates in forecasting inflation