12 research outputs found
Ensemble Kalman Filter (EnKF) for Reinforcement Learning (RL)
This paper is concerned with the problem of representing and learning the
optimal control law for the linear quadratic Gaussian (LQG) optimal control
problem. In recent years, there is a growing interest in re-visiting this
classical problem, in part due to the successes of reinforcement learning (RL).
The main question of this body of research (and also of our paper) is to
approximate the optimal control law {\em without} explicitly solving the
Riccati equation. For this purpose, a novel simulation-based algorithm, namely
an ensemble Kalman filter (EnKF), is introduced in this paper. The algorithm is
used to obtain formulae for optimal control, expressed entirely in terms of the
EnKF particles. For the general partially observed LQG problem, the proposed
EnKF is combined with a standard EnKF (for the estimation problem) to obtain
the optimal control input based on the use of the separation principle. A
nonlinear extension of the algorithm is also discussed which clarifies the
duality roots of the proposed EnKF. The theoretical results and algorithms are
illustrated with numerical experiments
The Hitchhiker's Guide to Nonlinear Filtering
Nonlinear filtering is the problem of online estimation of a dynamic hidden
variable from incoming data and has vast applications in different fields,
ranging from engineering, machine learning, economic science and natural
sciences. We start our review of the theory on nonlinear filtering from the
simplest `filtering' task we can think of, namely static Bayesian inference.
From there we continue our journey through discrete-time models, which is
usually encountered in machine learning, and generalize to and further
emphasize continuous-time filtering theory. The idea of changing the
probability measure connects and elucidates several aspects of the theory, such
as the parallels between the discrete- and continuous-time problems and between
different observation models. Furthermore, it gives insight into the
construction of particle filtering algorithms. This tutorial is targeted at
scientists and engineers and should serve as an introduction to the main ideas
of nonlinear filtering, and as a segway to more advanced and specialized
literature.Comment: 64 page