Nonparametric Infinite Horizon Kullback-Leibler Stochastic Control

We present two nonparametric approaches to Kullback-Leibler (KL) control, or linearly-solvable Markov decision problem (LMDP) based on Gaussian processes (GP) and Nystr\"{o}m approximation. Compared to recently developed parametric methods, the proposed data-driven frameworks feature accurate function approximation and efficient on-line operations. Theoretically, we derive the mathematical connection of KL control based on dynamic programming with earlier work in control theory which relies on information theoretic dualities for the infinite time horizon case. Algorithmically, we give explicit optimal control policies in nonparametric forms, and propose on-line update schemes with budgeted computational costs. Numerical results demonstrate the effectiveness and usefulness of the proposed frameworks

Optimal Navigation Functions for Nonlinear Stochastic Systems

This paper presents a new methodology to craft navigation functions for nonlinear systems with stochastic uncertainty. The method relies on the transformation of the Hamilton-Jacobi-Bellman (HJB) equation into a linear partial differential equation. This approach allows for optimality criteria to be incorporated into the navigation function, and generalizes several existing results in navigation functions. It is shown that the HJB and that existing navigation functions in the literature sit on ends of a spectrum of optimization problems, upon which tradeoffs may be made in problem complexity. In particular, it is shown that under certain criteria the optimal navigation function is related to Laplace's equation, previously used in the literature, through an exponential transform. Further, analytical solutions to the HJB are available in simplified domains, yielding guidance towards optimality for approximation schemes. Examples are used to illustrate the role that noise, and optimality can potentially play in navigation system design.Comment: Accepted to IROS 2014. 8 Page

Weakly nonlinear stability analysis of MHD channel flow using an efficient numerical approach

We analyze weakly nonlinear stability of a flow of viscous conducting liquid driven by pressure gradient in the channel between two parallel walls subject to a transverse magnetic field. Using a non-standard numerical approach, we compute the linear growth rate correction and the first Landau coefficient, which in a sufficiently strong magnetic field vary with the Hartmann number as $\mu_{1}\sim(0.814-\mathrm{i}19.8)\times10^{-3}\textit{Ha}$ and $\mu_{2}\sim(2.73-\mathrm{i}1.50)\times10^{-5}\textit{Ha}^{-4}$. These coefficients describe a subcritical transverse velocity perturbation with the equilibrium amplitude $|A|^{2}=\Re[\mu_{1}]/\Re[\mu_{2}](\textit{Re}_{c}-\textit{Re})\sim29.8\textit{Ha}^{5}(\textit{Re}_{c}-\textit{Re})$ which exists at Reynolds numbers below the linear stability threshold $\textit{Re}_{c}\sim 4.83\times10^{4}\textit{Ha}.$ We find that the flow remains subcritically unstable regardless of the magnetic field strength. Our method for computing Landau coefficients differs from the standard one by the application of the solvability condition to the discretized rather than continuous problem. This allows us to bypass both the solution of the adjoint problem and the subsequent evaluation of the integrals defining the inner products, which results in a significant simplification of the method.Comment: 16 pages, 10 figures, revised version (to appear in Phys Fluids

Hierarchical Linearly-Solvable Markov Decision Problems

We present a hierarchical reinforcement learning framework that formulates each task in the hierarchy as a special type of Markov decision process for which the Bellman equation is linear and has analytical solution. Problems of this type, called linearly-solvable MDPs (LMDPs) have interesting properties that can be exploited in a hierarchical setting, such as efficient learning of the optimal value function or task compositionality. The proposed hierarchical approach can also be seen as a novel alternative to solving LMDPs with large state spaces. We derive a hierarchical version of the so-called Z-learning algorithm that learns different tasks simultaneously and show empirically that it significantly outperforms the state-of-the-art learning methods in two classical hierarchical reinforcement learning domains: the taxi domain and an autonomous guided vehicle task.Comment: 11 pages, 6 figures, 26th International Conference on Automated Planning and Schedulin

Solving the inverse problem of high numerical aperture focusing using vector Slepian harmonics and vector Slepian multipole fields

A technique using vector Slepian harmonics and multipole fields is presented for a general treatment of the inverse problem of high numerical aperture focusing. A prescribed intensity distribution or electric field distribution in the focal volume is approximated using numerical optimization and the corresponding illuminating field at the entrance pupil is constructed. Three examples from the recent literature have been chosen to illustrate the method