Domain Decomposition for Stochastic Optimal Control

This work proposes a method for solving linear stochastic optimal control (SOC) problems using sum of squares and semidefinite programming. Previous work had used polynomial optimization to approximate the value function, requiring a high polynomial degree to capture local phenomena. To improve the scalability of the method to problems of interest, a domain decomposition scheme is presented. By using local approximations, lower degree polynomials become sufficient, and both local and global properties of the value function are captured. The domain of the problem is split into a non-overlapping partition, with added constraints ensuring $C^1$ continuity. The Alternating Direction Method of Multipliers (ADMM) is used to optimize over each domain in parallel and ensure convergence on the boundaries of the partitions. This results in improved conditioning of the problem and allows for much larger and more complex problems to be addressed with improved performance.Comment: 8 pages. Accepted to CDC 201

Nonparametric Infinite Horizon Kullback-Leibler Stochastic Control

We present two nonparametric approaches to Kullback-Leibler (KL) control, or linearly-solvable Markov decision problem (LMDP) based on Gaussian processes (GP) and Nystr\"{o}m approximation. Compared to recently developed parametric methods, the proposed data-driven frameworks feature accurate function approximation and efficient on-line operations. Theoretically, we derive the mathematical connection of KL control based on dynamic programming with earlier work in control theory which relies on information theoretic dualities for the infinite time horizon case. Algorithmically, we give explicit optimal control policies in nonparametric forms, and propose on-line update schemes with budgeted computational costs. Numerical results demonstrate the effectiveness and usefulness of the proposed frameworks

StocHy: automated verification and synthesis of stochastic processes

StocHy is a software tool for the quantitative analysis of discrete-time stochastic hybrid systems (SHS). StocHy accepts a high-level description of stochastic models and constructs an equivalent SHS model. The tool allows to (i) simulate the SHS evolution over a given time horizon; and to automatically construct formal abstractions of the SHS. Abstractions are then employed for (ii) formal verification or (iii) control (policy, strategy) synthesis. StocHy allows for modular modelling, and has separate simulation, verification and synthesis engines, which are implemented as independent libraries. This allows for libraries to be easily used and for extensions to be easily built. The tool is implemented in C++ and employs manipulations based on vector calculus, the use of sparse matrices, the symbolic construction of probabilistic kernels, and multi-threading. Experiments show StocHy's markedly improved performance when compared to existing abstraction-based approaches: in particular, StocHy beats state-of-the-art tools in terms of precision (abstraction error) and computational effort, and finally attains scalability to large-sized models (12 continuous dimensions). StocHy is available at www.gitlab.com/natchi92/StocHy

ToyArchitecture: Unsupervised Learning of Interpretable Models of the World

Research in Artificial Intelligence (AI) has focused mostly on two extremes: either on small improvements in narrow AI domains, or on universal theoretical frameworks which are usually uncomputable, incompatible with theories of biological intelligence, or lack practical implementations. The goal of this work is to combine the main advantages of the two: to follow a big picture view, while providing a particular theory and its implementation. In contrast with purely theoretical approaches, the resulting architecture should be usable in realistic settings, but also form the core of a framework containing all the basic mechanisms, into which it should be easier to integrate additional required functionality. In this paper, we present a novel, purposely simple, and interpretable hierarchical architecture which combines multiple different mechanisms into one system: unsupervised learning of a model of the world, learning the influence of one's own actions on the world, model-based reinforcement learning, hierarchical planning and plan execution, and symbolic/sub-symbolic integration in general. The learned model is stored in the form of hierarchical representations with the following properties: 1) they are increasingly more abstract, but can retain details when needed, and 2) they are easy to manipulate in their local and symbolic-like form, thus also allowing one to observe the learning process at each level of abstraction. On all levels of the system, the representation of the data can be interpreted in both a symbolic and a sub-symbolic manner. This enables the architecture to learn efficiently using sub-symbolic methods and to employ symbolic inference.Comment: Revision: changed the pdftitl

Abstraction-Based Data-Driven Control

Our world is living a paradigm shift in technology policy, often referred to as the Cyber-Physical Revolution or Industry 4.0. Nowadays, Cyber-Physical Systems are ubiquitous in modern control engineering, including automobiles, aircraft, building control systems, chemical plants, transportation systems, and so on. The interactions of the physical processes with the machines that control them are becoming increasingly complex, and in a growing number of situations either the model of the system is unavailable, or it is too difficult to describe accurately. Therefore, embedded computers need to "learn" the optimal way to control the systems by the mere observation of data. What seems the best approach to control these complex systems is often by discretizing the different variables, thus transforming the model into a combinatorial problem on a finite-state automaton, which is called an abstraction of the real system. Until now, this approach, often referred to as "abstraction-based control" or "symbolic control", has not been proved useful beyond small academic examples. In this project I aim to show the potential of this approach by implementing a novel data-driven approach based on a probabilistic interpretation of the discretization error. I have developed a toolbox (github.com/davidedl-ucl/master-thesis) implementing this kind of control with the aim of integrating it in the Dionysos software github.com/dionysos-dev). With this software, I succeeded in efficiently solving problems for non-linear control systems such as a path planning for an autonomous vehicle and a cart-pole balancing problem. The long-term objective of this project is to improve the methods implemented in my current software by employing a variable discretization of the state space and to consider complex specifications such as LTL formulas.Our world is living a paradigm shift in technology policy, often referred to as the Cyber-Physical Revolution or Industry 4.0. Nowadays, Cyber-Physical Systems are ubiquitous in modern control engineering, including automobiles, aircraft, building control systems, chemical plants, transportation systems, and so on. The interactions of the physical processes with the machines that control them are becoming increasingly complex, and in a growing number of situations either the model of the system is unavailable, or it is too difficult to describe accurately. Therefore, embedded computers need to "learn" the optimal way to control the systems by the mere observation of data. What seems the best approach to control these complex systems is often by discretizing the different variables, thus transforming the model into a combinatorial problem on a finite-state automaton, which is called an abstraction of the real system. Until now, this approach, often referred to as "abstraction-based control" or "symbolic control", has not been proved useful beyond small academic examples. In this project I aim to show the potential of this approach by implementing a novel data-driven approach based on a probabilistic interpretation of the discretization error. I have developed a toolbox (github.com/davidedl-ucl/master-thesis) implementing this kind of control with the aim of integrating it in the Dionysos software github.com/dionysos-dev). With this software, I succeeded in efficiently solving problems for non-linear control systems such as a path planning for an autonomous vehicle and a cart-pole balancing problem. The long-term objective of this project is to improve the methods implemented in my current software by employing a variable discretization of the state space and to consider complex specifications such as LTL formulas