Model-Based Reinforcement Learning for Stochastic Hybrid Systems

Optimal control of general nonlinear systems is a central challenge in automation. Enabled by powerful function approximators, data-driven approaches to control have recently successfully tackled challenging robotic applications. However, such methods often obscure the structure of dynamics and control behind black-box over-parameterized representations, thus limiting our ability to understand closed-loop behavior. This paper adopts a hybrid-system view of nonlinear modeling and control that lends an explicit hierarchical structure to the problem and breaks down complex dynamics into simpler localized units. We consider a sequence modeling paradigm that captures the temporal structure of the data and derive an expectation-maximization (EM) algorithm that automatically decomposes nonlinear dynamics into stochastic piecewise affine dynamical systems with nonlinear boundaries. Furthermore, we show that these time-series models naturally admit a closed-loop extension that we use to extract local polynomial feedback controllers from nonlinear experts via behavioral cloning. Finally, we introduce a novel hybrid relative entropy policy search (Hb-REPS) technique that incorporates the hierarchical nature of hybrid systems and optimizes a set of time-invariant local feedback controllers derived from a local polynomial approximation of a global state-value function

Model predictive control techniques for hybrid systems

This paper describes the main issues encountered when applying model predictive control to hybrid processes. Hybrid model predictive control (HMPC) is a research field non-fully developed with many open challenges. The paper describes some of the techniques proposed by the research community to overcome the main problems encountered. Issues related to the stability and the solution of the optimization problem are also discussed. The paper ends by describing the results of a benchmark exercise in which several HMPC schemes were applied to a solar air conditioning plant.Ministerio de Eduación y Ciencia DPI2007-66718-C04-01Ministerio de Eduación y Ciencia DPI2008-0581

Reachability problems for PAMs

Piecewise affine maps (PAMs) are frequently used as a reference model to show the openness of the reachability questions in other systems. The reachability problem for one-dimentional PAM is still open even if we define it with only two intervals. As the main contribution of this paper we introduce new techniques for solving reachability problems based on p-adic norms and weights as well as showing decidability for two classes of maps. Then we show the connections between topological properties for PAM's orbits, reachability problems and representation of numbers in a rational base system. Finally we show a particular instance where the uniform distribution of the original orbit may not remain uniform or even dense after making regular shifts and taking a fractional part in that sequence.Comment: 16 page

On the Construction of Safe Controllable Regions for Affine Systems with Applications to Robotics

This paper studies the problem of constructing in-block controllable (IBC) regions for affine systems. That is, we are concerned with constructing regions in the state space of affine systems such that all the states in the interior of the region are mutually accessible through the region's interior by applying uniformly bounded inputs. We first show that existing results for checking in-block controllability on given polytopic regions cannot be easily extended to address the question of constructing IBC regions. We then explore the geometry of the problem to provide a computationally efficient algorithm for constructing IBC regions. We also prove the soundness of the algorithm. We then use the proposed algorithm to construct safe speed profiles for different robotic systems, including fully-actuated robots, ground robots modeled as unicycles with acceleration limits, and unmanned aerial vehicles (UAVs). Finally, we present several experimental results on UAVs to verify the effectiveness of the proposed algorithm. For instance, we use the proposed algorithm for real-time collision avoidance for UAVs.Comment: 17 pages, 18 figures, under review for publication in Automatic

A Framework for Worst-Case and Stochastic Safety Verification Using Barrier Certificates

This paper presents a methodology for safety verification of continuous and hybrid systems in the worst-case and stochastic settings. In the worst-case setting, a function of state termed barrier certificate is used to certify that all trajectories of the system starting from a given initial set do not enter an unsafe region. No explicit computation of reachable sets is required in the construction of barrier certificates, which makes it possible to handle nonlinearity, uncertainty, and constraints directly within this framework. In the stochastic setting, our method computes an upper bound on the probability that a trajectory of the system reaches the unsafe set, a bound whose validity is proven by the existence of a barrier certificate. For polynomial systems, barrier certificates can be constructed using convex optimization, and hence the method is computationally tractable. Some examples are provided to illustrate the use of the method

Digital implementation of hierarchical piecewise-affine controllers

This paper proposes the design of hierarchical piecewise-affine (PWA) controllers to alleviate the processing time or prohibitive memory requirements of large controller structures. The constituent PWA modules of the hierarchical solution have fewer inputs and/or coarser partitions, so that they can reduce considerably the hardware resources required and/or the time response of the controller. A design methodology aided by CAD tools is employed to design the parameters of the controller, implement its architecture in an FPGA, and verify the static and dynamic behavior of the digital implementation by applying hardware-in-the-loop testing.Comunidad Europea FP7-IST-248858Ministerio de Ciencia e Innovación TEC2008-04920 y DPI2008-03847Junta de Andalucía P08-TIC-0367