An Iterative Scheme for the Approximate Linear Programming Solution to the Optimal Control of a Markov Decision Process

This paper addresses the computational issues involved in the solution to an infinite-horizon optimal control problem for a Markov Decision Process (MDP) with a continuous state component and a discrete control input. The optimal Markov policy for the MDP can be determined based on the fixed point solution to the Bellman equation, which can be rephrased as a constrained Linear Program (LP) with an infinite number of constraints and an infinite dimensional optimization variable (the optimal value function). To compute an (approximate) solution to the LP, an iterative randomized scheme is proposed where the optimization variable is expressed as a linear combination of basis functions in a given class: at each iteration, the resulting semi-infinite LP is solved via constraint sampling, whereas the number of basis functions is progressively increased through the iterations so as to meet some performance goal. The effectiveness of the proposed scheme is shown on a multi-room heating system example

Multi-aircraft conflict detection and resolution based on probabilistic reach sets

In this brief, a novel scheme to multi-aircraft conflict detection and resolution is introduced. A key feature of the proposed scheme is that uncertainty affecting the aircraft future positions along some look-ahead prediction horizon is accounted for via a probabilistic reachability analysis approach. In particular, ellipsoidal probabilistic reach sets are determined by formulating a chance-constrained optimization problem and solving it via a simulation-based method called scenario approach. Conflict detection is then performed by verifying if the ellipsoidal reach sets of different aircraft intersect. If a conflict is detected, then the aircraft flight plans are redesigned by solving a second-order cone program resting on the approximation of the ellipsoidal reach sets with spheres with constant radius along the look-ahead horizon. A bisection procedure allows one to determine the minimum radius such that the ellipsoidal reach sets of different aircraft along the corresponding new flight plans do not intersect. Some numerical examples are presented to show the efficacy of the proposed scheme

Model reduction of switched affine systems

This paper addresses model reduction and extends balanced truncation to the class of switched affine systems with endogenous switching. The switched affine system is rewritten as a switched linear one with state resets that account for the affine terms. Balanced truncation can then be applied to each mode dynamics, independently. As a result, different reduced state vectors are associated with the different modes, and reset maps are here appropriately redefined so as to account and compensate for this mismatch, possibly preserving the continuity of the output. The overall behavior of the reduced switched system is determined by both the selected reduction per mode and the adopted reset maps. In this paper, we consider a stochastic setting and propose a randomized method for the selection of the reduced order. The performance of the proposed approach is illustrated through a multi-room temperature control example

Performance assessment and design of abstracted models for stochastic hybrid systems through a randomized approach

In this paper, a simulation-based method for the analysis and design of abstracted models for a stochastic hybrid system is proposed. The accuracy of a model is evaluated in terms of its capability to reproduce the system output for all the realizations of the stochastic input except for a set of (small) probability epsilon (epsilon-abstraction). This naturally leads to chance-constrained optimization problems, which are here tackled by means of a recently developed randomized approach. The main thrust of this paper is that, by testing how close the model and system outputs are over a finite number N of input realizations only, conclusions can be drawn about the model capability as an epsilon-abstraction. The key feature of the proposed method is its high versatility since it does not require specific assumptions on the system to be approximated. The only requirement is that of being able to run multiple simulations of the system behavior for different input realizations.Comment: Preliminary version of a paper accepted for publication in Automatic