2,514 research outputs found
Temporal Logic Control of POMDPs via Label-based Stochastic Simulation Relations
The synthesis of controllers guaranteeing linear temporal logic specifications on partially observable Markov decision processes (POMDP) via their belief models causes computational issues due to the continuous spaces. In this work, we construct a finite-state abstraction on which a control policy is synthesized and refined back to the original belief model. We introduce a new notion of label-based approximate stochastic simulation to quantify the deviation between belief models. We develop a robust synthesis methodology that yields a lower bound on the satisfaction probability, by compensating for deviations a priori, and that utilizes a less conservative control refinement
Temporal Logic Control of POMDPs via Label-based Stochastic Simulation Relations
The synthesis of controllers guaranteeing linear temporal logic specifications on partially observable Markov decision processes (POMDP) via their belief models causes computational issues due to the continuous spaces. In this work, we construct a finite-state abstraction on which a control policy is synthesized and refined back to the original belief model. We introduce a new notion of label-based approximate stochastic simulation to quantify the deviation between belief models. We develop a robust synthesis methodology that yields a lower bound on the satisfaction probability, by compensating for deviations a priori, and that utilizes a less conservative control refinement
Formal Controller Synthesis for Continuous-Space MDPs via Model-Free Reinforcement Learning
A novel reinforcement learning scheme to synthesize policies for
continuous-space Markov decision processes (MDPs) is proposed. This scheme
enables one to apply model-free, off-the-shelf reinforcement learning
algorithms for finite MDPs to compute optimal strategies for the corresponding
continuous-space MDPs without explicitly constructing the finite-state
abstraction. The proposed approach is based on abstracting the system with a
finite MDP (without constructing it explicitly) with unknown transition
probabilities, synthesizing strategies over the abstract MDP, and then mapping
the results back over the concrete continuous-space MDP with approximate
optimality guarantees. The properties of interest for the system belong to a
fragment of linear temporal logic, known as syntactically co-safe linear
temporal logic (scLTL), and the synthesis requirement is to maximize the
probability of satisfaction within a given bounded time horizon. A key
contribution of the paper is to leverage the classical convergence results for
reinforcement learning on finite MDPs and provide control strategies maximizing
the probability of satisfaction over unknown, continuous-space MDPs while
providing probabilistic closeness guarantees. Automata-based reward functions
are often sparse; we present a novel potential-based reward shaping technique
to produce dense rewards to speed up learning. The effectiveness of the
proposed approach is demonstrated by applying it to three physical benchmarks
concerning the regulation of a room's temperature, control of a road traffic
cell, and of a 7-dimensional nonlinear model of a BMW 320i car.Comment: This work is accepted at the 11th ACM/IEEE Conference on
Cyber-Physical Systems (ICCPS
Recommended from our members
Compositional Abstraction-based Synthesis of General MDPs via Approximate Probabilistic Relations
We propose a compositional approach for constructing abstractions of general Markov decision processes (gMDPs) using approximate probabilistic relations. The abstraction framework is based on the notion of δ-lifted relations, using which one can quantify the distance in probability between the interconnected gMDPs and that of their abstractions. This new approximate relation unifies compositionality results in the literature by incorporating the dependencies between state transitions explicitly and by allowing abstract models to have either infinite or finite state spaces. Accordingly, one can leverage the proposed results to perform analysis and synthesis over abstract models, and then carry the results over concrete ones. To this end, we first propose our compositionality results using the new approximate probabilistic relation which is based on lifting. We then focus on a class of stochastic nonlinear dynamical systems and construct their abstractions using both model order reduction and space discretization in a unified framework. We provide conditions for simultaneous existence of relations incorporating the structure of the network. Finally, we demonstrate the effectiveness of the proposed results by considering a network of four nonlinear dynamical subsystems (together 12 dimensions) and constructing finite abstractions from their reduced-order versions (together 4 dimensions) in a unified compositional framework. We benchmark our results against the compositional abstraction techniques that construct both infinite abstractions (reduced-order models) and finite MDPs in two consecutive steps. We show that our approach is less conservative than the ones available in the literature.
</p
Similarity quantification for linear stochastic systems as a set-theoretic control problem
For the formal verification and design of control systems, abstractions with
quantified accuracy are crucial. Such similarity quantification is hindered by
the challenging computation of approximate stochastic simulation relations.
This is especially the case when considering accurate deviation bounds between
a stochastic continuous-state model and its finite-state abstraction. In this
work, we give a comprehensive computational approach and analysis for linear
stochastic systems. More precisely, we develop a computational method that
characterizes the set of possible simulation relations and optimally trades off
the error contributions on the system's output with deviations in the
transition probability. To this end, we establish an optimal coupling between
the models and simultaneously solve the approximate simulation relation problem
as a set-theoretic control problem using the concept of invariant sets. We show
the variation of the guaranteed satisfaction probability as a function of the
error trade-off in a case study where a formal specification is given as a
temporal logic formula.Comment: 16 pages, 9 figures, submitted to Automatic
- …