2,437 research outputs found
Temporal Logic Control of POMDPs via Label-based Stochastic Simulation Relations
The synthesis of controllers guaranteeing linear temporal logic specifications on partially observable Markov decision processes (POMDP) via their belief models causes computational issues due to the continuous spaces. In this work, we construct a finite-state abstraction on which a control policy is synthesized and refined back to the original belief model. We introduce a new notion of label-based approximate stochastic simulation to quantify the deviation between belief models. We develop a robust synthesis methodology that yields a lower bound on the satisfaction probability, by compensating for deviations a priori, and that utilizes a less conservative control refinement
Temporal Logic Control of POMDPs via Label-based Stochastic Simulation Relations
The synthesis of controllers guaranteeing linear temporal logic specifications on partially observable Markov decision processes (POMDP) via their belief models causes computational issues due to the continuous spaces. In this work, we construct a finite-state abstraction on which a control policy is synthesized and refined back to the original belief model. We introduce a new notion of label-based approximate stochastic simulation to quantify the deviation between belief models. We develop a robust synthesis methodology that yields a lower bound on the satisfaction probability, by compensating for deviations a priori, and that utilizes a less conservative control refinement
Formal Controller Synthesis for Continuous-Space MDPs via Model-Free Reinforcement Learning
A novel reinforcement learning scheme to synthesize policies for
continuous-space Markov decision processes (MDPs) is proposed. This scheme
enables one to apply model-free, off-the-shelf reinforcement learning
algorithms for finite MDPs to compute optimal strategies for the corresponding
continuous-space MDPs without explicitly constructing the finite-state
abstraction. The proposed approach is based on abstracting the system with a
finite MDP (without constructing it explicitly) with unknown transition
probabilities, synthesizing strategies over the abstract MDP, and then mapping
the results back over the concrete continuous-space MDP with approximate
optimality guarantees. The properties of interest for the system belong to a
fragment of linear temporal logic, known as syntactically co-safe linear
temporal logic (scLTL), and the synthesis requirement is to maximize the
probability of satisfaction within a given bounded time horizon. A key
contribution of the paper is to leverage the classical convergence results for
reinforcement learning on finite MDPs and provide control strategies maximizing
the probability of satisfaction over unknown, continuous-space MDPs while
providing probabilistic closeness guarantees. Automata-based reward functions
are often sparse; we present a novel potential-based reward shaping technique
to produce dense rewards to speed up learning. The effectiveness of the
proposed approach is demonstrated by applying it to three physical benchmarks
concerning the regulation of a room's temperature, control of a road traffic
cell, and of a 7-dimensional nonlinear model of a BMW 320i car.Comment: This work is accepted at the 11th ACM/IEEE Conference on
Cyber-Physical Systems (ICCPS
Similarity quantification for linear stochastic systems as a set-theoretic control problem
For the formal verification and design of control systems, abstractions with
quantified accuracy are crucial. Such similarity quantification is hindered by
the challenging computation of approximate stochastic simulation relations.
This is especially the case when considering accurate deviation bounds between
a stochastic continuous-state model and its finite-state abstraction. In this
work, we give a comprehensive computational approach and analysis for linear
stochastic systems. More precisely, we develop a computational method that
characterizes the set of possible simulation relations and optimally trades off
the error contributions on the system's output with deviations in the
transition probability. To this end, we establish an optimal coupling between
the models and simultaneously solve the approximate simulation relation problem
as a set-theoretic control problem using the concept of invariant sets. We show
the variation of the guaranteed satisfaction probability as a function of the
error trade-off in a case study where a formal specification is given as a
temporal logic formula.Comment: 16 pages, 9 figures, submitted to Automatic
Certified Reinforcement Learning with Logic Guidance
This paper proposes the first model-free Reinforcement Learning (RL)
framework to synthesise policies for unknown, and continuous-state Markov
Decision Processes (MDPs), such that a given linear temporal property is
satisfied. We convert the given property into a Limit Deterministic Buchi
Automaton (LDBA), namely a finite-state machine expressing the property.
Exploiting the structure of the LDBA, we shape a synchronous reward function
on-the-fly, so that an RL algorithm can synthesise a policy resulting in traces
that probabilistically satisfy the linear temporal property. This probability
(certificate) is also calculated in parallel with policy learning when the
state space of the MDP is finite: as such, the RL algorithm produces a policy
that is certified with respect to the property. Under the assumption of finite
state space, theoretical guarantees are provided on the convergence of the RL
algorithm to an optimal policy, maximising the above probability. We also show
that our method produces ''best available'' control policies when the logical
property cannot be satisfied. In the general case of a continuous state space,
we propose a neural network architecture for RL and we empirically show that
the algorithm finds satisfying policies, if there exist such policies. The
performance of the proposed framework is evaluated via a set of numerical
examples and benchmarks, where we observe an improvement of one order of
magnitude in the number of iterations required for the policy synthesis,
compared to existing approaches whenever available.Comment: This article draws from arXiv:1801.08099, arXiv:1809.0782
- …