1,145 research outputs found
Formal Controller Synthesis for Continuous-Space MDPs via Model-Free Reinforcement Learning
A novel reinforcement learning scheme to synthesize policies for
continuous-space Markov decision processes (MDPs) is proposed. This scheme
enables one to apply model-free, off-the-shelf reinforcement learning
algorithms for finite MDPs to compute optimal strategies for the corresponding
continuous-space MDPs without explicitly constructing the finite-state
abstraction. The proposed approach is based on abstracting the system with a
finite MDP (without constructing it explicitly) with unknown transition
probabilities, synthesizing strategies over the abstract MDP, and then mapping
the results back over the concrete continuous-space MDP with approximate
optimality guarantees. The properties of interest for the system belong to a
fragment of linear temporal logic, known as syntactically co-safe linear
temporal logic (scLTL), and the synthesis requirement is to maximize the
probability of satisfaction within a given bounded time horizon. A key
contribution of the paper is to leverage the classical convergence results for
reinforcement learning on finite MDPs and provide control strategies maximizing
the probability of satisfaction over unknown, continuous-space MDPs while
providing probabilistic closeness guarantees. Automata-based reward functions
are often sparse; we present a novel potential-based reward shaping technique
to produce dense rewards to speed up learning. The effectiveness of the
proposed approach is demonstrated by applying it to three physical benchmarks
concerning the regulation of a room's temperature, control of a road traffic
cell, and of a 7-dimensional nonlinear model of a BMW 320i car.Comment: This work is accepted at the 11th ACM/IEEE Conference on
Cyber-Physical Systems (ICCPS
Recommended from our members
New Program Abstractions for Privacy
Static program analysis, once seen primarily as a tool for optimising programs, is now increasingly important as a means to provide quality guarantees about programs. One measure of quality is the extent to which programs respect the privacy of user data. Differential privacy is a rigorous quantified definition of privacy which guarantees a bound on the loss of privacy due to the release of statistical queries. Among the benefits enjoyed by the definition of differential privacy are compositionality properties that allow differentially private analyses to be built from pieces and combined in various ways. This has led to the development of frameworks for the construction of differentially private program analyses which are private-by-construction. Past frameworks assume that the sensitive data is collected centrally, and processed by a trusted curator. However, the main examples of differential privacy applied in practice - for example in the use of differential privacy in Google Chrome’s collection of browsing statistics, or Apple’s training of predictive messaging in iOS 10 -use a purely local mechanism applied at the data source, thus avoiding the collection of sensitive data altogether. While this is a benefit of the local approach, with systems like Apple’s, users are required to completely trust that the analysis running on their system has the claimed privacy properties.
In this position paper we outline some key challenges in developing static analyses for analysing differential privacy, and propose novel abstractions for describing the behaviour of probabilistic programs not previously used in static analyses
- …