324,704 research outputs found
Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictions
We introduce Conformal Decision Theory, a framework for producing safe
autonomous decisions despite imperfect machine learning predictions. Examples
of such decisions are ubiquitous, from robot planning algorithms that rely on
pedestrian predictions, to calibrating autonomous manufacturing to exhibit high
throughput and low error, to the choice of trusting a nominal policy versus
switching to a safe backup policy at run-time. The decisions produced by our
algorithms are safe in the sense that they come with provable statistical
guarantees of having low risk without any assumptions on the world model
whatsoever; the observations need not be I.I.D. and can even be adversarial.
The theory extends results from conformal prediction to calibrate decisions
directly, without requiring the construction of prediction sets. Experiments
demonstrate the utility of our approach in robot motion planning around humans,
automated stock trading, and robot manufacturing.Comment: 8 pages, 5 figure
Safe Reinforcement Learning in Tensor Reproducing Kernel Hilbert Space
This paper delves into the problem of safe reinforcement learning (RL) in a
partially observable environment with the aim of achieving safe-reachability
objectives. In traditional partially observable Markov decision processes
(POMDP), ensuring safety typically involves estimating the belief in latent
states. However, accurately estimating an optimal Bayesian filter in POMDP to
infer latent states from observations in a continuous state space poses a
significant challenge, largely due to the intractable likelihood. To tackle
this issue, we propose a stochastic model-based approach that guarantees RL
safety almost surely in the face of unknown system dynamics and partial
observation environments. We leveraged the Predictive State Representation
(PSR) and Reproducing Kernel Hilbert Space (RKHS) to represent future
multi-step observations analytically, and the results in this context are
provable. Furthermore, we derived essential operators from the kernel Bayes'
rule, enabling the recursive estimation of future observations using various
operators. Under the assumption of \textit{undercompleness}, a polynomial
sample complexity is established for the RL algorithm for the infinite size of
observation and action spaces, ensuring an suboptimal safe policy
guarantee
State-Wise Safe Reinforcement Learning With Pixel Observations
In the context of safe exploration, Reinforcement Learning (RL) has long
grappled with the challenges of balancing the tradeoff between maximizing
rewards and minimizing safety violations, particularly in complex environments
with contact-rich or non-smooth dynamics, and when dealing with
high-dimensional pixel observations. Furthermore, incorporating state-wise
safety constraints in the exploration and learning process, where the agent
must avoid unsafe regions without prior knowledge, adds another layer of
complexity. In this paper, we propose a novel pixel-observation safe RL
algorithm that efficiently encodes state-wise safety constraints with unknown
hazard regions through a newly introduced latent barrier-like function learning
mechanism. As a joint learning framework, our approach begins by constructing a
latent dynamics model with low-dimensional latent spaces derived from pixel
observations. We then build and learn a latent barrier-like function on top of
the latent dynamics and conduct policy optimization simultaneously, thereby
improving both safety and the total expected return. Experimental evaluations
on the safety-gym benchmark suite demonstrate that our proposed method
significantly reduces safety violations throughout the training process, and
demonstrates faster safety convergence compared to existing methods while
achieving competitive results in reward return.Comment: 10 pages, 5 figure
Safety-aware apprenticeship learning
It is well acknowledged in the AI community that finding a good reward function for reinforcement learning is extremely challenging. Apprenticeship learning (AL) is a class of “learning from demonstration” techniques where the reward function of a Markov Decision Process (MDP) is unknown to the learning agent and the agent uses inverse reinforcement learning (IRL) methods to recover expert policy from a set of expert demonstrations. However, as the agent learns exclusively from observations, given a constraint on the probability of the agent running into unwanted situations, there is no verification, nor guarantee, for the learnt policy on the satisfaction of the restriction. In this dissertation, we study the problem of how to guide AL to learn a policy that is inherently safe while still meeting its learning objective. By combining formal methods with imitation learning, a Counterexample-Guided Apprenticeship Learning algorithm is proposed. We consider a setting where the unknown reward function is assumed to be a linear combination of a set of state features, and the safety property is specified in Probabilistic Computation Tree Logic (PCTL). By embedding probabilistic model checking inside AL, we propose a novel counterexample-guided approach that can ensure both safety and performance of the learnt policy. This algorithm guarantees that given some formal safety specification defined by probabilistic temporal logic, the learnt policy shall satisfy this specification. We demonstrate the effectiveness of our approach on several challenging AL scenarios where safety is essential
GLAS: Global-to-Local Safe Autonomy Synthesis for Multi-Robot Motion Planning with End-to-End Learning
We present GLAS: Global-to- Local Autonomy Synthesis, a provably-safe, automated distributed policy generation for multi-robot motion planning. Our approach combines the advantage of centralized planning of avoiding local minima with the advantage of decentralized controllers of scalability and distributed computation. In particular, our synthesized policies only require relative state information of nearby neighbors and obstacles, and compute a provably-safe action. Our approach has three major components: i) we generate demonstration trajectories using a global planner and extract local observations from them, ii) we use deep imitation learning to learn a decentralized policy that can run efficiently online, and iii) we introduce a novel differentiable safety module to ensure collision-free operation, thereby allowing for end-to-end policy training. Our numerical experiments demonstrate that our policies have a 20% higher success rate than optimal reciprocal collision avoidance, ORCA, across a wide range of robot and obstacle densities. We demonstrate our method on an aerial swarm, executing the policy on low-end microcontrollers in real-time
Discovering Blind Spots in Reinforcement Learning
Agents trained in simulation may make errors in the real world due to
mismatches between training and execution environments. These mistakes can be
dangerous and difficult to discover because the agent cannot predict them a
priori. We propose using oracle feedback to learn a predictive model of these
blind spots to reduce costly errors in real-world applications. We focus on
blind spots in reinforcement learning (RL) that occur due to incomplete state
representation: The agent does not have the appropriate features to represent
the true state of the world and thus cannot distinguish among numerous states.
We formalize the problem of discovering blind spots in RL as a noisy supervised
learning problem with class imbalance. We learn models to predict blind spots
in unseen regions of the state space by combining techniques for label
aggregation, calibration, and supervised learning. The models take into
consideration noise emerging from different forms of oracle feedback, including
demonstrations and corrections. We evaluate our approach on two domains and
show that it achieves higher predictive performance than baseline methods, and
that the learned model can be used to selectively query an oracle at execution
time to prevent errors. We also empirically analyze the biases of various
feedback types and how they influence the discovery of blind spots.Comment: To appear at AAMAS 201
Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
AI systems are increasingly applied to complex tasks that involve interaction
with humans. During training, such systems are potentially dangerous, as they
haven't yet learned to avoid actions that could cause serious harm. How can an
AI system explore and learn without making a single mistake that harms humans
or otherwise causes serious damage? For model-free reinforcement learning,
having a human "in the loop" and ready to intervene is currently the only way
to prevent all catastrophes. We formalize human intervention for RL and show
how to reduce the human labor required by training a supervised learner to
imitate the human's intervention decisions. We evaluate this scheme on Atari
games, with a Deep RL agent being overseen by a human for four hours. When the
class of catastrophes is simple, we are able to prevent all catastrophes
without affecting the agent's learning (whereas an RL baseline fails due to
catastrophic forgetting). However, this scheme is less successful when
catastrophes are more complex: it reduces but does not eliminate catastrophes
and the supervised learner fails on adversarial examples found by the agent.
Extrapolating to more challenging environments, we show that our implementation
would not scale (due to the infeasible amount of human labor required). We
outline extensions of the scheme that are necessary if we are to train
model-free agents without a single catastrophe
- …