16,493 research outputs found
Safety-Critical Learning of Robot Control with Temporal Logic Specifications
Reinforcement learning (RL) is a promising approach. However, success is
limited to real-world applications, because ensuring safe exploration and
facilitating adequate exploitation is a challenge for controlling robotic
systems with unknown models and measurement uncertainties. The learning problem
becomes even more difficult for complex tasks over continuous state-action. In
this paper, we propose a learning-based robotic control framework consisting of
several aspects: (1) we leverage Linear Temporal Logic (LTL) to express complex
tasks over infinite horizons that are translated to a novel automaton
structure; (2) we detail an innovative reward scheme for LTL satisfaction with
a probabilistic guarantee. Then, by applying a reward shaping technique, we
develop a modular policy-gradient architecture exploiting the benefits of the
automaton structure to decompose overall tasks and enhance the performance of
learned controllers; (3) by incorporating Gaussian Processes (GPs) to estimate
the uncertain dynamic systems, we synthesize a model-based safe exploration
during the learning process using Exponential Control Barrier Functions (ECBFs)
that generalize systems with high-order relative degrees; (4) to further
improve the efficiency of exploration, we utilize the properties of LTL
automata and ECBFs to propose a safe guiding process. Finally, we demonstrate
the effectiveness of the framework via several robotic environments. We show an
ECBF-based modular deep RL algorithm that achieves near-perfect success rates
and safety guarding with high probability confidence during training.Comment: Under Review. arXiv admin note: text overlap with arXiv:2102.1285
ISAACS: Iterative Soft Adversarial Actor-Critic for Safety
The deployment of robots in uncontrolled environments requires them to
operate robustly under previously unseen scenarios, like irregular terrain and
wind conditions. Unfortunately, while rigorous safety frameworks from robust
optimal control theory scale poorly to high-dimensional nonlinear dynamics,
control policies computed by more tractable "deep" methods lack guarantees and
tend to exhibit little robustness to uncertain operating conditions. This work
introduces a novel approach enabling scalable synthesis of robust
safety-preserving controllers for robotic systems with general nonlinear
dynamics subject to bounded modeling error by combining game-theoretic safety
analysis with adversarial reinforcement learning in simulation. Following a
soft actor-critic scheme, a safety-seeking fallback policy is co-trained with
an adversarial "disturbance" agent that aims to invoke the worst-case
realization of model error and training-to-deployment discrepancy allowed by
the designer's uncertainty. While the learned control policy does not
intrinsically guarantee safety, it is used to construct a real-time safety
filter (or shield) with robust safety guarantees based on forward reachability
rollouts. This shield can be used in conjunction with a safety-agnostic control
policy, precluding any task-driven actions that could result in loss of safety.
We evaluate our learning-based safety approach in a 5D race car simulator,
compare the learned safety policy to the numerically obtained optimal solution,
and empirically validate the robust safety guarantee of our proposed safety
shield against worst-case model discrepancy.Comment: Accepted in 5th Annual Learning for Dynamics & Control Conference
(L4DC), University of Pennsylvani
Output Reachable Set Estimation and Verification for Multi-Layer Neural Networks
In this paper, the output reachable estimation and safety verification
problems for multi-layer perceptron neural networks are addressed. First, a
conception called maximum sensitivity in introduced and, for a class of
multi-layer perceptrons whose activation functions are monotonic functions, the
maximum sensitivity can be computed via solving convex optimization problems.
Then, using a simulation-based method, the output reachable set estimation
problem for neural networks is formulated into a chain of optimization
problems. Finally, an automated safety verification is developed based on the
output reachable set estimation result. An application to the safety
verification for a robotic arm model with two joints is presented to show the
effectiveness of proposed approaches.Comment: 8 pages, 9 figures, to appear in TNNL
Episodic Learning with Control Lyapunov Functions for Uncertain Robotic Systems
Many modern nonlinear control methods aim to endow systems with guaranteed
properties, such as stability or safety, and have been successfully applied to
the domain of robotics. However, model uncertainty remains a persistent
challenge, weakening theoretical guarantees and causing implementation failures
on physical systems. This paper develops a machine learning framework centered
around Control Lyapunov Functions (CLFs) to adapt to parametric uncertainty and
unmodeled dynamics in general robotic systems. Our proposed method proceeds by
iteratively updating estimates of Lyapunov function derivatives and improving
controllers, ultimately yielding a stabilizing quadratic program model-based
controller. We validate our approach on a planar Segway simulation,
demonstrating substantial performance improvements by iteratively refining on a
base model-free controller
Assuring Safety under Uncertainty in Learning-Based Control Systems
Learning-based controllers have recently shown impressive results for different robotic tasks in well-defined environments, successfully solving a Rubiks cube and sorting objects in a bin. These advancements promise to enable a host of new capabilities for complex robotic systems. However, these learning-based controllers cannot yet be deployed in highly uncertain environments due to significant issues relating to learning reliability, robustness, and safety.
To overcome these issues, this thesis proposes new methods for integrating model information (e.g. model-based control priors) into the reinforcement learning framework, which is crucial to ensuring reliability and safety. I show, both empirically and theoretically, that this model information greatly reduces variance in learning and can effectively constrain the policy search space, thus enabling significant improvements in sample complexity for the underlying RL algorithms. Furthermore, by leveraging control barrier functions and Gaussian process uncertainty models, I show how system safety can be maintained under uncertainty without interfering with the learning process (e.g. distorting the policy gradients).
The last part of the thesis will discuss fundamental limitations that arise when utilizing machine learning to derive safety guarantees. In particular, I show that widely used uncertainty models can be highly inaccurate when predicting rare events, and examine the implications of this for safe learning. To overcome some of these limitations, a novel framework is developed based on assume-guarantee contracts in order to ensure safety in multi-agent human environments. The proposed approach utilizes contracts to impose loose responsibilities on agents in the environment, which are learned from data. Imposing these responsibilities on agents, rather than treating their uncertainty as a purely random process, allows us to achieve both safety and efficiency in interactions.</p
Safe Multi-Agent Interaction through Robust Control Barrier Functions with Learned Uncertainties
Robots operating in real world settings must navigate and maintain safety while interacting with many heterogeneous agents and obstacles. Multi-Agent Control Barrier Functions (CBF) have emerged as a computationally efficient tool to guarantee safety in multi-agent environments, but they assume perfect knowledge of both the robot dynamics and other agents' dynamics. While knowledge of the robot's dynamics might be reasonably well known, the heterogeneity of agents in real-world environments means there will always be considerable uncertainty in our prediction of other agents' dynamics. This work aims to learn high-confidence bounds for these dynamic uncertainties using Matrix-Variate Gaussian Process models, and incorporates them into a robust multi-agent CBF framework. We transform the resulting min-max robust CBF into a quadratic program, which can be efficiently solved in real time. We verify via simulation results that the nominal multi-agent CBF is often violated during agent interactions, whereas our robust formulation maintains safety with a much higher probability and adapts to learned uncertainties
Actor-Critic Reinforcement Learning for Control with Stability Guarantee
Reinforcement Learning (RL) and its integration with deep learning have
achieved impressive performance in various robotic control tasks, ranging from
motion planning and navigation to end-to-end visual manipulation. However,
stability is not guaranteed in model-free RL by solely using data. From a
control-theoretic perspective, stability is the most important property for any
control system, since it is closely related to safety, robustness, and
reliability of robotic systems. In this paper, we propose an actor-critic RL
framework for control which can guarantee closed-loop stability by employing
the classic Lyapunov's method in control theory. First of all, a data-based
stability theorem is proposed for stochastic nonlinear systems modeled by
Markov decision process. Then we show that the stability condition could be
exploited as the critic in the actor-critic RL to learn a controller/policy. At
last, the effectiveness of our approach is evaluated on several well-known
3-dimensional robot control tasks and a synthetic biology gene network tracking
task in three different popular physics simulation platforms. As an empirical
evaluation on the advantage of stability, we show that the learned policies can
enable the systems to recover to the equilibrium or way-points when interfered
by uncertainties such as system parametric variations and external disturbances
to a certain extent.Comment: IEEE RA-L + IROS 202
- …