4 research outputs found
Multi Pseudo Q-learning Based Deterministic Policy Gradient for Tracking Control of Autonomous Underwater Vehicles
This paper investigates trajectory tracking problem for a class of
underactuated autonomous underwater vehicles (AUVs) with unknown dynamics and
constrained inputs. Different from existing policy gradient methods which
employ single actor-critic but cannot realize satisfactory tracking control
accuracy and stable learning, our proposed algorithm can achieve high-level
tracking control accuracy of AUVs and stable learning by applying a hybrid
actors-critics architecture, where multiple actors and critics are trained to
learn a deterministic policy and action-value function, respectively.
Specifically, for the critics, the expected absolute Bellman error based
updating rule is used to choose the worst critic to be updated in each time
step. Subsequently, to calculate the loss function with more accurate target
value for the chosen critic, Pseudo Q-learning, which uses sub-greedy policy to
replace the greedy policy in Q-learning, is developed for continuous action
spaces, and Multi Pseudo Q-learning (MPQ) is proposed to reduce the
overestimation of action-value function and to stabilize the learning. As for
the actors, deterministic policy gradient is applied to update the weights, and
the final learned policy is defined as the average of all actors to avoid large
but bad updates. Moreover, the stability analysis of the learning is given
qualitatively. The effectiveness and generality of the proposed MPQ-based
Deterministic Policy Gradient (MPQ-DPG) algorithm are verified by the
application on AUV with two different reference trajectories. And the results
demonstrate high-level tracking control accuracy and stable learning of
MPQ-DPG. Besides, the results also validate that increasing the number of the
actors and critics will further improve the performance.Comment: IEEE Transactions on Neural Networks and Learning System
Model-Reference Reinforcement Learning Control of Autonomous Surface Vehicles with Uncertainties
This paper presents a novel model-reference reinforcement learning control
method for uncertain autonomous surface vehicles. The proposed control combines
a conventional control method with deep reinforcement learning. With the
conventional control, we can ensure the learning-based control law provides
closed-loop stability for the overall system, and potentially increase the
sample efficiency of the deep reinforcement learning. With the reinforcement
learning, we can directly learn a control law to compensate for modeling
uncertainties. In the proposed control, a nominal system is employed for the
design of a baseline control law using a conventional control approach. The
nominal system also defines the desired performance for uncertain autonomous
vehicles to follow. In comparison with traditional deep reinforcement learning
methods, our proposed learning-based control can provide stability guarantees
and better sample efficiency. We demonstrate the performance of the new
algorithm via extensive simulation results.Comment: 9 pages, 10 figures
A selected review on reinforcement learning based control for autonomous underwater vehicles
Recently, reinforcement learning (RL) has been extensively studied and
achieved promising results in a wide range of control tasks. Meanwhile,
autonomous underwater vehicle (AUV) is an important tool for executing complex
and challenging underwater tasks. The advances in RL offers ample opportunities
for developing intelligent AUVs. This paper provides a selected review on RL
based control for AUVs with the focus on applications of RL to low-level
control tasks for underwater regulation and tracking. To this end, we first
present a concise introduction to the RL based control framework. Then, we
provide an overview of RL methods for AUVs control problems, where the main
challenges and recent progresses are discussed. Finally, two representative
cases of RL-based controllers are given in detail for the model-free RL methods
on AUVs
Model-Reference Reinforcement Learning for Collision-Free Tracking Control of Autonomous Surface Vehicles
This paper presents a novel model-reference reinforcement learning algorithm
for the intelligent tracking control of uncertain autonomous surface vehicles
with collision avoidance. The proposed control algorithm combines a
conventional control method with reinforcement learning to enhance control
accuracy and intelligence. In the proposed control design, a nominal system is
considered for the design of a baseline tracking controller using a
conventional control approach. The nominal system also defines the desired
behaviour of uncertain autonomous surface vehicles in an obstacle-free
environment. Thanks to reinforcement learning, the overall tracking controller
is capable of compensating for model uncertainties and achieving collision
avoidance at the same time in environments with obstacles. In comparison to
traditional deep reinforcement learning methods, our proposed learning-based
control can provide stability guarantees and better sample efficiency. We
demonstrate the performance of the new algorithm using an example of autonomous
surface vehicles.Comment: Extension of arXiv:2003.1383