Chen, Anthony Siming

Herrmann, Guido

English

This paper proposes two novel adaptive optimal control algorithms for continuous-time nonlinear afﬁne systems based on reinforcement learning: i) generalised policy iteration (GPI) and ii) Q-learning. As a result, the a priori knowledge of the system drift f(x) is not needed via GPI, which gives us a partially model-free and online solution. We then for the ﬁrst time extend the idea of Q-learning to the nonlinear continuoustime optimal control problem in a noniterative manner. This leads to a completely model-free method where neither the system drift f(x) nor the input gain g(x) is needed. For both methods, the adaptive critic and actor are continuously and simultaneously updating each other without iterative steps, which effectively avoids the hybrid structure and the need for an initial stabilising control policy. Moreover, ﬁnite-time convergence is guaranteed by using a sliding mode technique in the new adaptive approach, where the persistent excitation (PE) condition can be directly veriﬁed online. We also prove the overall Lyapunov stability and demonstrate the effectiveness of the proposed algorithms using numerical examples

The University of Manchester - Institutional Repository

Adaptive Optimal Control via Continuous-Time Q-Learning for Unknown Nonlinear Afﬁne Systems

This paper proposes two novel adaptive optimal control algorithms for continuous-time nonlinear affine systems based on reinforcement learning: i) generalized policy iteration (GPI) and ii) Q-learning. As a result, the a priori knowledge of the system drift f (x) is not needed via GPI, which gives us a partially model-free and online solution. We then for the first time extend the idea of Q-learning to the nonlinear continuous time optimal control problem in a noniterative manner. Thisleads to a completely model-free method where neither the system drift f (x) nor the input gain g(x) is needed. For both methods, the adaptive critic and actor are continuously and simultaneously updating each other without iterative steps, which effectively avoids the hybrid structure and the need or an initial stabilizing control policy. Moreover, finite-time convergence is guaranteed by using a sliding mode technique in the new adaptive approach, where the persistent excitation (PE) condition can be directly verified online. We also prove the overall Lyapunov stability and demonstrate the effectiveness of the proposed algorithms using numerical examples

Explore Bristol Research

Adaptive Optimal Control via Continuous-Time Q-Learning for Unknown Nonlinear Affine Systems

https://pure.manchester.ac.uk/ws/files/147419094/CDC19_1168_FI.pdf

Adaptive Optimal Control via Continuous-Time Q-Learning for Unknown Nonlinear Affine Systems

Abstract

Similar works

Full text

Available Versions

The University of Manchester - Institutional Repository

Explore Bristol Research