Actor-Critic Reinforcement Learning for Control with Stability Guarantee

Reinforcement Learning (RL) and its integration with deep learning have achieved impressive performance in various robotic control tasks, ranging from motion planning and navigation to end-to-end visual manipulation. However, stability is not guaranteed in model-free RL by solely using data. From a control-theoretic perspective, stability is the most important property for any control system, since it is closely related to safety, robustness, and reliability of robotic systems. In this paper, we propose an actor-critic RL framework for control which can guarantee closed-loop stability by employing the classic Lyapunov's method in control theory. First of all, a data-based stability theorem is proposed for stochastic nonlinear systems modeled by Markov decision process. Then we show that the stability condition could be exploited as the critic in the actor-critic RL to learn a controller/policy. At last, the effectiveness of our approach is evaluated on several well-known 3-dimensional robot control tasks and a synthetic biology gene network tracking task in three different popular physics simulation platforms. As an empirical evaluation on the advantage of stability, we show that the learned policies can enable the systems to recover to the equilibrium or way-points when interfered by uncertainties such as system parametric variations and external disturbances to a certain extent.Comment: IEEE RA-L + IROS 202

Feedback control of unsupported standing in paraplegia. Part I: optimal control approach

This is the first of a pair of papers which describe an investigation into the feasibility of providing artificial balance to paraplegics using electrical stimulation of the paralyzed muscles. By bracing the body above the shanks, only stimulation of the plantarflexors is necessary. This arrangement prevents any influence from the intact neuromuscular system above the spinal cord lesion. Here, the authors extend the design of the controllers to a nested-loop LQG (linear quadratic Gaussian) stimulation controller which has ankle moment feedback (inner loops) and inverted pendulum angle feedback (outer loop). Each control loop is tuned by two parameters, the control weighting and an observer rise-time, which together determine the behavior. The nested structure was chosen because it is robust, despite changes in the muscle properties (fatigue) and interference from spasticity

The disturbance model in model based predictive control

Model Based Predictive Control (MBPC) is a control methodology which uses a process model on-line in the control computer; this model is used for calculating output predictions and optimizing control actions. The importance of the system model has been generally recognized, but less attention has been paid to the role of the disturbance model. In this paper the importance of the disturbance model is indicated with respect to the EPSAC approach to MBPC. To illustrate this importance, an example of this advanced control methodology applied to a typical mechatronic system is presented, to compare the performances obtained by using different disturbance models. It clearly shows the benefits of using an "intelligent" disturbance model instead of the "default" model generally adopted in practice