Reinforcement Learning Based on Real-Time Iteration NMPC

Reinforcement Learning (RL) has proven a stunning ability to learn optimal policies from data without any prior knowledge on the process. The main drawback of RL is that it is typically very difficult to guarantee stability and safety. On the other hand, Nonlinear Model Predictive Control (NMPC) is an advanced model-based control technique which does guarantee safety and stability, but only yields optimality for the nominal model. Therefore, it has been recently proposed to use NMPC as a function approximator within RL. While the ability of this approach to yield good performance has been demonstrated, the main drawback hindering its applicability is related to the computational burden of NMPC, which has to be solved to full convergence. In practice, however, computationally efficient algorithms such as the Real-Time Iteration (RTI) scheme are deployed in order to return an approximate NMPC solution in very short time. In this paper we bridge this gap by extending the existing theoretical framework to also cover RL based on RTI NMPC. We demonstrate the effectiveness of this new RL approach with a nontrivial example modeling a challenging nonlinear system subject to stochastic perturbations with the objective of optimizing an economic cost.Comment: accepted for the IFAC World Congress 202

Data-driven Economic NMPC using Reinforcement Learning

Reinforcement Learning (RL) is a powerful tool to perform data-driven optimal control without relying on a model of the system. However, RL struggles to provide hard guarantees on the behavior of the resulting control scheme. In contrast, Nonlinear Model Predictive Control (NMPC) and Economic NMPC (ENMPC) are standard tools for the closed-loop optimal control of complex systems with constraints and limitations, and benefit from a rich theory to assess their closed-loop behavior. Unfortunately, the performance of (E)NMPC hinges on the quality of the model underlying the control scheme. In this paper, we show that an (E)NMPC scheme can be tuned to deliver the optimal policy of the real system even when using a wrong model. This result also holds for real systems having stochastic dynamics. This entails that ENMPC can be used as a new type of function approximator within RL. Furthermore, we investigate our results in the context of ENMPC and formally connect them to the concept of dissipativity, which is central for the ENMPC stability. Finally, we detail how these results can be used to deploy classic RL tools for tuning (E)NMPC schemes. We apply these tools on both a classical linear MPC setting and a standard nonlinear example from the ENMPC literature

Modeling and supervisory control design for a combined cycle power plant

The traditional control strategy based on PID controllers may be unsatisfactory when dealing with processes with large time delay and constraints. This paper presents a supervisory model based constrained predictive controller (MPC) for a combined cycle power plant (CCPP). First, a non-linear dynamic model of CCPP using the laws of physics was proposed. Then, the supervisory control using the linear constrained MPC method was designed to tune the performance of the PID controllers by including output constraints and manipulating the set points. This scheme showed excellent tracking and disturbance rejection results and improved performance compared with a stand-alone PID controller’s scheme

Distributed model predictive control of steam/water loop in large scale ships

In modern steam power plants, the ever-increasing complexity requires great reliability and flexibility of the control system. Hence, in this paper, the feasibility of a distributed model predictive control (DiMPC) strategy with an extended prediction self-adaptive control (EPSAC) framework is studied, in which the multiple controllers allow each sub-loop to have its own requirement flexibility. Meanwhile, the model predictive control can guarantee a good performance for the system with constraints. The performance is compared against a decentralized model predictive control (DeMPC) and a centralized model predictive control (CMPC). In order to improve the computing speed, a multiple objective model predictive control (MOMPC) is proposed. For the stability of the control system, the convergence of the DiMPC is discussed. Simulation tests are performed on the five different sub-loops of steam/water loop. The results indicate that the DiMPC may achieve similar performance as CMPC while outperforming the DeMPC method

Application of Robust Model Predictive Control to a Renewable Hydrogen-based Microgrid

In order to cope with uncertainties present in the renewable energy generation, as well as in the demand consumer, we propose in this paper the formulation and comparison of three robust model predictive control techniques, i. i. e., multi-scenario, tree-based, and chance-constrained model predictive control, which are applied to a nonlinear plant-replacement model that corresponds to a real laboratory-scale plant located in the facilities of the University of Seville. Results show the effectiveness of these three techniques considering the stochastic nature, proper of these systems