Combining MPC and reinforcement learning in a model-reference framework for urban traffic signal control

Both model predictive control (MPC) and reinforcement learning (RL) have shown promising results in the control of traffic signals in urban traffic networks. There are, however, a few drawbacks. MPC controllers are not adaptive and therefore perform suboptimal in the presence of the uncertainties that always occur in urban traffic systems. Although very advanced prediction models for urban traffic signal control systems exist, these models also come with a price: the computational complexity of MPC controllers increases with the accuracy of the model. RL techniques involve a time-consuming and data-dependent offline computation, as the agent needs to pursue a training process. The training process is also the main reason why RL techniques have not been employed in real-world urban traffic systems. Through exploration in the training phase the controller may cause a suboptimal and potentially unacceptable bad performance in the system. Besides, most RL techniques do not have any stability and feasibility guarantees. With the goal of mitigating these drawbacks, the model-reference RL adaptive control framework is introduced. RL is used to obtain an adaptive law to adjust a stable baseline controller to follow a set reference. This thesis focusses on the design and analysis of this scheme where MPC control is used to obtain the baseline control input. The computed baseline control input combined with the traffic model used, determines the reference state to be followed. By performing a case study, the training characteristics of the framework are compared to those of a conventional RL-based controller. Besides, the system performance framework is compared to that of a fixed-time controller a conventional MPC controller and a conventional RL-based controller. The simulation shows that the framework outperforms the RL-based controller in terms of performance during training and the general simulation performance of the MPC controller.Mechanical Engineering | Systems and Contro

Combined MPC and reinforcement learning for traffic signal control in urban traffic networks

In general, the performance of model-based controllers cannot be guaranteed under model uncertainties or disturbances, while learning-based controllers require an extensively sufficient training process to perform well. These issues especially hold for large-scale nonlinear systems such as urban traffic networks. In this paper, a new framework is proposed by combining model predictive control (MPC) and reinforcement learning (RL) to provide desired performance for urban traffic networks even during the learning process, despite model uncertainties and disturbances. MPC and RL complement each other very well, since MPC provides a sub-optimal and constraint-satisfying control input while RL provides adaptive control laws and can handle uncertainties and disturbances. The resulting combined framework is applied for traffic signal control (TSC) of an urban traffic network. A case study is carried out to compare the performance of the proposed framework and other baseline controllers. Results show that the proposed combined framework outperforms conventional control methods under system uncertainties, in terms of reducing traffic congestion. Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Team Bart De SchutterControl & SimulationDelft Center for Systems and Contro