Towards Robust Deep Reinforcement Learning for Traffic Signal Control: Demand Surges, Incidents and Sensor Failures

Reinforcement learning (RL) constitutes a promising solution for alleviating the problem of traffic congestion. In particular, deep RL algorithms have been shown to produce adaptive traffic signal controllers that outperform conventional systems. However, in order to be reliable in highly dynamic urban areas, such controllers need to be robust with the respect to a series of exogenous sources of uncertainty. In this paper, we develop an open-source callback-based framework for promoting the flexible evaluation of different deep RL configurations under a traffic simulation environment. With this framework, we investigate how deep RL-based adaptive traffic controllers perform under different scenarios, namely under demand surges caused by special events, capacity reductions from incidents and sensor failures. We extract several key insights for the development of robust deep RL algorithms for traffic control and propose concrete designs to mitigate the impact of the considered exogenous uncertainties.Comment: 8 page

Macroscopic modelling and robust control of bi-modal multi-region urban road networks

The paper concerns the integration of a bi-modal Macroscopic Fundamental Diagram (MFD) modelling for mixed traffic in a robust control framework for congested single- and multi-region urban networks. The bi-modal MFD relates the accumulation of cars and buses and the outflow (or circulating flow) in homogeneous (both in the spatial distribution of congestion and the spatial mode mixture) bi-modal traffic networks. We introduce the composition of traffic in the network as a parameter that affects the shape of the bi-modal MFD. A linear parameter varying model with uncertain parameter the vehicle composition approximates the original nonlinear system of aggregated dynamics when it is near the equilibrium point for single- and multi-region cities governed by bi-modal MFDs. This model aims at designing a robust perimeter and boundary flow controller for single- and multi-region networks that guarantees robust regulation and stability, and thus smooth and efficient operations, given that vehicle composition is a slow time-varying parameter. The control gain of the robust controller is calculated off-line using convex optimisation. To evaluate the proposed scheme, an extensive simulation-based study for single- and multi-region networks is carried out. To this end, the heterogeneous network of San Francisco where buses and cars share the same infrastructure is partitioned into two homogeneous regions with different modes of composition. The proposed robust control is compared with an optimised pre-timed signal plan and a single-region perimeter control strategy. Results show that the proposed robust control can significantly: (i) reduce the overall congestion in the network; (ii) improve the traffic performance of buses in terms of travel delays and schedule reliability, and; (iii) avoid queues and gridlocks on critical paths of the network

Networked PID control design : a pseudo-probabilistic robust approach

Networked Control Systems (NCS) are feedback/feed-forward control systems where control components (sensors, actuators and controllers) are distributed across a common communication network. In NCS, there exist network-induced random delays in each channel. This paper proposes a method to compensate the effects of these delays for the design and tuning of PID controllers. The control design is formulated as a constrained optimization problem and the controller stability and robustness criteria are incorporated as design constraints. The design is based on a polytopic description of the system using a Poisson pdf distribution of the delay. Simulation results are presented to demonstrate the performance of the proposed method

Robust control tools for traffic monitoring in TCP/AQM networks

Several studies have considered control theory tools for traffic control in communication networks, as for example the congestion control issue in IP (Internet Protocol) routers. In this paper, we propose to design a linear observer for time-delay systems to address the traffic monitoring issue in TCP/AQM (Transmission Control Protocol/Active Queue Management) networks. Due to several propagation delays and the queueing delay, the set TCP/AQM is modeled as a multiple delayed system of a particular form. Hence, appropriate robust control tools as quadratic separation are adopted to construct a delay dependent observer for TCP flows estimation. Note that, the developed mechanism enables also the anomaly detection issue for a class of DoS (Denial of Service) attacks. At last, simulations via the network simulator NS-2 and an emulation experiment validate the proposed methodology

A genetic algorithm for the design of a fuzzy controller for active queue management

Active queue management (AQM) policies are those policies of router queue management that allow for the detection of network congestion, the notification of such occurrences to the hosts on the network borders, and the adoption of a suitable control policy. This paper proposes the adoption of a fuzzy proportional integral (FPI) controller as an active queue manager for Internet routers. The analytical design of the proposed FPI controller is carried out in analogy with a proportional integral (PI) controller, which recently has been proposed for AQM. A genetic algorithm is proposed for tuning of the FPI controller parameters with respect to optimal disturbance rejection. In the paper the FPI controller design metodology is described and the results of the comparison with random early detection (RED), tail drop, and PI controller are presented

Nonminimal state space approach to multivariable ramp metering control of motorway bottlenecks

The paper discusses the automatic control of motorway traffic flows utilising ramp metering, i.e. traffic lights on the on-ramp entrances. A multivariable ramp metering system is developed, based on the nonminimal state space (NMSS) approach to control system design using adaptive proportional-integral-plus, linear quadratic (PIP–LQ) optimal controllers. The controller is evaluated on a nonlinear statistical traffic model (STM) simulation of the Amsterdam motorway ring road near the Coen Tunnel

Fuzzy-logic-based control, filtering, and fault detection for networked systems: A Survey

This paper is concerned with the overview of the recent progress in fuzzy-logic-based filtering, control, and fault detection problems. First, the network technologies are introduced, the networked control systems are categorized from the aspects of fieldbuses and industrial Ethernets, the necessity of utilizing the fuzzy logic is justified, and the network-induced phenomena are discussed. Then, the fuzzy logic control strategies are reviewed in great detail. Special attention is given to the thorough examination on the latest results for fuzzy PID control, fuzzy adaptive control, and fuzzy tracking control problems. Furthermore, recent advances on the fuzzy-logic-based filtering and fault detection problems are reviewed. Finally, conclusions are given and some possible future research directions are pointed out, for example, topics on two-dimensional networked systems, wireless networked control systems, Quality-of-Service (QoS) of networked systems, and fuzzy access control in open networked systems.This work was supported in part by the National Natural Science Foundation of China under Grants 61329301, 61374039, 61473163, and 61374127, the Hujiang Foundation of China under Grants C14002 andD15009, the Engineering and Physical Sciences Research Council (EPSRC) of the UK, the Royal Society of the UK, and the Alexander von Humboldt Foundation of Germany