Towards Robust Deep Reinforcement Learning for Traffic Signal Control: Demand Surges, Incidents and Sensor Failures

Reinforcement learning (RL) constitutes a promising solution for alleviating the problem of traffic congestion. In particular, deep RL algorithms have been shown to produce adaptive traffic signal controllers that outperform conventional systems. However, in order to be reliable in highly dynamic urban areas, such controllers need to be robust with the respect to a series of exogenous sources of uncertainty. In this paper, we develop an open-source callback-based framework for promoting the flexible evaluation of different deep RL configurations under a traffic simulation environment. With this framework, we investigate how deep RL-based adaptive traffic controllers perform under different scenarios, namely under demand surges caused by special events, capacity reductions from incidents and sensor failures. We extract several key insights for the development of robust deep RL algorithms for traffic control and propose concrete designs to mitigate the impact of the considered exogenous uncertainties.Comment: 8 page

Review of SDN-based load-balancing methods, issues, challenges, and roadmap

The development of the Internet and smart end systems, such as smartphones and portable laptops, along with the emergence of cloud computing, social networks, and the Internet of Things, has brought about new network requirements. To meet these requirements, a new architecture called software-defined network (SDN) has been introduced. However, traffic distribution in SDN has raised challenges, especially in terms of uneven load distribution impacting network performance. To address this issue, several SDN load balancing (LB) techniques have been developed to improve efficiency. This article provides an overview of SDN and its effect on load balancing, highlighting key elements and discussing various load-balancing schemes based on existing solutions and research challenges. Additionally, the article outlines performance metrics used to evaluate these algorithms and suggests possible future research directions

Adaptive Coordination Offsets for Signalized Arterial Intersections using Deep Reinforcement Learning

One of the most critical components of an urban transportation system is the coordination of intersections in arterial networks. With the advent of data-driven approaches for traffic control systems, deep reinforcement learning (RL) has gained significant traction in traffic control research. Proposed deep RL solutions to traffic control are designed to directly modify either phase order or timings; such approaches can lead to unfair situations -- bypassing low volume links for several cycles -- in the name of optimizing traffic flow. To address the issues and feasibility of the present approach, we propose a deep RL framework that dynamically adjusts the offsets based on traffic states and preserves the planned phase timings and order derived from model-based methods. This framework allows us to improve arterial coordination while preserving the notion of fairness for competing streams of traffic in an intersection. Using a validated and calibrated traffic model, we trained the policy of a deep RL agent that aims to reduce travel delays in the network. We evaluated the resulting policy by comparing its performance against the phase offsets obtained by a state-of-the-practice baseline, SYNCHRO. The resulting policy dynamically readjusts phase offsets in response to changes in traffic demand. Simulation results show that the proposed deep RL agent outperformed SYNCHRO on average, effectively reducing delay time by 13.21% in the AM Scenario, 2.42% in the noon scenario, and 6.2% in the PM scenario. Finally, we also show the robustness of our agent to extreme traffic conditions, such as demand surges and localized traffic incidents

Reinforcement learning for traffic signal control : comparison with commercial systems

Intelligent Transportation Systems are leveraging the power of increased sensory coverage and computing power to deliver data-intensive solutions achieving higher levels of performance than traditional systems. Within Traffic Signal Control (TSC), this has allowed the emergence of Machine Learning (ML) based systems. Among this group, Reinforcement Learning (RL) approaches have performed particularly well. Given the lack of industry standards in ML for TSC, literature exploring RL often lacks comparison against commercially available systems and straightforward formulations of how the agents operate. Here we attempt to bridge that gap. We propose three different architectures for TSC RL agents and compare them against the currently used commercial systems MOVA, SurTrac and Cyclic controllers and provide pseudo-code for them. The agents use variations of Deep Q-Learning and Actor Critic, using states and rewards based on queue lengths. Their performance is compared in across different map scenarios with variable demand, assessing them in terms of the global delay and average queue length. We find that the RL-based systems can significantly and consistently achieve lower delays when compared with existing commercial systems

Traffic Optimization Through Waiting Prediction and Evolutive Algorithms

Traffic optimization systems require optimization procedures to optimize traffic light timing settings in order to improve pedestrian and vehicle mobility. Traffic simulators allow obtaining accurate estimates of traffic behavior by applying different timing configurations, but require considerable computational time to perform validation tests. For this reason, this project proposes the development of traffic optimizations based on the estimation of vehicle waiting times through the use of different prediction techniques and the use of this estimation to subsequently apply evolutionary algorithms that allow the optimizations to be carried out. The combination of these two techniques leads to a considerable reduction in calculation time, which makes it possible to apply this system at runtime. The tests have been carried out on a real traffic junction on which different traffic volumes have been applied to analyze the performance of the system