Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning

Recent advances in combining deep neural network architectures with reinforcement learning techniques have shown promising potential results in solving complex control problems with high dimensional state and action spaces. Inspired by these successes, in this paper, we build two kinds of reinforcement learning algorithms: deep policy-gradient and value-function based agents which can predict the best possible traffic signal for a traffic intersection. At each time step, these adaptive traffic light control agents receive a snapshot of the current state of a graphical traffic simulator and produce control signals. The policy-gradient based agent maps its observation directly to the control signal, however the value-function based agent first estimates values for all legal control signals. The agent then selects the optimal control action with the highest value. Our methods show promising results in a traffic network simulated in the SUMO traffic simulator, without suffering from instability issues during the training process

Neurofuzzy control to address stochastic variation in actuated-coordinated systems at closely-spaced intersections

This dissertation documents a method of addressing stochastic variation at closely-spaced signalized intersections using neurofuzzy control. Developed on the conventional actuated-coordinated control system, the neurofuzzy traffic signal control keeps the advantage of the conventional control system. Beyond this, the neurofuzzy signal control coordinates the coordinated phase with one of the non-coordinated phases with no reduction of the green band assigned to the coordination along the arterial, reduces variations of traffic signal times in the cycle caused by early return to green , hence, makes more sufficient utilization of green time at closely-spaced intersections. The neurofuzzy signal control system manages a non-coordinated movement in order to manage queue spillbacks and variations of signal timings.Specifically, the neurofuzzy controller establishes a secondary coordination between the upstream coordinated phase (through phase) and the downstream non-coordinated phase (left turn phase) based on real-time traffic demand. Under the fuzzy logic signal control, the traffic from the upstream intersection can arrive and join the queue at the downstream left turn lane and be served, and hence, less possibly be blocked on the downstream left turn lane. This secondary coordination favors left turn progression and, hence, reduces the queue spillbacks. The fuzzy logic method overcomes the natural disadvantage of currently widely used actuated-coordinated traffic signal control in that the fuzzy logic method could coordinate a coordinated movement with a non-coordinated movement. The experiment was conducted and evaluated using a simulation model created using the microscopic simulation program - VISSIM.The neurofuzzy control algorithm was coded with MATLAB which interacts with the traffic simulation model via VISSIM\u27s COM interface. The membership functions in the neurofuzzy signal control system were calibrated using reinforcement learning to further the performance. Comparisons were made between the trained neurofuzzy control, the untrained neurofuzzy control, and the conventional actuated-coordinated control under five different traffic volumes. The simulation results indicated that the trained neurofuzzy signal control outperformed the other two for each traffic case. Comparing to the conventional actuated-coordinated control, the trained neurofuzzy signal control reduced the average delay by 7% and the average number of stops by 6% under the original traffic volume; as traffic volume increasing to 120%, the reductions doubled

Traffic Optimization Through Waiting Prediction and Evolutive Algorithms

Traffic optimization systems require optimization procedures to optimize traffic light timing settings in order to improve pedestrian and vehicle mobility. Traffic simulators allow obtaining accurate estimates of traffic behavior by applying different timing configurations, but require considerable computational time to perform validation tests. For this reason, this project proposes the development of traffic optimizations based on the estimation of vehicle waiting times through the use of different prediction techniques and the use of this estimation to subsequently apply evolutionary algorithms that allow the optimizations to be carried out. The combination of these two techniques leads to a considerable reduction in calculation time, which makes it possible to apply this system at runtime. The tests have been carried out on a real traffic junction on which different traffic volumes have been applied to analyze the performance of the system

Improving Traffic Safety and Efficiency by Adaptive Signal Control Systems Based on Deep Reinforcement Learning

As one of the most important Active Traffic Management strategies, Adaptive Traffic Signal Control (ATSC) helps improve traffic operation of signalized arterials and urban roads by adjusting the signal timing to accommodate real-time traffic conditions. Recently, with the rapid development of artificial intelligence, many researchers have employed deep reinforcement learning (DRL) algorithms to develop ATSCs. However, most of them are not practice-ready. The reasons are two-fold: first, they are not developed based on real-world traffic dynamics and most of them require the complete information of the entire traffic system. Second, their impact on traffic safety is always a concern by researchers and practitioners but remains unclear. Aiming at making the DRL-based ATSC more implementable, existing traffic detection systems on arterials were reviewed and investigated to provide high-quality data feeds to ATSCs. Specifically, a machine-learning frameworks were developed to improve the quality of and pedestrian and bicyclist\u27s count data. Then, to evaluate the effectiveness of DRL-based ATSC on the real-world traffic dynamics, a decentralized network-level ATSC using multi-agent DRL was developed and evaluated in a simulated real-world network. The evaluation results confirmed that the proposed ATSC outperforms the actuated traffic signals in the field in terms of travel time reduction. To address the potential safety issue of DRL based ATSC, an ATSC algorithm optimizing simultaneously both traffic efficiency and safety was proposed based on multi-objective DRL. The developed ATSC was tested in a simulated real-world intersection and it successfully improved traffic safety without deteriorating efficiency. In conclusion, the proposed ATSCs are capable of effectively controlling real-world traffic and benefiting both traffic efficiency and safety

HARL: A Novel Hierachical Adversary Reinforcement Learning for Automoumous Intersection Management

As an emerging technology, Connected Autonomous Vehicles (CAVs) are believed to have the ability to move through intersections in a faster and safer manner, through effective Vehicle-to-Everything (V2X) communication and global observation. Autonomous intersection management is a key path to efficient crossing at intersections, which reduces unnecessary slowdowns and stops through adaptive decision process of each CAV, enabling fuller utilization of the intersection space. Distributed reinforcement learning (DRL) offers a flexible, end-to-end model for AIM, adapting for many intersection scenarios. While DRL is prone to collisions as the actions of multiple sides in the complicated interactions are sampled from a generic policy, restricting the application of DRL in realistic scenario. To address this, we propose a hierarchical RL framework where models at different levels vary in receptive scope, action step length, and feedback period of reward. The upper layer model accelerate CAVs to prevent them from being clashed, while the lower layer model adjust the trends from upper layer model to avoid the change of mobile state causing new conflicts. And the real action of CAV at each step is co-determined by the trends from both levels, forming a real-time balance in the adversarial process. The proposed model is proven effective in the experiment undertaken in a complicated intersection with 4 branches and 4 lanes each branch, and show better performance compared with baselines