CoLight: Learning Network-level Cooperation for Traffic Signal Control

Cooperation among the traffic signals enables vehicles to move through intersections more quickly. Conventional transportation approaches implement cooperation by pre-calculating the offsets between two intersections. Such pre-calculated offsets are not suitable for dynamic traffic environments. To enable cooperation of traffic signals, in this paper, we propose a model, CoLight, which uses graph attentional networks to facilitate communication. Specifically, for a target intersection in a network, CoLight can not only incorporate the temporal and spatial influences of neighboring intersections to the target intersection, but also build up index-free modeling of neighboring intersections. To the best of our knowledge, we are the first to use graph attentional networks in the setting of reinforcement learning for traffic signal control and to conduct experiments on the large-scale road network with hundreds of traffic signals. In experiments, we demonstrate that by learning the communication, the proposed model can achieve superior performance against the state-of-the-art methods.Comment: 10 pages. Proceedings of the 28th ACM International on Conference on Information and Knowledge Management. ACM, 201

Improving Traffic Safety and Efficiency by Adaptive Signal Control Systems Based on Deep Reinforcement Learning

As one of the most important Active Traffic Management strategies, Adaptive Traffic Signal Control (ATSC) helps improve traffic operation of signalized arterials and urban roads by adjusting the signal timing to accommodate real-time traffic conditions. Recently, with the rapid development of artificial intelligence, many researchers have employed deep reinforcement learning (DRL) algorithms to develop ATSCs. However, most of them are not practice-ready. The reasons are two-fold: first, they are not developed based on real-world traffic dynamics and most of them require the complete information of the entire traffic system. Second, their impact on traffic safety is always a concern by researchers and practitioners but remains unclear. Aiming at making the DRL-based ATSC more implementable, existing traffic detection systems on arterials were reviewed and investigated to provide high-quality data feeds to ATSCs. Specifically, a machine-learning frameworks were developed to improve the quality of and pedestrian and bicyclist\u27s count data. Then, to evaluate the effectiveness of DRL-based ATSC on the real-world traffic dynamics, a decentralized network-level ATSC using multi-agent DRL was developed and evaluated in a simulated real-world network. The evaluation results confirmed that the proposed ATSC outperforms the actuated traffic signals in the field in terms of travel time reduction. To address the potential safety issue of DRL based ATSC, an ATSC algorithm optimizing simultaneously both traffic efficiency and safety was proposed based on multi-objective DRL. The developed ATSC was tested in a simulated real-world intersection and it successfully improved traffic safety without deteriorating efficiency. In conclusion, the proposed ATSCs are capable of effectively controlling real-world traffic and benefiting both traffic efficiency and safety

Neurofuzzy control to address stochastic variation in actuated-coordinated systems at closely-spaced intersections

This dissertation documents a method of addressing stochastic variation at closely-spaced signalized intersections using neurofuzzy control. Developed on the conventional actuated-coordinated control system, the neurofuzzy traffic signal control keeps the advantage of the conventional control system. Beyond this, the neurofuzzy signal control coordinates the coordinated phase with one of the non-coordinated phases with no reduction of the green band assigned to the coordination along the arterial, reduces variations of traffic signal times in the cycle caused by early return to green , hence, makes more sufficient utilization of green time at closely-spaced intersections. The neurofuzzy signal control system manages a non-coordinated movement in order to manage queue spillbacks and variations of signal timings.Specifically, the neurofuzzy controller establishes a secondary coordination between the upstream coordinated phase (through phase) and the downstream non-coordinated phase (left turn phase) based on real-time traffic demand. Under the fuzzy logic signal control, the traffic from the upstream intersection can arrive and join the queue at the downstream left turn lane and be served, and hence, less possibly be blocked on the downstream left turn lane. This secondary coordination favors left turn progression and, hence, reduces the queue spillbacks. The fuzzy logic method overcomes the natural disadvantage of currently widely used actuated-coordinated traffic signal control in that the fuzzy logic method could coordinate a coordinated movement with a non-coordinated movement. The experiment was conducted and evaluated using a simulation model created using the microscopic simulation program - VISSIM.The neurofuzzy control algorithm was coded with MATLAB which interacts with the traffic simulation model via VISSIM\u27s COM interface. The membership functions in the neurofuzzy signal control system were calibrated using reinforcement learning to further the performance. Comparisons were made between the trained neurofuzzy control, the untrained neurofuzzy control, and the conventional actuated-coordinated control under five different traffic volumes. The simulation results indicated that the trained neurofuzzy signal control outperformed the other two for each traffic case. Comparing to the conventional actuated-coordinated control, the trained neurofuzzy signal control reduced the average delay by 7% and the average number of stops by 6% under the original traffic volume; as traffic volume increasing to 120%, the reductions doubled

Development and evaluation of an arterial adaptive traffic signal control system using reinforcement learning

This dissertation develops and evaluates a new adaptive traffic signal control system for arterials. This control system is based on reinforcement learning, which is an important research area in distributed artificial intelligence and has been extensively used in many applications including real-time control. In this dissertation, a systematic comparison between the reinforcement learning control methods and existing adaptive traffic control methods is first presented from the theoretical perspective. This comparison shows both the connections between them and the benefits of using reinforcement learning. A Neural-Fuzzy Actor-Critic Reinforcement Learning (NFACRL) method is then introduced for traffic signal control. NFACRL integrates fuzzy logic and neural networks into reinforcement learning and can better handle the curse of dimensionality and generalization problems associated with ordinary reinforcement learning methods. This NFACRL method is first applied to isolated intersection control. Two different implementation schemes are considered. The first scheme uses a fixed phase sequence and variable cycle length, while the second one optimizes phase sequence in real time and is not constrained to the concept of cycle. Both schemes are further extended for arterial control, with each intersection being controlled by one NFACRL controller. Different strategies used for coordinating reinforcement learning controllers are reviewed, and a simple but robust method is adopted for coordinating traffic signals along the arterial. The proposed NFACRL control system is tested at both isolated intersection and arterial levels based on VISSIM simulation. The testing is conducted under different traffic volume scenarios using real-world traffic data collected during morning, noon, and afternoon peak periods. The performance of the NFACRL control system is compared with that of the optimized pre-timed and actuated control. Testing results based on VISSIM simulation show that the proposed NFACRL control has very promising performance. It outperforms optimized pre-timed and actuated control in most cases for both isolated intersection and arterial control. At the end of this dissertation, issues on how to further improve the NFACRL method and implement it in real world are discussed

Reinforcement Learning Approaches for Traffic Signal Control under Missing Data

The emergence of reinforcement learning (RL) methods in traffic signal control tasks has achieved better performance than conventional rule-based approaches. Most RL approaches require the observation of the environment for the agent to decide which action is optimal for a long-term reward. However, in real-world urban scenarios, missing observation of traffic states may frequently occur due to the lack of sensors, which makes existing RL methods inapplicable on road networks with missing observation. In this work, we aim to control the traffic signals in a real-world setting, where some of the intersections in the road network are not installed with sensors and thus with no direct observations around them. To the best of our knowledge, we are the first to use RL methods to tackle the traffic signal control problem in this real-world setting. Specifically, we propose two solutions: the first one imputes the traffic states to enable adaptive control, and the second one imputes both states and rewards to enable adaptive control and the training of RL agents. Through extensive experiments on both synthetic and real-world road network traffic, we reveal that our method outperforms conventional approaches and performs consistently with different missing rates. We also provide further investigations on how missing data influences the performance of our model.Comment: Published as a conference paper at IJCAI202