CoLight: Learning Network-level Cooperation for Traffic Signal Control

Cooperation among the traffic signals enables vehicles to move through intersections more quickly. Conventional transportation approaches implement cooperation by pre-calculating the offsets between two intersections. Such pre-calculated offsets are not suitable for dynamic traffic environments. To enable cooperation of traffic signals, in this paper, we propose a model, CoLight, which uses graph attentional networks to facilitate communication. Specifically, for a target intersection in a network, CoLight can not only incorporate the temporal and spatial influences of neighboring intersections to the target intersection, but also build up index-free modeling of neighboring intersections. To the best of our knowledge, we are the first to use graph attentional networks in the setting of reinforcement learning for traffic signal control and to conduct experiments on the large-scale road network with hundreds of traffic signals. In experiments, we demonstrate that by learning the communication, the proposed model can achieve superior performance against the state-of-the-art methods.Comment: 10 pages. Proceedings of the 28th ACM International on Conference on Information and Knowledge Management. ACM, 201

Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks

Future wireless networks have a substantial potential in terms of supporting a broad range of complex compelling applications both in military and civilian fields, where the users are able to enjoy high-rate, low-latency, low-cost and reliable information services. Achieving this ambitious goal requires new radio techniques for adaptive learning and intelligent decision making because of the complex heterogeneous nature of the network structures and wireless services. Machine learning (ML) algorithms have great success in supporting big data analytics, efficient parameter estimation and interactive decision making. Hence, in this article, we review the thirty-year history of ML by elaborating on supervised learning, unsupervised learning, reinforcement learning and deep learning. Furthermore, we investigate their employment in the compelling applications of wireless networks, including heterogeneous networks (HetNets), cognitive radios (CR), Internet of things (IoT), machine to machine networks (M2M), and so on. This article aims for assisting the readers in clarifying the motivation and methodology of the various ML algorithms, so as to invoke them for hitherto unexplored services as well as scenarios of future wireless networks.Comment: 46 pages, 22 fig

Modeling, Control, and Impact Analysis of The Next Generation Transportation System

This dissertation aims to develop a systematic tool designated for connected and autonomous vehicles, integrating the simulation of traffic dynamics, traffic control strategies, and impact analysis at the network level. The first part of the dissertation is devoted to the traffic flow modeling of connected vehicles. This task is the foundation step for transportation planning, optimized network design, efficient traffic control strategies, etc, of the next generation transportation system. Chapter 2 proposes a cell-based simulation approach to model the proactive driving behavior of connected vehicles. Firstly, a state variable of connected vehicle is introduced to track the trajectory of connected vehicles. Then the exit flow of cells containing connected vehicles is adjusted to simulate the proactive driving behavior, such that the traffic light is green when the connected vehicle arrives at the signalized intersection. Extensive numerical simulation results consistently show that the presence of connected vehicles contributes significantly to the smoothing of traffic flow and vehicular emission reductions in the network. Chapter 3 proposes an optimal estimation approach to calibrate connected vehicles\u27 car-following behavior in a mixed traffic environment. Particularly, the state-space system dynamics is captured by the simplified car-following model with disturbances, where the trajectory of non-connected vehicles are considered as unknown states and the trajectory of connected vehicles are considered as measurements with errors. Objective of the reformulation is to obtain an optimal estimation of states and model parameters simultaneously. It is shown that the customized state-space model is identifiable with the mild assumption that the disturbance covariance of the state update process is diagonal. Then a modified Expectation-Maximization (EM) algorithm based on Kalman smoother is developed to solve the optimal estimation problem. The second part of the dissertation is on traffic control strategies. This task drives the next generation transportation system to a better performance state in terms of safety, mobility, travel time saving, vehicular emission reduction, etc. Chapter 4 develops a novel reinforcement learning algorithm for the challenging coordinated signal control problem. Traffic signals are modeled as intelligent agents interacting with the stochastic traffic environment. The model is built on the framework of coordinated reinforcement learning. The Junction Tree Algorithm based reinforcement learning is proposed to obtain an exact inference of the best joint actions for all the coordinated intersections. The algorithm is implemented and tested with a network containing 18 signalized intersections from a microscopic traffic simulator. Chapter 5 develops a novel linear programming formulation for autonomous intersection control (LPAIC) accounting for traffic dynamics within a connected vehicle environment. Firstly, a lane based bi-level optimization model is introduced to propagate traffic flows in the network. Then the bi-level optimization model is transformed to the linear programming formulation by relaxing the nonlinear constraints with a set of linear inequalities. One special feature of the LPAIC formulation is that the entries of the constraint matrix has only values in {-1, 0, 1}. Moreover, it is proved that the constraint matrix is totally unimodular, the optimal solution exists and contains only integer values. Further, it shows that traffic flows from different lanes pass through the conflict points of the intersection safely and there are no holding flows in the solution. Three numerical case studies are conducted to demonstrate the properties and effectiveness of the LPAIC formulation to solve autonomous intersection control. The third part of the dissertation moves on to the impact analysis of connected vehicles and autonomous vehicles at the network level. This task assesses the positive and negative impacts of the system and provides guidance on transportation planning, traffic control, transportation budget spending, etc. In this part, the impact of different penetration rates of connected vehicle and autonomous vehicles is revealed on the network efficiency of a transportation system. Chapter 6 sets out to model an efficient and fair transportation system accounting for both departure time choice and route choice of a general multi OD network within a dynamic traffic assignment environment. Firstly, a bi-level optimization formulation is introduced based on the link-based traffic flow model. The upper level of the formulation minimizes the total system travel time, whereas the lower level captures traffic flow propagation and the user equilibrium constraint. Then the bi-level formulation is relaxed to a linear programming formulation that produces a lower bound of an efficient and fair system state. An efficient iterative algorithm is proposed to obtain the exact solution. It is shown that the number of iterations is bounded, and the output traffic flow solution is efficient and fair. Finally, two numerical cases (including a single OD network and a multi-OD network) are conducted to demonstrate the performance of the algorithm. The results consistently show that the travel time of different departure rates of the same OD pair are identical and the algorithm converges within two iterations across all test scenarios

Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning

Recent advances in combining deep neural network architectures with reinforcement learning techniques have shown promising potential results in solving complex control problems with high dimensional state and action spaces. Inspired by these successes, in this paper, we build two kinds of reinforcement learning algorithms: deep policy-gradient and value-function based agents which can predict the best possible traffic signal for a traffic intersection. At each time step, these adaptive traffic light control agents receive a snapshot of the current state of a graphical traffic simulator and produce control signals. The policy-gradient based agent maps its observation directly to the control signal, however the value-function based agent first estimates values for all legal control signals. The agent then selects the optimal control action with the highest value. Our methods show promising results in a traffic network simulated in the SUMO traffic simulator, without suffering from instability issues during the training process