Sharing diverse information gets driver agents to learn faster : an application in en route trip building

With the increase in the use of private transportation, developing more efficient ways to distribute routes in a traffic network has become more and more important. Several attempts to address this issue have already been proposed, either by using a central authority to assign routes to the vehicles, or by means of a learning process where drivers select their best routes based on their previous experiences. The present work addresses a way to connect reinforcement learning to new technologies such as car-to-infrastructure communication in order to augment the drivers knowledge in an attempt to accelerate the learning process. Our method was compared to both a classical, iterative approach, as well as to standard reinforcement learning without communication. Results show that our method outperforms both of them. Further, we have performed robustness tests, by allowing messages to be lost, and by reducing the storage capacity of the communication devices. We were able to show that our method is not only tolerant to information loss, but also points out to improved performance when not all agents get the same information. Hence, we stress the fact that, before deploying communication in urban scenarios, it is necessary to take into consideration that the quality and diversity of information shared are key aspects

Book of abstracts of the 24th Euro Working Group on Transportation Meeting

Sem resumo disponível.publishe

Deep Reinforcement Learning Approach for Lagrangian Control: Improving Freeway Bottleneck Throughput Via Variable Speed Limit

Connected vehicles (CVs) will enable new applications to improve traffic flow. The focus of this dissertation is to investigate how reinforcement learning (RL) control for the variable speed limit (VSL) through CVs can be generalized to improve traffic flow at different freeway bottlenecks. Three different bottlenecks are investigated: A sag curve, where the gradient changes from negative to positive values causes a reduction in the roadway capacity and congestion; a lane reduction, where three lanes merge to two lanes and cause congestion, and finally, an on-ramp, where increase in demand on a multilane freeway causes capacity drop. An RL algorithm is developed and implemented in a simulation environment for controlling a VSL in the upstream to manipulate the inflow of vehicles to the bottleneck on a freeway to minimize delays and increase the throughput. CVs are assumed to receive VSL messages through Infrastructure-to-Vehicle (I2V) communications technologies. Asynchronous Advantage Actor-Critic (A3C) algorithms are developed for each bottleneck to determine optimal VSL policies. Through these RL control algorithms, the speed of CVs are manipulated in the upstream of the bottleneck to avoid or minimize congestion. Various market penetration rates for CVs are considered in the simulations. It is demonstrated that the RL algorithm is able to adapt to stochastic arrivals of CVs and achieve significant improvements even at low market penetration rates of CVs, and the RL algorithm is able to find solution for all three bottlenecks. The results also show that the RL-based solutions outperform feedback-control-based solutions

Modeling, Control, and Impact Analysis of The Next Generation Transportation System

This dissertation aims to develop a systematic tool designated for connected and autonomous vehicles, integrating the simulation of traffic dynamics, traffic control strategies, and impact analysis at the network level. The first part of the dissertation is devoted to the traffic flow modeling of connected vehicles. This task is the foundation step for transportation planning, optimized network design, efficient traffic control strategies, etc, of the next generation transportation system. Chapter 2 proposes a cell-based simulation approach to model the proactive driving behavior of connected vehicles. Firstly, a state variable of connected vehicle is introduced to track the trajectory of connected vehicles. Then the exit flow of cells containing connected vehicles is adjusted to simulate the proactive driving behavior, such that the traffic light is green when the connected vehicle arrives at the signalized intersection. Extensive numerical simulation results consistently show that the presence of connected vehicles contributes significantly to the smoothing of traffic flow and vehicular emission reductions in the network. Chapter 3 proposes an optimal estimation approach to calibrate connected vehicles\u27 car-following behavior in a mixed traffic environment. Particularly, the state-space system dynamics is captured by the simplified car-following model with disturbances, where the trajectory of non-connected vehicles are considered as unknown states and the trajectory of connected vehicles are considered as measurements with errors. Objective of the reformulation is to obtain an optimal estimation of states and model parameters simultaneously. It is shown that the customized state-space model is identifiable with the mild assumption that the disturbance covariance of the state update process is diagonal. Then a modified Expectation-Maximization (EM) algorithm based on Kalman smoother is developed to solve the optimal estimation problem. The second part of the dissertation is on traffic control strategies. This task drives the next generation transportation system to a better performance state in terms of safety, mobility, travel time saving, vehicular emission reduction, etc. Chapter 4 develops a novel reinforcement learning algorithm for the challenging coordinated signal control problem. Traffic signals are modeled as intelligent agents interacting with the stochastic traffic environment. The model is built on the framework of coordinated reinforcement learning. The Junction Tree Algorithm based reinforcement learning is proposed to obtain an exact inference of the best joint actions for all the coordinated intersections. The algorithm is implemented and tested with a network containing 18 signalized intersections from a microscopic traffic simulator. Chapter 5 develops a novel linear programming formulation for autonomous intersection control (LPAIC) accounting for traffic dynamics within a connected vehicle environment. Firstly, a lane based bi-level optimization model is introduced to propagate traffic flows in the network. Then the bi-level optimization model is transformed to the linear programming formulation by relaxing the nonlinear constraints with a set of linear inequalities. One special feature of the LPAIC formulation is that the entries of the constraint matrix has only values in {-1, 0, 1}. Moreover, it is proved that the constraint matrix is totally unimodular, the optimal solution exists and contains only integer values. Further, it shows that traffic flows from different lanes pass through the conflict points of the intersection safely and there are no holding flows in the solution. Three numerical case studies are conducted to demonstrate the properties and effectiveness of the LPAIC formulation to solve autonomous intersection control. The third part of the dissertation moves on to the impact analysis of connected vehicles and autonomous vehicles at the network level. This task assesses the positive and negative impacts of the system and provides guidance on transportation planning, traffic control, transportation budget spending, etc. In this part, the impact of different penetration rates of connected vehicle and autonomous vehicles is revealed on the network efficiency of a transportation system. Chapter 6 sets out to model an efficient and fair transportation system accounting for both departure time choice and route choice of a general multi OD network within a dynamic traffic assignment environment. Firstly, a bi-level optimization formulation is introduced based on the link-based traffic flow model. The upper level of the formulation minimizes the total system travel time, whereas the lower level captures traffic flow propagation and the user equilibrium constraint. Then the bi-level formulation is relaxed to a linear programming formulation that produces a lower bound of an efficient and fair system state. An efficient iterative algorithm is proposed to obtain the exact solution. It is shown that the number of iterations is bounded, and the output traffic flow solution is efficient and fair. Finally, two numerical cases (including a single OD network and a multi-OD network) are conducted to demonstrate the performance of the algorithm. The results consistently show that the travel time of different departure rates of the same OD pair are identical and the algorithm converges within two iterations across all test scenarios