Supervised learning from human performance at the computationally hard problem of optimal traffic signal control on a network of junctions

Optimal switching of traffic lights on a network of junctions is a computationally intractable problem. In this research, road traffic networks containing signallized junctions are simulated. A computer game interface is used to enable a human ‘player’ to control the traffic light settings on the junctions within the simulation. A supervised learning approach, based on simple neural network classifiers can be used to capture human player's strategies in the game and thus develop a human-trained machine control (HuTMaC) system that approaches human levels of performance. Experiments conducted within the simulation compare the performance of HuTMaC to two well-established traffic-responsive control systems that are widely deployed in the developed world and also to a temporal difference learning-based control method. In all experiments, HuTMaC outperforms the other control methods in terms of average delay and variance over delay. The conclusion is that these results add weight to the suggestion that HuTMaC may be a viable alternative, or supplemental method, to approximate optimization for some practical engineering control problems where the optimal strategy is computationally intractable

Modeling, Control, and Impact Analysis of The Next Generation Transportation System

This dissertation aims to develop a systematic tool designated for connected and autonomous vehicles, integrating the simulation of traffic dynamics, traffic control strategies, and impact analysis at the network level. The first part of the dissertation is devoted to the traffic flow modeling of connected vehicles. This task is the foundation step for transportation planning, optimized network design, efficient traffic control strategies, etc, of the next generation transportation system. Chapter 2 proposes a cell-based simulation approach to model the proactive driving behavior of connected vehicles. Firstly, a state variable of connected vehicle is introduced to track the trajectory of connected vehicles. Then the exit flow of cells containing connected vehicles is adjusted to simulate the proactive driving behavior, such that the traffic light is green when the connected vehicle arrives at the signalized intersection. Extensive numerical simulation results consistently show that the presence of connected vehicles contributes significantly to the smoothing of traffic flow and vehicular emission reductions in the network. Chapter 3 proposes an optimal estimation approach to calibrate connected vehicles\u27 car-following behavior in a mixed traffic environment. Particularly, the state-space system dynamics is captured by the simplified car-following model with disturbances, where the trajectory of non-connected vehicles are considered as unknown states and the trajectory of connected vehicles are considered as measurements with errors. Objective of the reformulation is to obtain an optimal estimation of states and model parameters simultaneously. It is shown that the customized state-space model is identifiable with the mild assumption that the disturbance covariance of the state update process is diagonal. Then a modified Expectation-Maximization (EM) algorithm based on Kalman smoother is developed to solve the optimal estimation problem. The second part of the dissertation is on traffic control strategies. This task drives the next generation transportation system to a better performance state in terms of safety, mobility, travel time saving, vehicular emission reduction, etc. Chapter 4 develops a novel reinforcement learning algorithm for the challenging coordinated signal control problem. Traffic signals are modeled as intelligent agents interacting with the stochastic traffic environment. The model is built on the framework of coordinated reinforcement learning. The Junction Tree Algorithm based reinforcement learning is proposed to obtain an exact inference of the best joint actions for all the coordinated intersections. The algorithm is implemented and tested with a network containing 18 signalized intersections from a microscopic traffic simulator. Chapter 5 develops a novel linear programming formulation for autonomous intersection control (LPAIC) accounting for traffic dynamics within a connected vehicle environment. Firstly, a lane based bi-level optimization model is introduced to propagate traffic flows in the network. Then the bi-level optimization model is transformed to the linear programming formulation by relaxing the nonlinear constraints with a set of linear inequalities. One special feature of the LPAIC formulation is that the entries of the constraint matrix has only values in {-1, 0, 1}. Moreover, it is proved that the constraint matrix is totally unimodular, the optimal solution exists and contains only integer values. Further, it shows that traffic flows from different lanes pass through the conflict points of the intersection safely and there are no holding flows in the solution. Three numerical case studies are conducted to demonstrate the properties and effectiveness of the LPAIC formulation to solve autonomous intersection control. The third part of the dissertation moves on to the impact analysis of connected vehicles and autonomous vehicles at the network level. This task assesses the positive and negative impacts of the system and provides guidance on transportation planning, traffic control, transportation budget spending, etc. In this part, the impact of different penetration rates of connected vehicle and autonomous vehicles is revealed on the network efficiency of a transportation system. Chapter 6 sets out to model an efficient and fair transportation system accounting for both departure time choice and route choice of a general multi OD network within a dynamic traffic assignment environment. Firstly, a bi-level optimization formulation is introduced based on the link-based traffic flow model. The upper level of the formulation minimizes the total system travel time, whereas the lower level captures traffic flow propagation and the user equilibrium constraint. Then the bi-level formulation is relaxed to a linear programming formulation that produces a lower bound of an efficient and fair system state. An efficient iterative algorithm is proposed to obtain the exact solution. It is shown that the number of iterations is bounded, and the output traffic flow solution is efficient and fair. Finally, two numerical cases (including a single OD network and a multi-OD network) are conducted to demonstrate the performance of the algorithm. The results consistently show that the travel time of different departure rates of the same OD pair are identical and the algorithm converges within two iterations across all test scenarios

Uma abordagem de consciência de máquina ao controle de semáforos de tráfego urbano

Orientador: Ricardo Ribeiro GudwinTese (doutorado) - Universidade Estadual de Campinas, Faculdade de Engenharia Elétrica e de ComputaçãoResumo: Neste trabalho, apresentamos uma arquitetura cognitiva distribuída usada para o controle de tráfego em uma rede urbana. Essa arquitetura se baseia em uma abordagem de consciência de máquina - Teoria do Workspace Global - de forma a usar competição e difusão em broadcast, permitindo que um grupo de controladores de tráfego locais interajam, resultando em melhor desempenho do grupo. A ideia principal é que controladores locais geralmente realizam um comportamento reativo, definindo os tempos de verde e vermelho do semáforo, de acordo com informações locais. Esses controladores locais competem de forma a definir qual deles está experienciando a situação mais crítica. O controlador nas piores condições ganha acesso ao workspace global, e depois realiza uma difusão em broadcast de sua condição (e sua localização) para todos os outros controladores, pedindo sua ajuda para lidar com sua situação. Essa chamada do controlador que acessa o workspace global causará uma interferência no comportamento local reativo, para aqueles controladores locais com alguma chance de ajudar o controlador na situação crítica, contendo o tráfego na sua direção. Esse comportamento do grupo, coordenado pela estratégia do workspace global, transforma o comportamento reativo anterior em uma forma de comportamento deliberativo. Nós mostramos que essa estratégia é capaz de melhorar a média do tempo de viagem de todos os veículos que fluem na rede urbana. Um ganho consistente no desempenho foi conseguido com o controlador "Consciência de Máquina" durante todo o tempo da simulação, em diferentes cenários, indo de 10% até maisde 20%, quando comparado ao controlador "Reativo Paralelo" sem o mecanismo de consciência artificial, produzindo evidência para suportar a hipótese de que um mecanismo de consciência artificial, que difunde serialmente em broadcast conteúdo para processos automáticos, pode trazer vantagens para uma tarefa global realizada por uma sociedade de agentes paralelos que operam juntos por uma meta comumAbstract: In this work, we present a distributed cognitive architecture used to control the traffic in an urban network. This architecture relies on a machine consciousness approach - Global Workspace Theory - in order to use competition and broadcast, allowing a group of local traffic controllers to interact, resulting in a better group performance.The main idea is that the local controllers usually perform a purely reactive behavior, defining the times of red and green lights, according just to local information. These local controllers compete in order to define which of them is experiencing the most critical traffic situation. The controller in the worst condition gains access to the global workspace, further broadcasting its condition (and its location) to all other controllers, asking for their help in dealing with its situation. This call from the controller accessing the global workspace will cause an interference in the reactive local behavior, for those local controllers with some chance in helping the controller in a critical condition, by containing traffic in its direction. This group behavior, coordinated by the global workspace strategy, turns the once reactive behavior into a kind of deliberative one. We show that this strategy is capable of improving the overall mean travel time of vehicles flowing through the urban network. A consistent gain in performance with the "Machine Consciousness" traffic signal controller during all simulation time, throughout different simulated scenarios, could be observed, ranging from around 10% to more than 20%, when compared to the "Parallel Reactive" controller without the artificial consciousness mechanism, producing evidence to support the hypothesis that an artificial consciousness mechanism, which serially broadcasts content to automatic processes, can bring advantages to the global task performed by a society of parallel agents working together for a common goalDoutoradoEngenharia de ComputaçãoDoutor em Engenharia Elétrica153206/2010-1CNPQCAPESFAPES

Improving the generalizability and robustness of large-scale traffic signal control

A number of deep reinforcement-learning (RL) approaches propose to control traffic signals. In this work, we study the robustness of such methods along two axes. First, sensor failures and GPS occlusions create missing-data challenges and we show that recent methods remain brittle in the face of these missing data. Second, we provide a more systematic study of the generalization ability of RL methods to new networks with different traffic regimes. Again, we identify the limitations of recent approaches. We then propose using a combination of distributional and vanilla reinforcement learning through a policy ensemble. Building upon the state-of-the-art previous model which uses a decentralized approach for large-scale traffic signal control with graph convolutional networks (GCNs), we first learn models using a distributional reinforcement learning (DisRL) approach. In particular, we use implicit quantile networks (IQN) to model the state-action return distribution with quantile regression. For traffic signal control problems, an ensemble of standard RL and DisRL yields superior performance across different scenarios, including different levels of missing sensor data and traffic flow patterns. Furthermore, the learning scheme of the resulting model can improve zero-shot transferability to different road network structures, including both synthetic networks and real-world networks (e.g., Luxembourg, Manhattan). We conduct extensive experiments to compare our approach to multi-agent reinforcement learning and traditional transportation approaches. Results show that the proposed method improves robustness and generalizability in the face of missing data, varying road networks, and traffic flows