Fine-grained acceleration control for autonomous intersection management using deep reinforcement learning

Recent advances in combining deep learning and Reinforcement Learning have shown a promising path for designing new control agents that can learn optimal policies for challenging control tasks. These new methods address the main limitations of conventional Reinforcement Learning methods such as customized feature engineering and small action/state space dimension requirements. In this paper, we leverage one of the state-of-the-art Reinforcement Learning methods, known as Trust Region Policy Optimization, to tackle intersection management for autonomous vehicles. We show that using this method, we can perform fine-grained acceleration control of autonomous vehicles in a grid street plan to achieve a global design objective.Comment: Accepted in IEEE Smart World Congress 201

Doctor of Philosophy

dissertationTraffic congestion occurs because the available capacity cannot serve the desired demand on a portion of the roadway at a particular time. Major sources of congestion include recurring bottlenecks, incidents, work zones, inclement weather, poor signal timing, and day-to-day fluctuations in normal traffic demand. This dissertation addresses a series of critical and challenging issues in evaluating the benefits of Advanced Traveler Information Strategies under different uncertainty modeling approaches are integrated in this dissertation, namely: mathematical programming, dynamic simulation and analytical approximation. The proposed models aim to 1) represent static-state network user equilibrium conditions, knowledge quality and accessibility of traveler information systems under both stochastic capacity and stochastic demand distributions; 2) characterize day-to-day learning behavior with different information groups under stochastic capacity and 3) quantify travel time variability from stochastic capacity distribution functions on critical bottlenecks. First, a nonlinear optimization-based conceptual framework is proposed for incorporating stochastic capacity, stochastic demand, travel time performance functions and varying degrees of traveler knowledge in an advanced traveler information provision environment. This method categorizes commuters into two classes: (1) those with access to perfect traffic information every day, and (2) those with knowledge of the expected traffic conditions across different days. Using a gap function framework, two mathematical programming models are further formulated to describe the route choice behavior of the perfect information and expected travel time user classes under stochastic day-dependent travel time. This dissertation also presents adaptive day-to-day traveler learning and route choice behavioral models under the travel time variability. To account for different levels of information availability and cognitive limitations of individual travelers, a set of "bounded rationality" rules are adapted to describe route choice rules for a traffic system with inherent process noise and different information provision strategies. In addition, this dissertation investigates a fundamental problem of quantifying travel time variability from its root sources: stochastic capacity and demand variations that follow commonly used log-normal distributions. The proposed models provide theoretically rigorous and practically usefully tools to understand the causes of travel time unreliability and evaluate the system-wide benefit of reducing demand and capacity variability

Data-driven Methodologies and Applications in Urban Mobility

The world is urbanizing at an unprecedented rate where urbanization goes from 39% in 1980 to 58% in 2019 (World Bank, 2019). This poses more and more transportation demand and pressure on the already at or over-capacity old transport infrastructure, especially in urban areas. Along the same timeline, more data generated as a byproduct of daily activity are being collected via the advancement of the internet of things, and computers are getting more and more powerful. These are shown by the statistics such as 90% of the world’s data is generated within the last two years and IBM’s computer is now processing at the speed of 120,000 GPS points per second. Thus, this dissertation discusses the challenges and opportunities arising from the growing demand for urban mobility, particularly in cities with outdated infrastructure, and how to capitalize on the unprecedented growth in data in solving these problems by ways of data-driven transportation-specific methodologies. The dissertation identifies three primary challenges and/or opportunities, which are (1) optimally locating dynamic wireless charging to promote the adoption of electric vehicles, (2) predicting dynamic traffic state using an enormously large dataset of taxi trips, and (3) improving the ride-hailing system with carpooling, smart dispatching, and preemptive repositioning. The dissertation presents potential solutions/methodologies that have become available only recently thanks to the extraordinary growth of data and computers with explosive power, and these methodologies are (1) bi-level optimization planning frameworks for locating dynamic wireless charging facilities, (2) Traffic Graph Convolutional Network for dynamic urban traffic state estimation, and (3) Graph Matching and Reinforcement Learning for the operation and management of mixed autonomous electric taxi fleets. These methodologies are then carefully calibrated, methodically scrutinized under various performance metrics and procedures, and validated with previous research and ground truth data, which is gathered directly from the real world. In order to bridge the gap between scientific discoveries and practical applications, the three methodologies are applied to the case study of (1) Montgomery County, MD, (2) the City of New York, and (3) the City of Chicago and from which, real-world implementation are suggested. This dissertation’s contribution via the provided methodologies, along with the continual increase in data, have the potential to significantly benefit urban mobility and work toward a sustainable transportation system

Formulation, existence, and computation of boundedly rational dynamic user equilibrium with fixed or endogenous user tolerance

This paper analyzes dynamic user equilibrium (DUE) that incorporates the notion of boundedly rational (BR) user behavior in the selection of departure times and routes. Intrinsically, the boundedly rational dynamic user equilibrium (BR-DUE) model we present assumes that travelers do not always seek the least costly route-and-departure-time choice. Rather, their perception of travel cost is affected by an indifference band describing travelers’ tolerance of the difference between their experienced travel costs and the minimum travel cost. An extension of the BR-DUE problem is the so-called variable tolerance dynamic user equilibrium (VT-BR-DUE) wherein endogenously determined tolerances may depend not only on paths, but also on the established path departure rates. This paper presents a unified approach for modeling both BR-DUE and VT-BR-DUE, which makes significant contributions to the model formulation, analysis of existence, solution characterization, and numerical computation of such problems. The VT-BR-DUE problem, together with the BR-DUE problem as a special case, is formulated as a variational inequality. We provide a very general existence result for VT-BR-DUE and BR-DUE that relies on assumptions weaker than those required for normal DUE models. Moreover, a characterization of the solution set is provided based on rigorous topological analysis. Finally, three computational algorithms with convergence results are proposed based on the VI and DVI formulations. Numerical studies are conducted to assess the proposed algorithms in terms of solution quality, convergence, and computational efficiency

Pricing local emission exposure of road traffic: An agent-based approach

This paper proposes a new approach to iteratively calculate local air pollution exposure tolls in large-scale urban settings by taking the exposure times and locations of individuals into consideration. It explicitly avoids detailed air pollution concentration calculations and is therefore characterized by little data requirements, reasonable computation times for iterative calculations, and open-source compatibility. In a first step, the paper shows how to derive time-dependent vehicle-specific exposure tolls in an agent-based model. It closes the circle from the polluting entity, to the receiving entity, to damage costs, to tolls, and back to the behavioral change of the polluting entity. In a second step, the approach is applied to a large-scale real-world scenario of the Munich metropolitan area in Germany. Changes in emission levels, exposure costs, and user benefits are calculated. These figures are compared to a flat emission toll, and to a regulatory measure (a speed reduction in the inner city), respectively. The results indicate that the flat emission toll reduces overall emissions more significantly than the exposure toll, but its exposure cost reductions are rather small. For the exposure toll, overall emissions increase for freight traffic which implies a potential conflict between pricing schemes to optimize local emission exposure and others to abate climate change. Regarding the mitigation of exposure costs caused by urban travelers, the regulatory measure is found to be an effective strategy, but it implies losses in user benefits

On inverse reinforcement learning and dynamic discrete choice for predicting path choices

La modélisation du choix d'itinéraire est un sujet de recherche bien étudié avec des implications, par exemple, pour la planification urbaine et l'analyse des flux d'équilibre du trafic. En raison de l'ampleur des effets que ces problèmes peuvent avoir sur les communautés, il n'est pas surprenant que plusieurs domaines de recherche aient tenté de résoudre le même problème. Les défis viennent cependant de la taille des réseaux eux-mêmes, car les grandes villes peuvent avoir des dizaines de milliers de segments de routes reliés par des dizaines de milliers d'intersections. Ainsi, les approches discutées dans cette thèse se concentreront sur la comparaison des performances entre des modèles de deux domaines différents, l'économétrie et l'apprentissage par renforcement inverse (IRL). Tout d'abord, nous fournissons des informations sur le sujet pour que des chercheurs d'un domaine puissent se familiariser avec l'autre domaine. Dans un deuxième temps, nous décrivons les algorithmes utilisés avec une notation commune, ce qui facilite la compréhension entre les domaines. Enfin, nous comparons les performances des modèles sur des ensembles de données du monde réel, à savoir un ensemble de données couvrant des choix d’itinéraire de cyclistes collectés dans un réseau avec 42 000 liens. Nous rapportons nos résultats pour les deux modèles de l'économétrie que nous discutons, mais nous n'avons pas pu générer les mêmes résultats pour les deux modèles IRL. Cela était principalement dû aux instabilités numériques que nous avons rencontrées avec le code que nous avions modifié pour fonctionner avec nos données. Nous proposons une discussion de ces difficultés parallèlement à la communication de nos résultats.Route choice modeling is a well-studied topic of research with implications, for example, for city planning and traffic equilibrium flow analysis. Due to the scale of effects these problems can have on communities, it is no surprise that diverse fields have attempted solutions to the same problem. The challenges, however, come with the size of networks themselves, as large cities may have tens of thousands of road segments connected by tens of thousands of intersections. Thus, the approaches discussed in this thesis will be focusing on the performance comparison between models from two different fields, econometrics and inverse reinforcement learning (IRL). First, we provide background on the topic to introduce researchers from one field to become acquainted with the other. Secondly, we describe the algorithms used with a common notation to facilitate this building of understanding between the fields. Lastly, we aim to compare the performance of the models on real-world datasets, namely covering bike route choices collected in a network of 42,000 links. We report our results for the two models from econometrics that we discuss, but were unable to generate the same results for the two IRL models. This was primarily due to numerical instabilities we encountered with the code we had modified to work with our data. We provide a discussion of these difficulties alongside the reporting of our results

Towards Realistic Urban Traffic Experiments Using DFROUTER: Heuristic, Validation and Extensions

[EN] Traffic congestion is an important problem faced by Intelligent Transportation Systems (ITS), requiring models that allow predicting the impact of different solutions on urban traffic flow. Such an approach typically requires the use of simulations, which should be as realistic as possible. However, achieving high degrees of realism can be complex when the actual traffic patterns, defined through an Origin/Destination (O-D) matrix for the vehicles in a city, remain unknown. Thus, the main contribution of this paper is a heuristic for improving traffic congestion modeling. In particular, we propose a procedure that, starting from real induction loop measurements made available by traffic authorities, iteratively refines the output of DFROUTER, which is a module provided by the SUMO (Simulation of Urban MObility) tool. This way, it is able to generate an O-D matrix for traffic that resembles the real traffic distribution and that can be directly imported by SUMO. We apply our technique to the city of Valencia, and we then compare the obtained results against other existing traffic mobility data for the cities of Cologne (Germany) and Bologna (Italy), thereby validating our approach. We also use our technique to determine what degree of congestion is expectable if certain conditions cause additional traffic to circulate in the city, adopting both a uniform pattern and a hotspot-based pattern for traffic injection to demonstrate how to regulate the overall number of vehicles in the city. This study allows evaluating the impact of vehicle flow changes on the overall traffic congestion levels.This work was partially supported by Valencia’s Traffic Management Department and by the “Ministerio de Economía y Competitividad, Programa Estatal de Investigación, Desarrollo e Innovación Orientada a los Retos de la Sociedad, Proyectos I+D+I 2014”, Spain, under Grant TEC2014-52690-R, and the “Programa de Becas SENESCYT”de la República del Ecuador.Zambrano-Martinez, J.; Tavares De Araujo Cesariny Calafate, CM.; Soler Fernández, D.; Cano, J. (2017). Towards Realistic Urban Traffic Experiments Using DFROUTER: Heuristic, Validation and Extensions. Sensors. 17(12):1-29. doi:10.3390/s17122921S129171