The Real Deal: A Review of Challenges and Opportunities in Moving Reinforcement Learning-Based Traffic Signal Control Systems Towards Reality

Traffic signal control (TSC) is a high-stakes domain that is growing in importance as traffic volume grows globally. An increasing number of works are applying reinforcement learning (RL) to TSC; RL can draw on an abundance of traffic data to improve signalling efficiency. However, RL-based signal controllers have never been deployed. In this work, we provide the first review of challenges that must be addressed before RL can be deployed for TSC. We focus on four challenges involving (1) uncertainty in detection, (2) reliability of communications, (3) compliance and interpretability, and (4) heterogeneous road users. We show that the literature on RL-based TSC has made some progress towards addressing each challenge. However, more work should take a systems thinking approach that considers the impacts of other pipeline components on RL.Comment: 26 pages; accepted version, with shortened version published at the 12th International Workshop on Agents in Traffic and Transportation (ATT '22) at IJCAI 202

Alleviating Traffic Congestion: Developing and Evaluating Novel Multi-Agent Reinforcement Learning Traffic Light Coordination Techniques

Contract # 69A3551747111Traffic congestion costs American cities tens of billions of dollars per year, not to mention its negative impact on the environment or people\u2019s mental health. Novel Markov game models and advanced reinforcement learning algorithms hold the promise of drastically alleviating congestion through dynamic coordination of traffic signals and adaptive techniques to dynamically re-route traffic. This project involves a collaboration with Econolite, a leading provider of traffic management systems

Modeling and optimizing network infrastructure for autonomous vehicles

Autonomous vehicle (AV) technology has matured sufficiently to be in testing on public roads. However, traffic models of AVs are still in development. Most previous work has studied AV technologies in micro-simulation. The purpose of this dissertation is to model and optimize AV technologies for large city networks to predict how AVs might affect city traffic patterns and travel behaviors. To accomplish these goals, we construct a dynamic network loading model for AVs, consisting of link and node models of AV technologies, which is used to calculate time-dependent travel times in dynamic traffic assignment. We then study several applications of the dynamic network loading to predict how AVs might affect travel demand and traffic congestion. AVs admit reduced perception-reaction times through technologies such as (cooperative) adaptive cruise control, which can reduce following headways and increase capacity. Previous work has studied these in micro-simulation, but we construct a mesoscopic simulation model for analyses on large networks. To study scenarios with both autonomous and conventional vehicles, we modify the kinematic wave theory to include multiple classes of flow. The flow-density relationship also changes in space and time with the class proportions. We present multiclass cell transmission model and prove that it is a Godunov approximation to the multiclass kinematic wave theory. We also develop a car-following model to predict the fundamental diagram at arbitrary proportions of AVs. Complete market penetration scenarios admit dynamic lane reversal -- changing lane direction at high frequencies to more optimally allocate road capacity. We develop a kinematic wave theory in which the number of lanes changes in space and time, and approximately solve it with a cell transmission model. We study two methods of determining lane direction. First, we present a mixed integer linear program for system optimal dynamic traffic assignment. Since this program is computationally difficult to solve, we also study dynamic lane reversal on a single link with deterministic and stochastic demands. The resulting policy is shown to significantly reduce travel times on a city network. AVs also admit reservation-based intersection control, which can make greater use of intersection capacity than traffic signals. AVs communicate with the intersection manager to reserve space-time paths through the intersection. We create a mesoscopic node model by starting with the conflict point variant of reservations and aggregating conflict points into capacity-constrained conflict regions. This model yields an integer program that can be adapted to arbitrary objective functions. To motivate optimization, we present several examples on theoretical and realistic networks demonstrating that naïve reservation policies can perform worse than traffic signals. These occur due to asymmetric intersections affecting optimal capacity allocation and/or user equilibrium route choice behavior. To improve reservations, we adapt the decentralized backpressure wireless packet routing and P0 traffic signal policies for reservations. Results show significant reductions in travel times on a city network. Having developed link and node models, we explore how AVs might affect travel demand and congestion. First, we study how capacity increases and reservations might affect freeway, arterial, and city networks. Capacity increases consistently reduced congestion on all networks, but reservations were not always beneficial. Then, we use dynamic traffic assignment within a four-step planning model, adding the mode choice of empty repositioning trips to avoid parking costs. Results show that allowing empty repositioning to encourage adoption of AVs could reduce congestion. Also, once all vehicles are AVs, congestion will still be significantly reduced. Finally, we present a framework to use the dynamic network loading model to study shared AVs. Results show that shared AVs could reduce congestion if used in certain ways, such as with dynamic ride-sharing. However, shared AVs also cause significant congestion. To summarize, this dissertation presents a complete mesoscopic simulation model of AVs that could be used for a variety of studies of AVs by planners and practitioners. This mesoscopic model includes new node and link technologies that significantly improve travel times over existing infrastructure. In addition, we motivate and present more optimal policies for these AV technologies. Finally, we study several travel behavior scenarios to provide insights about how AV technologies might affect future traffic congestion. The models in this dissertation will provide a basis for future network analyses of AV technologies.Civil, Architectural, and Environmental Engineerin

Deep learning for real-time traffic signal control on urban networks

Real-time traffic signal controls are frequently challenged by (1) uncertain knowledge about the traffic states; (2) need for efficient computation to allow timely decisions; (3) multiple objectives such as traffic delays and vehicle emissions that are difficult to optimize; and (4) idealized assumptions about data completeness and quality that are often made in developing many theoretical signal control models. This thesis addresses these challenges by proposing two real-time signal control frameworks based on deep learning techniques, followed by extensive simulation tests that verifies their effectiveness in view of the aforementioned challenges. The first method, called the Nonlinear Decision Rule (NDR), defines a nonlinear mapping between network states and signal control parameters to network performances based on prevailing traffic conditions, and such a mapping is optimized via off-line simulation. The NDR is instantiated with two neural networks: feedforward neural network (FFNN) and recurrent neural network (RNN), which have different ways of processing traffic information in the near past. The NDR is implemented and tested within microscopic traffic simulation (S-Paramics) for a real-world network in West Glasgow, where the off-line training of the NDR amounts to a simulation-based optimization procedure aiming to reduce delay, CO2 and black carbon emissions. Extensive tests are performed to assess the NDR framework, not only in terms of its effectiveness in optimizing different traffic and environmental objectives, but also in relation to local vs. global benefits, trade-off between delay and emissions, impact of sensor locations, and different levels of network saturation. The second method, called the Advanced Reinforcement Learning (ARL), employs the potential-based reward shaping function using Q-learning and 3rd party advisor to enhance its performance over conventional reinforcement learning. The potential-based reward shaping in this thesis obtains an opinion from the 3rd party advisor when calculating reward. This technique can resolve the problem of sparse reward and slow learning speed. The ARL is tested with a range of existing reinforcement learning methods. The results clearly show that ARL outperforms the other models in almost all the scenarios. Lastly, this thesis evaluates the impact of information availability and quality on different real-time signal control methods, including the two proposed ones. This is driven by the observation that most responsive signal control models in the literature tend to make idealized assumptions on the quality and availability of data. This research shows the varying levels of performance deterioration of different signal controllers in the presence of missing data, data noise, and different data types. Such knowledge and insights are crucial for real-world implementation of these signal control methods.Open Acces

Toward Fault Adaptive Power Systems in Electric Ships

Shipboard Power Systems (SPS) play a significant role in next-generation Navy fleets. With the increasing power demand from propulsion loads, ship service loads, weaponry systems and mission systems, a stable and reliable SPS is critical to support different aspects of ship operation. It also becomes the technology-enabler to improve ship economy, efficiency, reliability, and survivability. Moreover, it is important to improve the reliability and robustness of the SPS while working under different operating conditions to ensure safe and satisfactory operation of the system. This dissertation aims to introduce novel and effective approaches to respond to different types of possible faults in the SPS. According to the type and duration, the possible faults in the Medium Voltage DC (MVDC) SPS have been divided into two main categories: transient and permanent faults. First, in order to manage permanent faults in MVDC SPS, a novel real-time reconfiguration strategy has been proposed. Onboard postault reconfiguration aims to ensure the maximum power/service delivery to the system loads following a fault. This study aims to implement an intelligent real-time reconfiguration algorithm in the RTDS platform through an optimization technique implemented inside the Real-Time Digital Simulator (RTDS). The simulation results demonstrate the effectiveness of the proposed real-time approach to reconfigure the system under different fault situations. Second, a novel approach to mitigate the effect of the unsymmetrical transient AC faults in the MVDC SPS has been proposed. In this dissertation, the application of combined Static Synchronous Compensator (STATCOM)-Super Conducting Fault Current Limiter (SFCL) to improve the stability of the MVDC SPS during transient faults has been investigated. A Fluid Genetic Algorithm (FGA) optimization algorithm is introduced to design the STATCOM\u27s controller. Moreover, a multi-objective optimization problem has been formulated to find the optimal size of SFCL\u27s impedance. In the proposed scheme, STATCOM can assist the SFCL to keep the vital load terminal voltage close to the normal state in an economic sense. The proposed technique provides an acceptable post-disturbance and postault performance to recover the system to its normal situation over the other alternatives