Integrated Freeway and Arterial Traffic Control to Improve Freeway Mobility without Compromising Arterial Traffic Conditions Using Q-Learning

Freeway and arterial transportation networks are operated individually in most cities nowadays. The lack of coordination between the two increases the severity of traffic congestion when they are heavily loaded. To address the issue, we propose an integrated traffic control strategy that coordinates freeway traffic control (variable speed limit control, lane change recommendations, ramp metering) and arterial signal timing using Q-learning. The agent is trained offline in a single-section road network first, and then implemented online in a large simulation network with real-world traffic demands. The online data are collected to further improve the agent's performance via continuous learning. We observe significant reductions in freeway travel time and number of stops and a slight increase in on-ramp queue lengths by implementing the proposed approach in scenarios with traffic congestion. Meanwhile, the queue lengths of adjacent arterial intersections are maintained at the same level. The benefits of the coordination mechanism is verified by comparing the proposed approach with an uncoordinated Q-learning algorithm and a decentralized feedback control strategy.Comment: 12 pages, 10 figures, 5 table

Control of a lane-drop bottleneck through variable speed limits

In this study, we formulate the VSL control problem for the traffic system in a zone upstream to a lane-drop bottleneck based on two traffic flow models: the Lighthill-Whitham-Richards (LWR) model, which is an infinite-dimensional partial differential equation, and the link queue model, which is a finite-dimensional ordinary differential equation. In both models, the discharging flow-rate is determined by a recently developed model of capacity drop, and the upstream in-flux is regulated by the speed limit in the VSL zone. Since the link queue model approximates the LWR model and is much simpler, we first analyze the control problem and develop effective VSL strategies based on the former. First for an open-loop control system with a constant speed limit, we prove that a constant speed limit can introduce an uncongested equilibrium state, in addition to a congested one with capacity drop, but the congested equilibrium state is always exponentially stable. Then we apply a feedback proportional-integral (PI) controller to form a closed-loop control system, in which the congested equilibrium state and, therefore, capacity drop can be removed by the I-controller. Both analytical and numerical results show that, with appropriately chosen controller parameters, the closed-loop control system is stable, effect, and robust. Finally, we show that the VSL strategies based on I- and PI-controllers are also stable, effective, and robust for the LWR model. Since the properties of the control system are transferable between the two models, we establish a dual approach for studying the control problems of nonlinear traffic flow systems. We also confirm that the VSL strategy is effective only if capacity drop occurs. The obtained method and insights can be useful for future studies on other traffic control methods and implementations of VSL strategies.Comment: 31 pages, 14 figure

2nd Symposium on Management of Future motorway and urban Traffic Systems (MFTS 2018): Booklet of abstracts: Ispra, 11-12 June 2018

The Symposium focuses on future traffic management systems, covering the subjects of traffic control, estimation, and modelling of motorway and urban networks, with particular emphasis on the presence of advanced vehicle communication and automation technologies. As connectivity and automation are being progressively introduced in our transport and mobility systems, there is indeed a growing need to understand the implications and opportunities for an enhanced traffic management as well as to identify innovative ways and tools to optimise traffic efficiency. In particular the debate on centralised versus decentralised traffic management in the presence of connected and automated vehicles has started attracting the attention of the research community. In this context, the Symposium provides a remarkable opportunity to share novel ideas and discuss future research directions.JRC.C.4-Sustainable Transpor

Deep Reinforcement Learning Approach for Lagrangian Control: Improving Freeway Bottleneck Throughput Via Variable Speed Limit

Connected vehicles (CVs) will enable new applications to improve traffic flow. The focus of this dissertation is to investigate how reinforcement learning (RL) control for the variable speed limit (VSL) through CVs can be generalized to improve traffic flow at different freeway bottlenecks. Three different bottlenecks are investigated: A sag curve, where the gradient changes from negative to positive values causes a reduction in the roadway capacity and congestion; a lane reduction, where three lanes merge to two lanes and cause congestion, and finally, an on-ramp, where increase in demand on a multilane freeway causes capacity drop. An RL algorithm is developed and implemented in a simulation environment for controlling a VSL in the upstream to manipulate the inflow of vehicles to the bottleneck on a freeway to minimize delays and increase the throughput. CVs are assumed to receive VSL messages through Infrastructure-to-Vehicle (I2V) communications technologies. Asynchronous Advantage Actor-Critic (A3C) algorithms are developed for each bottleneck to determine optimal VSL policies. Through these RL control algorithms, the speed of CVs are manipulated in the upstream of the bottleneck to avoid or minimize congestion. Various market penetration rates for CVs are considered in the simulations. It is demonstrated that the RL algorithm is able to adapt to stochastic arrivals of CVs and achieve significant improvements even at low market penetration rates of CVs, and the RL algorithm is able to find solution for all three bottlenecks. The results also show that the RL-based solutions outperform feedback-control-based solutions

Dissipation of stop-and-go waves via control of autonomous vehicles: Field experiments

Traffic waves are phenomena that emerge when the vehicular density exceeds a critical threshold. Considering the presence of increasingly automated vehicles in the traffic stream, a number of research activities have focused on the influence of automated vehicles on the bulk traffic flow. In the present article, we demonstrate experimentally that intelligent control of an autonomous vehicle is able to dampen stop-and-go waves that can arise even in the absence of geometric or lane changing triggers. Precisely, our experiments on a circular track with more than 20 vehicles show that traffic waves emerge consistently, and that they can be dampened by controlling the velocity of a single vehicle in the flow. We compare metrics for velocity, braking events, and fuel economy across experiments. These experimental findings suggest a paradigm shift in traffic management: flow control will be possible via a few mobile actuators (less than 5%) long before a majority of vehicles have autonomous capabilities

강화학습을 활용한 고속도로 가변제한속도 및 램프미터링 전략 개발

학위논문(석사) -- 서울대학교대학원 : 공과대학 건설환경공학부, 2022.2. 김동규.Recently, to resolve societal problems caused by traffic congestion, traffic control strategies have been developed to operate freeways efficiently. The representative strategies to effectively manage freeway flow are variable speed limit (VSL) control and the coordinated ramp metering (RM) strategy. This paper aims to develop a dynamic VSL and RM control algorithm to obtain efficient traffic flow on freeways using deep reinforcement learning (DRL). The traffic control strategies applying the deep deterministic policy gradient (DDPG) algorithm are tested through traffic simulation in the freeway section with multiple VSL and RM controls. The results show that implementing the strategy alleviates the congestion in the on-ramp section and shifts to the overall sections. For most cases, the VSL or RM strategy improves the overall flow rates by reducing the density and improving the average speed of the vehicles. However, VSL or RM control may not be appropriate, particularly at the high level of traffic flow. It is required to introduce the selective application of the integrated control strategies according to the level of traffic flow. It is found that the integrated strategy can be used when including the relationship between each state detector in multiple VSL sections and lanes by applying the adjacency matrix in the neural network layer. The result of this study implies the effectiveness of DRL-based VSL and the RM strategy and the importance of the spatial correlation between the state detectors.최근에는 교통혼잡으로 인한 사회적 문제를 해결하기 위해 고속도로를 효율적으로 운영하기 위한 교통통제 전략이 다양하게 개발되고 있다. 고속도로 교통류를 효과적으로 관리하기 위한 대표적인 전략으로는 차로별 제한속도를 다르게 적용하는 가변 속도 제한(VSL) 제어와 진입 램프에서 신호를 통해 차량을 통제하는 램프 미터링(RM) 전략 등이 있다. 본 연구의 목표는 심층 강화 학습(deep reinforcement learning)을 활용하여 고속도로의 효율적인 교통 흐름을 얻기 위해 동적 VSL 및 RM 제어 알고리즘을 개발하는 것이다. 고속도로의 여러 VSL과 RM 구간에서 시뮬레이션을 통해 심층 강화학습 알고리즘 중 하나인 deep deterministic policy gradient (DDPG) 알고리즘을 적용한 교통류 제어 전략을 검증한다. 실험 결과, 강화학습 기반 VSL 또는 RM 전략을 적용하는 것이 램프 진입로 구간의 혼잡을 완화하고 나아가 전체 구간의 혼잡을 줄이는 것으로 나타났다. 대부분의 경우 VSL이나 RM 전략은 본선과 진입로 구간의 밀도를 줄이고 차량의 평균 통행 속도를 증가시켜 전체 교통 흐름을 향상시킨다. VSL 또는 RM 전략들은 높은 수준의 교통류에서 적절하지 않을 수 있어 교통류 수준에 따른 전략의 선택적 도입이 필요하다. 또한 검지기간 지리적 거리와 관련한 인접 행렬을 포함하는 graph neural network layer이 여러 지점 검지기의 공간적 상관 관계를 감지하는 데 이용될 수 있다. 본 연구의 결과는 강화학습 기반 VSL과 RM 전략 도입의 필요성과 지점 검지기 간의 공간적 상관관계의 중요성을 반영하는 전략 도입의 효과를 시사한다.Chapter 1. Introduction 1 Chapter 2. Literature Review 4 Chapter 3. Methods 8 3.1. Study Area and the Collection of Data 8 3.2. Simulation Framework 11 3.3. Trip Generation and Route Choice 13 3.4. Deep Deterministic Policy Gradient (DDPG) Algorithm 14 3.5. Graph Convolution Network (GCN) Layer 17 3.6. RL Formulation 18 Chapter 4. Results 20 4.1. VSL and RM 20 4.2. Efficiency according to the flow rate 28 4.3. Effectiveness of the GCN Layer 33 Chapter 5. Conclusion 34 Bibliography 37 Abstract in Korean 44석

A Review of Traffic Signal Control.

The aim of this paper is to provide a starting point for the future research within the SERC sponsored project "Gating and Traffic Control: The Application of State Space Control Theory". It will provide an introduction to State Space Control Theory, State Space applications in transportation in general, an in-depth review of congestion control (specifically traffic signal control in congested situations), a review of theoretical works, a review of existing systems and will conclude with recommendations for the research to be undertaken within this project

Coordination and Analysis of Connected and Autonomous Vehicles in Freeway On-Ramp Merging Areas

Freeway on-ramps are typical bottlenecks in the freeway network, where the merging maneuvers of ramp vehicles impose frequent disturbances on the traffic flow and cause negative impacts on traffic safety and efficiency. The emerging Connected and Autonomous Vehicles (CAVs) hold the potential for regulating the behaviors of each individual vehicle and are expected to substantially improve the traffic operation at freeway on-ramps. The aim of this research is to explore the possibilities of optimally facilitating freeway on-ramp merging operation through the coordination of CAVs, and to discuss the impacts of CAVs on the traffic performance at on-ramp merging.In view of the existing research efforts and gaps in the field of CAV on-ramp merging operation, a novel CAV merging coordination strategy is proposed by creating large gaps on the main road and directing the ramp vehicles into the created gaps in the form of platoon. The combination of gap creation and platoon merging jointly facilitates the mainline and ramp traffic and targets at the optimal performance at the traffic flow level. The coordination consists of three components: (1) mainline vehicles proactively decelerate to create large merging gaps; (2) ramp vehicles form platoons before entering the main road; (3) the gaps created on the main road and the platoons formed on the ramp are coordinated with each other in terms of size, speed, and arrival time. The coordination is analytically formulated as an optimization problem, incorporating the macroscopic and microscopic traffic flow models. The model uses traffic state parameters as inputs and determines the optimal coordination plan adaptive to real-time traffic conditions.The impacts of CAV coordination strategies on traffic efficiency are investigated through illustrative case studies conducted on microscopic traffic simulation platforms. The results show substantial improvements in merging efficiency, throughput, and traffic flow stability. In addition, the safety benefits of CAVs in the absence of specially designed cooperation strategies are investigated to reveal the CAV’s ability to eliminate critical human factors in the ramp merging process