Tackling Occlusions & Limited Sensor Range with Set-based Safety Verification

Provable safety is one of the most critical challenges in automated driving. The behavior of numerous traffic participants in a scene cannot be predicted reliably due to complex interdependencies and the indiscriminate behavior of humans. Additionally, we face high uncertainties and only incomplete environment knowledge. Recent approaches minimize risk with probabilistic and machine learning methods - even under occlusions. These generate comfortable behavior with good traffic flow, but cannot guarantee safety of their maneuvers. Therefore, we contribute a safety verification method for trajectories under occlusions. The field-of-view of the ego vehicle and a map are used to identify critical sensing field edges, each representing a potentially hidden obstacle. The state of occluded obstacles is unknown, but can be over-approximated by intervals over all possible states. Then set-based methods are extended to provide occupancy predictions for obstacles with state intervals. The proposed method can verify the safety of given trajectories (e.g. if they ensure collision-free fail-safe maneuver options) w.r.t. arbitrary safe-state formulations. The potential for provably safe trajectory planning is shown in three evaluative scenarios

도심 교차로에서의 자율주행을 위한 주변 차량 경로 예측 및 거동 계획 알고리즘

학위논문(박사)--서울대학교 대학원 :공과대학 기계항공공학부,2020. 2. 이경수.차랑용 센싱 및 처리기술이 발달함에 따라 자동차 기술 연구가 수동 안전 기술에서 능동 안전 기술로 초점이 확장되고 있다. 최근, 주요 자동차 제작사들은 능동형 차간거리 제어, 차선 유지 보조, 그리고 긴급 자동 제동과 같은 능동 안전 기술이 이미 상업화하고 있다. 이러한 기술적 진보는 사상률 제로를 달성하기 위하여 기술 연구 분야를 능동 안전 기술을 넘어서 자율주행 시스템으로 확장시키고 있다. 특히, 도심 도로는 인도, 사각지대, 주차차량, 이륜차, 보행자 등과 같은 교통 위험 요소를 많이 갖고 있기 때문에 고속도로보다 사고 발생률과 사상률이 높으며, 이는 도심 도로에서의 자율주행은 핵심 이슈가 되고 있다. 많은 프로젝트들이 자율주행의 환경적, 인구학적, 사회적, 그리고 경제적 측면에서의 자율주행의 효과를 평가하기 위해 수행되었거나 수행 중에 있다. 예를 들어, 유럽의 AdaptIVE는 다양한 자율주행 기능을 개발하였으며, 구체적인 평가 방법론을 개발하였다. 또한, CityMobil2는 유럽 전역의 9개의 다른 환경에서 무인 지능형 차량을 성공적으로 통합하였다. 일본에서는 2014년 5월에 시작된 Automated Driving System Research Project는 자율주행 시스템과 차세대 도심 교통 수단의 개발 및 검증에 초점을 맞추었다. 기존 연구들에 대한 조사를 통해 자율주행 시스템은 교통 참여자들의 안전도를 향상시키고, 교통 혼잡을 감소시키며, 운전자 편의성을 증진시키는 것이 증명되었다. 다양한 방법론들이 인지, 거동 계획, 그리고 제어와 같은 도심 도로 자율주행차의 핵심 기술들을 개발하기 위하여 사용되었다. 하지만 많은 최신의 자율주행 연구들은 각 기술의 개발을 별개로 고려하여 진행해왔다. 결과적으로 통합적인 관점에서의 자율주행 기술 설계는 아직 충분히 고려되어 않았다. 따라서, 본 논문은 복잡한 도심 도로 환경에서 라이다, 카메라, GPS, 그리고 간단한 경로 맵에 기반한 완전 자율주행 알고리즘을 개발하는 것을 목표로 하였다. 제안된 자율주행 알고리즘은 비통제 교차로를 포함한 도심 도로 상황을 차량 거동 예측기와 모델 예측 제어 기법에 기반하여 설계되었다. 본 논문은 동적, 정적 환경 표현 및 종횡방향 거동 계획을 중점적으로 다루었다. 본 논문은 도심 도로 자율주행을 위한 거동 계획 알고리즘의 개요를 제시하였으며, 실제 교통 상황에서의 실험 결과는 제안된 알고리즘의 효과성과 운전자 거동과의 유사성을 보여주었다. 실차 실험 결과는 비통제 교차로를 포함한 도심 시나리오에서의 강건한 성능을 보여주었다.The foci of automotive researches have been expanding from passive safety systems to active safety systems with advances in sensing and processing technologies. Recently, the majority of automotive makers have already commercialized active safety systems, such as adaptive cruise control (ACC), lane keeping assistance (LKA), and autonomous emergency braking (AEB). Such advances have extended the research field beyond active safety systems to automated driving systems to achieve zero fatalities. Especially, automated driving on urban roads has become a key issue because urban roads possess numerous risk factors for traffic accidents, such as sidewalks, blind spots, on-street parking, motorcycles, and pedestrians, which cause higher accident rates and fatalities than motorways. Several projects have been conducted, and many others are still underway to evaluate the effects of automated driving in environmental, demographic, social, and economic aspects. For example, the European project AdaptIVe, develops various automated driving functions and defines specific evaluation methodologies. In addition, CityMobil2 successfully integrates driverless intelligent vehicles in nine other environments throughout Europe. In Japan, the Automated Driving System Research Project began on May 2014, which focuses on the development and verification of automated driving systems and next-generation urban transportation. From a careful review of a considerable amount of literature, automated driving systems have been proven to increase the safety of traffic users, reduce traffic congestion, and improve driver convenience. Various methodologies have been employed to develop the core technology of automated vehicles on urban roads, such as perception, motion planning, and control. However, the current state-of-the-art automated driving algorithms focus on the development of each technology separately. Consequently, designing automated driving systems from an integrated perspective is not yet sufficiently considered. Therefore, this dissertation focused on developing a fully autonomous driving algorithm in urban complex scenarios using LiDAR, vision, GPS, and a simple path map. The proposed autonomous driving algorithm covered the urban road scenarios with uncontrolled intersections based on vehicle motion prediction and model predictive control approach. Mainly, four research issues are considered: dynamic/static environment representation, and longitudinal/lateral motion planning. In the remainder of this thesis, we will provide an overview of the proposed motion planning algorithm for urban autonomous driving and the experimental results in real traffic, which showed the effectiveness and human-like behaviors of the proposed algorithm. The proposed algorithm has been tested and evaluated using both simulation and vehicle tests. The test results show the robust performance of urban scenarios, including uncontrolled intersections.Chapter 1 Introduction 1 1.1. Background and Motivation 1 1.2. Previous Researches 4 1.3. Thesis Objectives 9 1.4. Thesis Outline 10 Chapter 2 Overview of Motion Planning for Automated Driving System 11 Chapter 3 Dynamic Environment Representation with Motion Prediction 15 3.1. Moving Object Classification 17 3.2. Vehicle State based Direct Motion Prediction 20 3.2.1. Data Collection Vehicle 22 3.2.2. Target Roads 23 3.2.3. Dataset Selection 24 3.2.4. Network Architecture 25 3.2.5. Input and Output Features 33 3.2.6. Encoder and Decoder 33 3.2.7. Sequence Length 34 3.3. Road Structure based Interactive Motion Prediction 36 3.3.1. Maneuver Definition 38 3.3.2. Network Architecture 39 3.3.3. Path Following Model based State Predictor 47 3.3.4. Estimation of predictor uncertainty 50 3.3.5. Motion Parameter Estimation 53 3.3.6. Interactive Maneuver Prediction 56 3.4. Intersection Approaching Vehicle Motion Prediction 59 3.4.1. Driver Behavior Model at Intersections 59 3.4.2. Intention Inference based State Prediction 63 Chapter 4 Static Environment Representation 67 4.1. Static Obstacle Map Construction 69 4.2. Free Space Boundary Decision 74 4.3. Drivable Corridor Decision 76 Chapter 5 Longitudinal Motion Planning 81 5.1. In-Lane Target Following 82 5.2. Proactive Motion Planning for Narrow Road Driving 85 5.2.1. Motivation for Collision Preventive Velocity Planning 85 5.2.2. Desired Acceleration Decision 86 5.3. Uncontrolled Intersection 90 5.3.1. Driving Phase and Mode Definition 91 5.3.2. State Machine for Driving Mode Decision 92 5.3.3. Motion Planner for Approach Mode 95 5.3.4. Motion Planner for Risk Management Phase 98 Chapter 6 Lateral Motion Planning 105 6.1. Vehicle Model 107 6.2. Cost Function and Constraints 109 Chapter 7 Performance Evaluation 115 7.1. Motion Prediction 115 7.1.1. Prediction Accuracy Analysis of Vehicle State based Direct Motion Predictor 115 7.1.2. Prediction Accuracy and Effect Analysis of Road Structure based Interactive Motion Predictor 122 7.2. Prediction based Distance Control at Urban Roads 132 7.2.1. Driving Data Analysis of Direct Motion Predictor Application at Urban Roads 133 7.2.2. Case Study of Vehicle Test at Urban Roads 138 7.2.3. Analysis of Vehicle Test Results on Urban Roads 147 7.3. Complex Urban Roads 153 7.3.1. Case Study of Vehicle Test at Complex Urban Roads 154 7.3.2. Closed-loop Simulation based Safety Analysis 162 7.4. Uncontrolled Intersections 164 7.4.1. Simulation based Algorithm Comparison of Motion Planner 164 7.4.2. Monte-Carlo Simulation based Safety Analysis 166 7.4.3. Vehicle Tests Results in Real Traffic Conditions 172 7.4.4. Similarity Analysis between Human and Automated Vehicle 194 7.5. Multi-Lane Turn Intersections 197 7.5.1. Case Study of a Multi-Lane Left Turn Scenario 197 7.5.2. Analysis of Motion Planning Application Results 203 Chapter 8 Conclusion & Future Works 207 8.1. Conclusion 207 8.2. Future Works 209 Bibliography 210 Abstract in Korean 219Docto

교차로에서 자율주행 차량의 제한된 가시성과 불확실성을 고려한 종방향 거동계획

학위논문(박사) -- 서울대학교대학원 : 공과대학 기계공학부, 2023. 2. 이경수.This dissertation presents a novel longitudinal motion planning of autonomous vehicle at urban intersection to overcome the limited visibility due to complicated road structures and sensor specification, guaranteeing the safety from the potential collision with vehicles appearing from the occluded region. The intersection autonomous driving requires high level of safety due to congested traffics and environmental complexities. Due to complicated road structures and the detection range of perception sensors, the occluded region is generated in urban autonomous driving. The virtual target is one of the motion planning methods to react the sudden appearance of vehicles from the blind spot. The Gaussian Process Regression (GPR) is implemented to train the virtual target model to generate various future driving trajectories interacting with the motion of the ego vehicle. The GPR model provides not only the predicted trajectories of the virtual target but also the uncertainty of the future motion. Therefore, prediction results from GPR can be utilized to a position constraint for the Model Predictive Control (MPC), and the uncertainties are taken into account as a chance constraint in the MPC. In order to comprehend the surrounding environment including dynamic objects, a region of interest (ROI) is defined to determine targets of the interest. With the pre-determined driving route of the ego vehicle and the route information of the intersection, driving lanes intersecting with the ego driving lane can be determined, and the intersecting lanes are defined as ROI, reducing the computational load by eliminating targets of disinterest. Then the future motion of the selected target is predicted by a Long Short-Term Memory-Recurrent Neural Network (LSTM-RNN). Driving data for training are directly obtained with two different autonomous vehicles, providing their odometry information regardless to the limited field of view (FOV). For a widely known autonomous driving datasets such as Waymo and nuScenes, the vehicle odometry information are collected from the perceptive sensors mounted on the test vehicle. Thus, information of target that are out of the FOV of the test vehicle cant be obtained. The obtained training data are organized in the target centered coordinates for better input-domain adaptation and generalization. The mean squared error and the negative log likelihood loss functions are adapted to train and provide the uncertainty information of the target vehicle for the motion planning of the autonomous vehicle. The MPC with a chance constraint is formulated to optimize the longitudinal motion of the autonomous vehicle. The dynamic and actuator constraints are designed to provide ride comfort and safety to drivers. The position constraint with the chance constraint guarantees the safety and prevent the potential collision with target vehicles. The position constraint for the travel distance over the prediction horizon time is determined based on the clearance between the predicted trajectories of the target and ego vehicle at every prediction sample time. The performance and feasibility of the proposed algorithm are evaluated via computer simulation and test-data based simulation. The offline simulation validates the safety of the proposed algorithm, and the suggested motion planner has been implemented on an autonomous driving vehicle and tested in a real road. Through the implementation of the algorithm to an actual vehicle, the suggested algorithm is confirmed to be applicable in real life autonomous driving.본 논문은 복잡한 도로 구조와 센서 사양으로 인한 시야 제한을 극복하며 사각지대에서 등장하는 차량과의 잠재적인 충돌로부터 안전을 보장하기 위한 도심 교차로에서의 자율주행차의 새로운 종방향 거동 계획을 제시한다. 도심 자율주행은 교통체증과 환경의 복잡성으로 인해 높은 수준의 안전성이 요구됩니다. 복잡한 도로 구조와 인지 센서의 인지 범위로 인해 도심 자율주행에서는 사각지대가 발생한다. 가상 타겟은 사각지대에서 차량의 갑작스러운 출현에 대응하기 위한 거동 계획 방법 중 하나입니다. 자차량의 거동과 상호작용하는 다양한 미래 주행 궤적을 생성하는 가상 타겟 모델을 구현하기 위하여 Gaussian Process Regression (GPR) 방법을 사용합니다. GPR 모델은 가상 표적의 예측된 궤적뿐만 아니라 미래 궤적에 대한 불확실성도 제공합니다. 따라서 GPR의 예측 결과는 Model Predictive Control (MPC)에 대한 위치 제약 조건으로 활용될 수 있으며 불확실성은 MPC에서 기회 제약 조건으로 고려됩니다. 동적 객체를 포함한 주변 환경을 파악하기 위해 관심영역을 정의하여 목표 대상을 결정합니다. 미리 결정된 자차량의 주행경로와 교차로의 경로정보를 통하여 자차량의 주행차로와 교차하는 다른 차선을 판단하여 관심영역으로 정의함으로써 관심영역 밖의 차량을 제외하여 연산량을 감소시킬 수 있다. 다음으로 인지된 차량의 미래 이동 궤적은 LSTM-RNN (Long Short-Term Memory Recurrent Neural Network)에 의해 예측됩니다. 훈련을 위한 주행 데이터는 두 대의 자율주행 차량에서 직접 획득하여 제한된 시야에 관계없이 차량의 상태 정보를 제공합니다. 구글 Waymo 및 nuScenes와 같이 널리 알려진 자율주행 데이터의 경우 차량 상태 정보는 테스트 차량에 장착된 인지 센서에서 수집됩니다. 따라서 테스트 차량의 시야에서 벗어나 있는 차량 정보는 얻을 수 없습니다. 취득한 주행 데이터는 더 나은 입력 데이터 적응 및 일반화를 위해 자차가 아닌 타겟차량 중심 좌표로 구성됩니다. 손실함수로 평균 제곱 오차 및 음의 로그 우도함수를 사용하였고 음의 로그 우도함수는 자율주행 차량의 거동계획에 사용될 수 있게 타겟차량의 미래 궤적에 대한 불확실성 정보를 제공한다. 기회 제약 조건이 있는 MPC는 자율차량의 종방향 거동을 최적화하도록 구현됩니다. 동적 제약 조건 및 구동기 제약 조건은 운전자에게 승차감과 안전을 제공하도록 설계되었습니다. 기회 제약 조건은 위치 제약 조건을 강건하게 하여 안전을 보장하고 대상 차량과의 잠재적인 충돌을 방지합니다. 예측 시간동안 이동 거리에 대한 위치 제약 조건은 각 예측시간의 타겟과 자차량의 예측된 궤적 간의 거리 차이에 의해 결정된다. 제안한 알고리즘의 성능과 타당성은 컴퓨터 시뮬레이션과 테스트 데이터 기반 시뮬레이션을 통해 평가된다. 오프라인 시뮬레이션을 통해 제안한 알고리즘의 안전성을 검증하였으며 제안한 거동계획 알고리즘을 자율주행차에 구현하여 실제 도로에서 테스트하였다. 제안한 알고리즘을 실제 차량에 구현하여 실제 자율주행에 적용할 수 있음을 확인하였다.Chapter 1. Introduction 1 1.1. Research Background and Motivation of Intersection Autonomous Driving 1 1.2. Previous Researches on Intersection Autonomous Driving 9 1.2.1. Research on Trajectory Prediction and Intention Inference at Urban Intersection 10 1.2.2. Research on Intersection Motion Planning 11 1.3. Thesis Objectives 18 1.4. Thesis Outline 19 Chapter 2. Overall Architecture of Intersection Autonomous Driving System 22 2.1. Software Configuration of Intersection Autonomous Driving 22 2.2. Hardware Configuration of Autonomous Driving and Test Vehicle 24 2.3. Vehicle Test Environment for Intersection Autonomous Driving 25 Chapter 3. Virtual Target Modelling for Intersection Motion Planning 27 3.1. Limitation of Conventional Virtual Target Model for Intersection 27 3.2. Virtual Target Generation for Intersection Occlusion 31 3.3. Intersection Virtual Target Modeling 34 3.3.1. Gaussian Process Regression based Virtual Target Model at Intersection 35 3.3.2. Data Processing for Gaussian Process Regression based Virtual Target Model 38 3.3.3. Definition of Visibility Index of Virtual Target at Intersection 45 3.3.4. Long Short-Term Memory based Virtual Target Model at Intersection 51 Chapter 4. Surrounding Vehicle Motion Prediction at Intersection 54 4.1. Intersection Surrounding Vehicle Classification 54 4.2. Data-driven Vehicle State based Motion Prediction at Intersection 58 4.2.1. Network Architecture of Motion Predictor 58 4.2.2. Dataset Processing of the Network 65 Chapter 5. Intersection Longitudinal Motion Planning 68 5.1. Outlines of Longitudinal Motion Planning with Model Predictive Control 68 5.2. Stochastic Model Predictive Control of Intersection Motion Planner 69 5.2.1. Definition of System Dynamics Model 69 5.2.2. Ego Vehicle Prediction and Reference States Definition 70 5.2.3. Safety Clearance Decision for Intersection Collision Avoidance 71 5.2.4. Driving Mode Decision of Intersection Motion Planning 79 5.2.5. Formulation of Model Predictive Control with the Chance Constraint 83 Chapter 6. Performance Evaluation of Intersection Longitudinal Motion Planning 86 6.1. Performance Evaluation of Virtual Target Prediction at Intersection 86 6.1.1. GPR based Virtual Target Model Prediction Results 86 6.1.2. Intersection Autonomous Driving Computer Simulation Environment 90 6.1.2.1. Simulation Result of Effect of Virtual Target in Intersection Autonomous Driving 92 6.1.2.2. Virtual Target Simulation Result of the Right Turn Across Path Scenario in the Intersection 96 6.1.2.3. Virtual Target Simulation Result of the Straight Across Path Scenario in the Intersection 102 6.1.2.4. Virtual Target Simulation Result of the Left Turn Across Path Scenario in the Intersection 108 6.1.2.5. Virtual Target Simulation Result of Crooked T-shaped Intersection 113 6.2. Performance Evaluation of Data-driven Vehicle State based Motion Prediction at Intersection 124 6.2.1. Data-driven Motion Prediction Accuracy Analysis 124 6.2.2. Prediction Trajectory Accuracy Analysis 134 6.3. Vehicle Test for Intersection Autonomous Driving 146 6.3.1. Test Vehicle Configuration for Intersection Autonomous Driving 146 6.3.2. Software Configuration for Autonomous Vehicle Operation 147 6.3.3. Vehicle Test Environment for Intersection Autonomous Driving 148 6.3.4. Vehicle Test Result of Intersection Autonomous Driving 151 Chapter 7. Conclusion and Future Work 161 7.1. Conclusion 161 7.2. Future Work 164 Bibliography 166 Abstract in Korean 172박

Belief State Planning for Autonomous Driving: Planning with Interaction, Uncertain Prediction and Uncertain Perception

This thesis presents a behavior planning algorithm for automated driving in urban environments with an uncertain and dynamic nature. The uncertainty in the environment arises by the fact that the intentions as well as the future trajectories of the surrounding drivers cannot be measured directly but can only be estimated in a probabilistic fashion. Even the perception of objects is uncertain due to sensor noise or possible occlusions. When driving in such environments, the autonomous car must predict the behavior of the other drivers and plan safe, comfortable and legal trajectories. Planning such trajectories requires robust decision making when several high-level options are available for the autonomous car. Current planning algorithms for automated driving split the problem into different subproblems, ranging from discrete, high-level decision making to prediction and continuous trajectory planning. This separation of one problem into several subproblems, combined with rule-based decision making, leads to sub-optimal behavior. This thesis presents a global, closed-loop formulation for the motion planning problem which intertwines action selection and corresponding prediction of the other agents in one optimization problem. The global formulation allows the planning algorithm to make the decision for certain high-level options implicitly. Furthermore, the closed-loop manner of the algorithm optimizes the solution for various, future scenarios concerning the future behavior of the other agents. Formulating prediction and planning as an intertwined problem allows for modeling interaction, i.e. the future reaction of the other drivers to the behavior of the autonomous car. The problem is modeled as a partially observable Markov decision process (POMDP) with a discrete action and a continuous state and observation space. The solution to the POMDP is a policy over belief states, which contains different reactive plans for possible future scenarios. Surrounding drivers are modeled with interactive, probabilistic agent models to account for their prediction uncertainty. The field of view of the autonomous car is simulated ahead over the whole planning horizon during the optimization of the policy. Simulating the possible, corresponding, future observations allows the algorithm to select actions that actively reduce the uncertainty of the world state. Depending on the scenario, the behavior of the autonomous car is optimized in (combined lateral and) longitudinal direction. The algorithm is formulated in a generic way and solved online, which allows for applying the algorithm on various road layouts and scenarios. While such a generic problem formulation is intractable to solve exactly, this thesis demonstrates how a sufficiently good approximation to the optimal policy can be found online. The problem is solved by combining state of the art Monte Carlo tree search algorithms with near-optimal, domain specific roll-outs. The algorithm is evaluated in scenarios such as the crossing of intersections under unknown intentions of other crossing vehicles, interactive lane changes in narrow gaps and decision making at intersections with large occluded areas. It is shown that the behavior of the closed-loop planner is less conservative than comparable open-loop planners. More precisely, it is even demonstrated that the policy enables the autonomous car to drive in a similar way as an omniscient planner with full knowledge of the scene. It is also demonstrated how the autonomous car executes actions to actively gather more information about the surrounding and to reduce the uncertainty of its belief state

Belief State Planning for Autonomous Driving: Planning with Interaction, Uncertain Prediction and Uncertain Perception

This work presents a behavior planning algorithm for automated driving in urban environments with an uncertain and dynamic nature. The algorithm allows to consider the prediction uncertainty (e.g. different intentions), perception uncertainty (e.g. occlusions) as well as the uncertain interactive behavior of the other agents explicitly. Simulating the most likely future scenarios allows to find an optimal policy online that enables non-conservative planning under uncertainty

Dynamic Lambda-Field: A Counterpart of the Bayesian Occupancy Grid for Risk Assessment in Dynamic Environments

In the context of autonomous vehicles, one of the most crucial tasks is to estimate the risk of the undertaken action. While navigating in complex urban environments, the Bayesian occupancy grid is one of the most popular types of maps, where the information of occupancy is stored as the probability of collision. Although widely used, this kind of representation is not well suited for risk assessment: because of its discrete nature, the probability of collision becomes dependent on the tessellation size. Therefore, risk assessments on Bayesian occupancy grids cannot yield risks with meaningful physical units. In this article, we propose an alternative framework called Dynamic Lambda-Field that is able to assess generic physical risks in dynamic environments without being dependent on the tessellation size. Using our framework, we are able to plan safe trajectories where the risk function can be adjusted depending on the scenario. We validate our approach with quantitative experiments, showing the convergence speed of the grid and that the framework is suitable for real-world scenarios.Comment: 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other work

Development,Validation, and Integration of AI-Driven Computer Vision System and Digital-twin System for Traffic Safety Dignostics

The use of data and deep learning algorithms in transportation research have become increasingly popular in recent years. Many studies rely on real-world data. Collecting accurate traffic data is crucial for analyzing traffic safety. Still, traditional traffic data collection methods that rely on loop detectors and radar sensors are limited to collect macro-level data, and it may fail to monitor complex driver behaviors like lane changing and interactions between road users. With the development of new technologies like in-vehicle cameras, Unmanned Aerial Vehicle (UAV), and surveillance cameras, vehicle trajectory data can be collected from the recorded videos for more comprehensive and microscopic traffic safety analysis. This research presents the development, validation, and integration of three AI-driven computer vision systems for vehicle trajectory extraction and traffic safety research: 1) A.R.C.I.S, an automated framework for safety diagnosis utilizing multi-object detection and tracking algorithm for UAV videos. 2)N.M.E.D.S., A new framework with the ability to detect and predict the key points of vehicles and provide more precise vehicle occupying locations for traffic safety analysis. 3)D.V.E.D.S applied deep learning models to extract information related to drivers\u27 visual environment from the Google Street View (GSV) images. Based on the drone video collected and processed by A.R.C.I.S at various locations, CitySim: a new drone recorded vehicle trajectory dataset that aim to facilitate safety research was introduced. CitySim has vehicle interaction trajectories extracted from 1140- minutes of video recordings, which provide a large-scale naturalistic vehicle trajectory that covers a variety of locations, including basic freeway segments, freeway weaving segments, expressway segments, signalized intersections, stop-controlled intersections, and unique intersections without sign/signal control. The advantage of CitySim over other datasets is that it contains more critical safety events in quantity and severity and provides supporting scenarios for safety-oriented research. In addition, CitySim provides digital twin features, including the 3D base maps and signal timings, which enables a more comprehensive testing environment for safety research, such as autonomous vehicle safety. Based on these digital twin features provided by CitySim, we proposed a Digital Twin framework for CV and pedestrian in-the-loop simulation, which is based on Carla-Sumo Co-simulation and Cave automatic virtual environment (CAVE). The proposed framework is expected to guide the future Digital Twin research, and the architecture we build can serve as the testbed for further research and development