Driving in Dense Traffic with Model-Free Reinforcement Learning

Traditional planning and control methods could fail to find a feasible trajectory for an autonomous vehicle to execute amongst dense traffic on roads. This is because the obstacle-free volume in spacetime is very small in these scenarios for the vehicle to drive through. However, that does not mean the task is infeasible since human drivers are known to be able to drive amongst dense traffic by leveraging the cooperativeness of other drivers to open a gap. The traditional methods fail to take into account the fact that the actions taken by an agent affect the behaviour of other vehicles on the road. In this work, we rely on the ability of deep reinforcement learning to implicitly model such interactions and learn a continuous control policy over the action space of an autonomous vehicle. The application we consider requires our agent to negotiate and open a gap in the road in order to successfully merge or change lanes. Our policy learns to repeatedly probe into the target road lane while trying to find a safe spot to move in to. We compare against two model-predictive control-based algorithms and show that our policy outperforms them in simulation.Comment: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2020. Updated Github repository link

협소하고 복잡한 환경에서 자율 주행을 위한 샘플링 기반 모션 계획 방법

학위논문 (박사)-- 서울대학교 융합과학기술대학원 융합과학부, 2017. 8. 박재흥.Autonomous vehicles are being actively developed for fully autonomous driving without driver intervention. Motion planning is one of the most key technologies in terms of driving safety and efficiency. In particular, the motion planning in constrained narrow space such as a parking lot is very challenging because it requires many changes in forward and backward directions and adjustments of position and orientation of the vehicle. In this thesis, a sampling-based motion planning algorithm is proposed based on Rapidly-exploring Random Trees (RRT, RRT*) by specifying desired orientation during the tree expansion and the rewiring step. The contribution is as follows. First, efficient sampling method is proposed for narrow-cluttered area. In this area, the probability of obtaining a sample to pass through the area due to the obstacle area is relatively low than an open area. It may also fail to extend the path if sampled position is difficult to extend from near nodes. To solve this problem, a constraint model on the tangential direction of the random sample is proposed. Second, we propose an extension method based on tangential direction constraint. In the process of expanding the tree to random samples, a large number of nodes in narrow-cluttered regions cannot pass the collision test. This increases unnecessary iteration numbers and increases memory usage. To solve this problem, we propose a node extension method based on gradient descent. The proposed algorithm has been tested in various situations and its results demonstrated much faster target path search and convergence to the optimal path than the existing nonholonomic RRT*.I. Introduction 1 1.1 Autonomous Vehicles 1 1.2 Planning System of Autonomous Vehicles 2 1.3 Contribution of Thesis 4 II. Related Works 6 2.1 Motion Planning for Aunomous Vehicles 6 2.2 Sampling-based Motion Planning Algorithms 9 III. Sampling-based Kinodynamic Motion Planning Algorithm for Narrow Cluttered Environments 14 3.1 Overview 14 3.2 Preliminary Definition 15 3.2.1 Problem Statements 15 3.2.2 Autonomous Vehicle Model 16 3.3 Kinodynamic RRT and Limitations 16 3.3.1 Overview of DO-RRT Algorithm 20 3.4 Magnetic-like Field based Desired Orientation Model 20 3.4.1 Magnet-like Field Model 22 3.4.2 Pfaffian Constraints 24 3.4.3 DO(Desired Orientation) Model 26 3.5 Sampling Fuction of DO-RRT 28 3.6 Extend Function of DO-RRT 30 3.7 Experimental Results 31 3.7.1 Experimental Condition 31 3.7.2 Simulation Test Results 32 3.7.3 Vehicle Test Results 34 IV. Sampling-based Geometric Motion Planning Algorithm for Narrow Cluttered Environments 38 4.1 Overview 38 4.2 Backgrounds 39 4.2.1 Algorithm Description and Limitations 39 4.2.2 Overview of Proposed Algorithm 42 4.3 Desired Orientation based Random Sampling Method 44 4.4 Desired Orientation based Extend Method 47 4.5 Analysis 49 4.5.1 Probabilistic Completeness 49 4.5.2 Asymptotic Optimality 51 4.5.3 Configuration Space Analysis 53 4.6 Experimental Results 58 4.6.1 Experimental Condition 59 4.6.2 The Result of Desired Orientation-RRT Planner 59 4.6.3 Result of Desired Orientation-RRT 65 V. Experimental Platform Development 71 5.1 Hardware Architecture 71 5.2 Software Architecture 73 5.3 Valet Parking System 74 5.3.1 Perception System 75 5.3.2 Localization System 76 5.3.3 Planning System 77 5.3.4 Control System 79 5.4 Experimental Validation 81 VI. Conclusion 85 Bibliography 86Docto

Motion Planning and Safety for Autonomous Driving

This thesis discusses two different problems in motion planning for autonomous driving. The first is the problem of optimizing a lattice planner control set for any particular autonomous driving task, with the goal of reducing planning time for that task. The driving task is encoded in the form of a dataset of trajectories executed while performing said task. In addition to improving planning time, the optimized control set should capture the driving style of the dataset. In this sense, the control set is learned from the data and is tailored to a particular task. To determine the value of control actions to add to the control set, a modified version of the Fréchet distance is used to score how useful control actions are for generating paths similar to those in the dataset. This method is then compared to the state of the art lattice planner control set optimization technique in terms of planning runtime for the learned task. The second problem is the task of extending the Responsibility-Sensitive Safety (RSS) framework by introducing swerve manoeuvres in addition to the nominal braking manoeu- vres present in the framework. This includes comparing the clearance distances required by a swerve to the braking distances in the original framework. This comparison shows that swerve manoeuvres require less distance gap in order to reach safety from a braking agent in front of the autonomous vehicle at higher speeds. For more realistic swerve manoeuvres, the kinematic bicycle model is used rather than the 2-D double integrator model consid- ered in RSS. An upper bound is then computed on the required clearance distance for a swerve manoeuvre that satisfies bicycle kinematics. A longitudinal safe following distance is then derived that is provably safe, and is shown to be lower than the following distance required by RSS at higher speeds. The use of the kinematic bicycle model is then validated by computing swerve manoeuvres with a dynamic single-track car model and Pacejka tire model, and comparing the single-track swerves to the bicycle swerves

Autonomous Driving at Intersections: A Critical-Turning-Point Approach for Planning and Decision Making

Left-turning at unsignalized intersection is one of the most challenging tasks for urban automated driving, due to the various shapes of different intersections, and rapidly changing nature of the driving scenarios. Many algorithms including rule-based approach, graph-based approach, optimization-based approach, etc. have been developed to overcome the problems. However, most algorithms implemented were difficult to guarantee the safety at intersection scenarios in real time due to the large uncertainty of the intersections. Other algorithms that aim to always keep a safe distance in all cases often become overly conservative, which might also be dangerous and inefficient. This thesis addresses this challenge by proposing a generalized critical turning point (CTP) based hierarchical decision making and planning method, which enables safe and efficient planning and decision making of autonomous vehicles. The high-level candidate-paths planner takes the road map information and generates CTPs using a parameterized CTP extraction model which is proposed and verified by naturalistic driving data. CTP is a novel concept and the corresponding CTP model is used to generate behavior-oriented paths that adapt to various intersections. These modifications help to assure the high searching efficiency of the planning process, and in the meanwhile, enable human-like driving behavior of the autonomous vehicle. The low-level planner formulates the decision-making task to a POMDP problem which considers the uncertainties of the agent in the intersections. The POMDP problem is then solved with a Monte Carlo tree search (MCTS)-based framework to select proper candidate paths and decide the actions on that path. The proposed framework that uses CTPs is tested in several critical scenarios and has out-performed the methods of not using CTPs. The framework has shown the ability to adapt to various shapes of intersections with different numbers of road lanes and different width of median strips, and finishes the left turns while keeping proper safety distances. The uses of the CTP concept which is proposed through human-driving left-turning behaviors, enables the framework to perform human-like behaviors that is easier to be speculated by the other agents of the intersection, which improves the safety of the ego vehicle too. The framework is also capable of personalized modification of the desired real-time performance and the corresponding stability. The use of the POMDP model which considers the unknown intentions of the surrounding vehicles has also enabled the framework to provide commute-efficient two-dimensional planning and decision-making. In all, the proposed framework enables the ego vehicle to perform less conservative and human-like actions while considering the potential of crashes in real-time, which not only improves the commute-efficiency, but also enables urban driving autonomous vehicles to naturally integrate into scenarios with human-driven vehicles in a friendly manne

Belief Space-Guided Navigation for Robots and Autonomous Vehicles

Navigating through the environment is a fundamental capability for mobile robots, which is still very challenging today. Most robotic applications these days, such as mining, disaster response, and agriculture, require the robots to move and perform tasks in a variety of environments which are stochastic and sometimes even unpredictable. A robot often cannot directly observe its current state but instead estimates a distribution over the set of possible states based on sensor measurements that are both noisy and partial. The actual robot position differs from its prediction after applying a motion command, due to actuation noise. Classic algorithms for navigation should adapt themselves to where the behavior of the environment is stochastic, and the execution of the motions has great uncertainty. To solve such challenging problems, we propose to guide the robot's navigation in the belief space. Belief space-guided navigation differs fundamentally from planning without uncertainty where the state of the robot is always assumed to be known precisely. The robot senses its environment, estimates its current state due to perception uncertainty, and decides whether a new (or priori) action is appropriate. Based on that determination, it actuates its sensors to move with motion uncertainty in the environment. This inspires us to connect robot perception and motion planning, and reason about the uncertainty to improve the quality of plan so that the robot can follow a collision-free, feasible kinodynamic, and task-optimal trajectory. In this dissertation, we explore the belief space-guided robotic navigation problems, which include belief space-based scene understanding for autonomous vehicles and introduce belief space guided robotic planning. We first investigate how belief space can facilitate scene understanding under the context of lane marking quality assessment in the application of autonomous driving. We propose a new problem by measuring the quality of roads and ensuring they are ready for autonomous driving. We focus on developing three quality metrics for lane markings (LMs), correctness metric, shape metric, and visibility metric, and algorithms to assess LM qualities to facilitate scene understanding. As another example of using belief space for better scene understanding, we utilize crowdsourced images from multiple vehicles to help verify LMs for high-definition (HD) map maintenance. An LM is consistent if belief functions from the map and the image satisfy statistical hypothesis testing. We further extend the Bayesian belief model into a sequential belief update using crowdsourced images. LMs with a higher probability of existence are kept in the HD map whereas those with a lower probability of existence are removed from the HD map. Belief space can also help us to tightly connect perception and motion planning. As an example, we develop a motion planning strategy for autonomous vehicles. Named as virtual lane boundary approach, this framework considers obstacle avoidance, trajectory smoothness (to satisfy vehicle kinodynamic constraints), trajectory continuity (to avoid sudden movements), global positioning system (GPS) following quality (to execute the global plan), and lane following or partial direction following (to meet human expectation). Consequently, vehicle motion is more human-compatible than existing approaches. As another example of how belief space can help guide robots for different tasks, we propose to use it for the probabilistic boundary coverage of unknown target fields (UTFs). We employ Gaussian processes as a local belief function to approximate a field boundary distribution in an ellipse-shaped local region. The local belief function allows us to predict UTF boundary trends and establish an adjacent ellipse for further exploration. The process is governed by a depth-first search process until UTF is approximately enclosed by connected ellipses when the boundary coverage process ends. We formally prove that our boundary coverage process guarantees the enclosure above a given coverage ratio with a preset probability threshold