Advances in vision-based lane detection: algorithms, integration, assessment, and perspectives on ACP-based parallel vision

Lane detection is a fundamental aspect of most current advanced driver assistance systems (ADASs). A large number of existing results focus on the study of vision-based lane detection methods due to the extensive knowledge background and the low-cost of camera devices. In this paper, previous vision-based lane detection studies are reviewed in terms of three aspects, which are lane detection algorithms, integration, and evaluation methods. Next, considering the inevitable limitations that exist in the camera-based lane detection system, the system integration methodologies for constructing more robust detection systems are reviewed and analyzed. The integration methods are further divided into three levels, namely, algorithm, system, and sensor. Algorithm level combines different lane detection algorithms while system level integrates other object detection systems to comprehensively detect lane positions. Sensor level uses multi-modal sensors to build a robust lane recognition system. In view of the complexity of evaluating the detection system, and the lack of common evaluation procedure and uniform metrics in past studies, the existing evaluation methods and metrics are analyzed and classified to propose a better evaluation of the lane detection system. Next, a comparison of representative studies is performed. Finally, a discussion on the limitations of current lane detection systems and the future developing trends toward an Artificial Society, Computational experiment-based parallel lane detection framework is proposed

자율 주행 시스템의 차량 안전을 위한 적응형 관심 영역 기반 효율적 환경 인지

학위논문(박사)--서울대학교 대학원 :공과대학 기계항공공학부,2020. 2. 이경수.전 세계적으로 자동차 사고로 120 만 명이 사망하기 때문에 교통 사고에 대한 기본적인 예방 조치에 대한 논의가 진행되고 있다. 통계 자료에 따르면 교통 사고의 94 %가 인적 오류에 기인한다. 도로 안전 확보의 관점에서 자율 주행 기술은 이러한 심각한 문제를 해결하는 방법으로써 관심이 높아졌으며, 연구 개발을 통해 단계적 상용화가 이루어지고 있다. 주요 자동차 제조업체는 이미 차선 유지 보조장치 (LKAS: Lane Keeping Assistant System), 적응형 순항 제어 시스템(ACC: Adaptive Cruise Control), 주차 보조 시스템 (PAS: Parking Assistance System), 자동 긴급 제동장치 (AEB: Automated Emergency Braking) 등의 첨단 운전자 보조 시스템 (ADAS)을 개발하고 상용화하였다. 또한 Audi의 Audi AI Traffic Jam Pilot, Tesla의 Autopilot, Mercedes-Benz의 Distronic Plus, 현대자동차의 Highway Driving Assist 및 BMW의 Driving Assistant Plus 와 같은 부분 자율 주행 시스템이 출시되고 있다. 이러한 부분 자율 주행 시스템은 여전히 운전자의 주의가 수반되어야 함에도 불구하고, 안전성을 크게 향상시키는 데 효과적이기 때문에 지속적으로 그 수요가 증가하고 있다. 최근 몇 년간 많은 수의 자율주행 사고가 발생하였으며, 그 빈도수가 빠르게 증가하여 사회적으로 주목받고 있다. 차량 사고는 인명 사고와 직접적으로 연관되기 때문에 자율 주행 차량의 사고들은 자율 주행 기술 신뢰성의 저하를 야기하여 사회적인 불안감을 키운다. 최근 자율 주행 관련 사고들로 인해, 자율주행 차량의 안전성의 보장이 더욱 강조되고 있다. 따라서 본 연구에서는 자율 주행 차량의 거동 제어를 고려하여 자율 주행 시스템 관점에서 차량의 안전성을 우선적으로 확보하는 접근 방식을 제안한다. 또한 자율주행 기술 개발은 단순하게 운전을 대체하는 기술이 아니라, 첨단기술의 집약 체로써 산업적으로 매우 큰 파급력을 가진다고 전망된다. 현재 자율주행 시스템은 기존 자동차 산업의 고전적인 틀에서 확장되어, 다양한 분야의 관점에서 주도적으로 개발이 진행되고 있다. 자율 주행은 다양한 기술의 복합적인 결합으로 구성되기 때문에, 현재 각기 다른 다양한 환경에서 개발이 진행 중이며, 아직 표준화되어 있지 않은 실정이다. 대부분 각 모듈 단위의 지엽적인 성능향상을 추구하는 경향이 있으며, 구성 모듈 간 관계가 고려된 전체 시스템 단위의 접근방식은 미흡한 실정이다. 세부 모듈 단위의 지엽적인 연구 개발은 통합 시, 모듈 간 상호작용으로 인한 영향으로 시스템 관점에서 적절한 성능을 확보하기 어려울 수 있다. 각 모듈의 성능만을 고려한 일방적인 방향의 연구는 한계가 명확하며, 연관된 모듈들의 특성을 고려하여 반영할 필요가 있다. 따라서 자율주행 전체 시스템의 관점에서, 차량 안전을 우선적으로 확보하고 전체 성능을 극대화하는 효과적인 접근 방식을 본 연구에서 제안하고자 한다. 본 연구는 자율 주행 시스템의 안정적이고 높은 성능을 확보하기 위해 전체 시스템 작동 측면에서 구성된 모듈 간의 상호 작용을 고려하여 효율적인 환경 인식 알고리즘을 개발하는데 중점을 둔다. 실질적인 관점에서 효과적인 정보 처리를 수행하고 차량 안전을 확보하기 위해 적응형 관심 영역 (ROI) 기반 계산 부하 관리 전략을 제안한다. 차량의 거동 특성, 도로 설계 표준, 추월 및 차선 변경과 같은 주변 차량의 주행 특성이 적응형 ROI 설계 및 주행 상황에 따른 영역 확장에 반영된다. 또한, 자율 주행 차량의 실질적인 안전을 보장하기 위해 ROI 설계에서 자율 주행 제어를 위한 거동 계획 결과가 고려된다. 보다 넓은 주변 영역에 대한 환경 정보를 확보하기 위해 라이다 데이터는 설계된 ROI별로 분류되며, 영역별 중요도에 따라 연산 과정이 분리되어 수행된다. 목표 시스템을 구성하는 모듈 별 연산 시간이 측정된 데이터 기반으로 통계적으로 분석된다. 운전자의 반응 시간, 산업 표준, 대상 하드웨어 사양 및 센서 성능을 기반으로 결정된 시스템 성능 조건을 고려하여, 안전성을 확보하기 위한 자율 주행 시스템의 적절한 샘플링 주기가 정의된다. 데이터 기반 다중 선형 회귀 분석은 인식 모듈을 구성하는 함수 별 실행 시간을 예측하기 위해 적용되며, 안정적인 실시간 성능을 보장하기 위해 적응형 ROI를 기반으로 자율 주행 안전에 필요한 데이터를 선택적으로 분류하여 연산 부하가 감축된다. 연산 부하 평가 관리에서 환경 인지 모듈과 전체 시스템의 연산 부하가 대상 환경에서의 적절성을 평가하고, 연산 부하 관리에 문제가 있을 때 자율 주행 차량의 거동을 제한하여 시스템 안정성을 유지함으로써 차량 안전성을 확보한다. 제안된 자율주행 인지 전략 및 알고리즘의 성능은 데이터 기반 시뮬레이션 및 실차 실험을 통해 검증되었다. 실험 결과를 통해 제안된 환경 인식 알고리즘은 자율 주행 시스템을 구성하는 모듈 간의 상호 작용을 고려하여 도심 도로 환경에서 자율 주행 차량의 안전성과 시스템의 안정적인 성능을 보장할 수 있음을 확인하였다.Since annually 1.2 million people die from car crashes worldwide, discussions about fundamental preventive measures for traffic accidents are taking place. According to the statistical survey, 94 percent of all traffic accidents are caused by human error. From the perspective of securing road safety, automated driving technology became interesting as a way to solve this serious problem, and its commercialization was considered through a step-by-step application through research and development. Major carmakers already have developed and commercialized advanced driver assistance systems (ADAS), such as lane keeping assistance system (LKAS), adaptive cruise control (ACC), parking assistance system (PAS), automated emergency braking (AEB), and so on. Furthermore, partially automated driving systems are being installed in vehicles and released by carmakers. Audi AI Traffic Jam Pilot (Audi), Autopilot (Tesla), Distronic Plus (Mercedes-Benz), Highway Driving Assist (Hyundai Motor Company), and Driving Assistant Plus (BMW) are typical released examples of the partially automated driving system. These released partially automated driving systems are still must be accompanied by driver attention. Nevertheless, it is proving to be effective in significantly improving safety. In recent years, several automated driving accidents have occurred, and the frequency is rapidly increasing and attracting social attention. Since vehicle accidents are directly related to human casualty, accidents of automated vehicles cause social insecurity by causing a decrease in the reliability of automated driving technology. Due to recent automated driving-related accidents, the safety of the automated vehicle has been emphasized more. Therefore, in this study, we propose an approach to secure vehicle safety in terms of the entire system in consideration of the behavior control of the automated driving vehicle. In addition, the development of automated driving is not merely a replacement technology for driving, but it is expected to have an industrial assembly as integration of high technology. Currently, automated driving systems have been extended from the conventional framework of the existing automotive industry, and are being developed in various fields. Since automated driving is composed of a complex combination of various technologies, development is currently underway in various conditions and has not been standardized yet. Most developments tend to pursue local performance improvement in each module unit, and the overall system unit approaches considering the relationship between component modules is insufficient. Local research and development at the submodule level can be challenging to achieve adequate performance from a system-level due to the effects of module interaction in terms of system integration perspective. The one-way approach that considers only the performance of each module has its limitations. To overcome this problem, it is necessary to consider the characteristics of the modules involved. This dissertation focuses on developing an efficient environment perception algorithm by considering the interaction between configured modules in terms of entire system operation to secure the stable and high performance of an automated driving system. In order to perform effective information processing and secure vehicle safety from a practical perspective, we propose an adaptive ROI based computational load management strategy. The motion characteristics of the subject vehicle, road design standards, and driving tasks of the surrounding vehicles, such as overtaking, and lane change, are reflected in the design of adaptive ROI, and the expansion of the area according to the driving task is considered. Additionally, motion planning results for automated driving are considered in the ROI design in order to guarantee the practical safety of the automated vehicle. In order to secure reasonable and appropriate environment information for the wider areas, lidar sensor data is classified by the designed ROI, and separated processing is conducted according to area importance. Based on the driving data, the calculation time of each module constituting the target system is statistically analyzed. In consideration of the system performance constraint determined by using human reaction time and industry standards, target hardware specification and the performance of sensor, the appropriate sampling time for automated driving system is defined to enhance safety. The data-based multiple linear regression is applied to predict the computation time by each function constituting perception module, and the computational load reduction is applied sequentially by selecting the data essential for automated driving safety based on adaptive ROI to secure the stable real-time execution performance of the system. In computational load assessment, it evaluates whether the computational load of the environmental perception module and entire system are appropriate and restricts the vehicle behavior when there is a problem in the computational load management to ensure vehicle safety by maintaining system stability. The performance of the proposed strategy and algorithms is evaluated through driving data-based simulation and actual vehicle tests. Test results show that the proposed environment recognition algorithm, which considers the interactions between the modules that make up the automated driving system, guarantees the safety of automated vehicle and reliable performance of system in an urban environment scenario.Chapter 1 Introduction 1 1.1. Background and Motivation 1 1.2. Previous Researches 6 1.3. Thesis Objectives 11 1.4. Thesis Outline 13 Chapter 2 Overall Architecture 14 2.1. Automated Driving Architecture 14 2.2. Test Vehicle Configuration 19 Chapter 3 Design of Adaptive ROI and Processing 21 3.1. ROI Definition 25 3.1.1. ROI Design for Normal Driving Condition 30 3.1.2. ROI Design for Lane Change 50 3.1.3. ROI Design for Intersection 56 3.2. Data Processing based on Adaptive ROI 62 3.2.1. Point Cloud Categorization by Adaptive ROI 63 3.2.2. Separated Voxelization 66 3.2.3. Separated Clustering 70 Chapter 4 Environment Perception Algorithm for Automated Driving 75 4.1. Time Delay Compensation of Environment Sensor 77 4.1.1. Algorithm Structure of Time Delay Estimation and Compensation 78 4.1.2. Time Delay Compensation Algorithm 79 4.1.3. Analysis of Processing Delay 84 4.1.4. Test Data based Open-loop Simulation 91 4.2. Environment Representation 96 4.2.1. Static Obstacle Map Construction 98 4.2.2. Lane and Road Boundary Detection 100 4.3. Multiple Object State Estimation and Tracking based on Geometric Model-Free Approach 107 4.3.1. Prediction of Geometric Model-Free Approach 109 4.3.2. Track Management 111 4.3.3. Measurement Update 112 4.3.4. Performance Evaluation via vehicle test 114 Chapter 5 Computational Load Management 117 5.1. Processing Time Analysis of Driving Data 121 5.2. Processing Time Estimation based on Multiple Linear Regression 128 5.2.1. Clustering Processing Time Estimation 129 5.2.2. Multi Object Tracking (MOT) Processing Time Estimation 138 5.2.3. Validation through Data-based Simulation 146 5.3. Computational Load Management 149 5.3.1. Sequential Processing to Computation Load Reduction 151 5.3.2. Restriction of Driving Control 154 5.3.3. Validation through Data-based Simulation 159 Chapter 6 Vehicle Tests based Performance Evaluation 163 6.1. Test-data based Simulation 164 6.2. Vehicle Tests: Urban Automated Driving 171 6.2.1. Test Configuration 171 6.2.2. Motion Planning and Vehicle Control 172 6.2.3. Vehicle Tests Results 174 Chapter 7 Conclusions and Future Works 184 Bibliography 188 Abstract in Korean 200Docto

A Review of Sensor Technologies for Perception in Automated Driving

After more than 20 years of research, ADAS are common in modern vehicles available in the market. Automated Driving systems, still in research phase and limited in their capabilities, are starting early commercial tests in public roads. These systems rely on the information provided by on-board sensors, which allow to describe the state of the vehicle, its environment and other actors. Selection and arrangement of sensors represent a key factor in the design of the system. This survey reviews existing, novel and upcoming sensor technologies, applied to common perception tasks for ADAS and Automated Driving. They are put in context making a historical review of the most relevant demonstrations on Automated Driving, focused on their sensing setup. Finally, the article presents a snapshot of the future challenges for sensing technologies and perception, finishing with an overview of the commercial initiatives and manufacturers alliances that will show future market trends in sensors technologies for Automated Vehicles.This work has been partly supported by ECSEL Project ENABLE- S3 (with grant agreement number 692455-2), by the Spanish Government through CICYT projects (TRA2015- 63708-R and TRA2016-78886-C3-1-R)

Simultaneous Localization and Mapping (SLAM) for Autonomous Driving: Concept and Analysis

The Simultaneous Localization and Mapping (SLAM) technique has achieved astonishing progress over the last few decades and has generated considerable interest in the autonomous driving community. With its conceptual roots in navigation and mapping, SLAM outperforms some traditional positioning and localization techniques since it can support more reliable and robust localization, planning, and controlling to meet some key criteria for autonomous driving. In this study the authors first give an overview of the different SLAM implementation approaches and then discuss the applications of SLAM for autonomous driving with respect to different driving scenarios, vehicle system components and the characteristics of the SLAM approaches. The authors then discuss some challenging issues and current solutions when applying SLAM for autonomous driving. Some quantitative quality analysis means to evaluate the characteristics and performance of SLAM systems and to monitor the risk in SLAM estimation are reviewed. In addition, this study describes a real-world road test to demonstrate a multi-sensor-based modernized SLAM procedure for autonomous driving. The numerical results show that a high-precision 3D point cloud map can be generated by the SLAM procedure with the integration of Lidar and GNSS/INS. Online four–five cm accuracy localization solution can be achieved based on this pre-generated map and online Lidar scan matching with a tightly fused inertial system

A dynamic two-dimensional (D2D) weight-based map-matching algorithm

Existing map-Matching (MM) algorithms primarily localize positioning fixes along the centerline of a road and have largely ignored road width as an input. Consequently, vehicle lane-level localization, which is essential for stringent Intelligent Transport System (ITS) applications, seems difficult to accomplish, especially with the positioning data from low-cost GPS sensors. This paper aims to address this limitation by developing a new dynamic two-dimensional (D2D) weight-based MM algorithm incorporating dynamic weight coefficients and road width. To enable vehicle lane-level localization, a road segment is virtually expressed as a matrix of homogeneous grids with reference to a road centerline. These grids are then used to map-match positioning fixes as opposed to matching on a road centerline as carried out in traditional MM algorithms. In this developed algorithm, vehicle location identification on a road segment is based on the total weight score which is a function of four different weights: (i) proximity, (ii) kinematic, (iii) turn-intent prediction, and (iv) connectivity. Different parameters representing network complexity and positioning quality are used to assign the relative importance to different weight scores by employing an adaptive regression method. To demonstrate the transferability of the developed algorithm, it was tested by using 5,830 GPS positioning points collected in Nottingham, UK and 7,414 GPS positioning points collected in Mumbai and Pune, India. The developed algorithm, using stand-alone GPS position fixes, identifies the correct links 96.1% (for the Nottingham data) and 98.4% (for the Mumbai-Pune data) of the time. In terms of the correct lane identification, the algorithm was found to provide the accurate matching for 84% (Nottingham) and 79% (Mumbai-Pune) of the fixes obtained by stand-alone GPS. Using the same methodology adopted in this study, the accuracy of the lane identification could further be enhanced if the localization data from additional sensors (e.g. gyroscope) are utilized. ITS industry and vehicle manufacturers can implement this D2D map-matching algorithm for liability critical and in-vehicle information systems and services such as advanced driver assistant systems (ADAS)

Simulations for Training Machine Learning Models for Autonomous Vehicles

Masinõppe mudelite treenimine autonoomsete sõidukite jaoks nõuab palju andmeid, mille käsitsi märgendamine on aeganõudev. Simulatsioonid aitavad seda protsessi automatiseerida. Käesolev töö koostab ülevaate 12-st internetiotsingu abil leitud simulatsioonist ja analüüsib neid lähtuvalt nende sobivusest maastikul liikuvatele sõidukitele (säilitades võimaluse liikuda ka linnakeskkonnas).Training machine learning models for autonomous vehicles requires a lot of data which is time consuming and tedious to label manually. Simulated virtual environments help to automate this process. In this work these virtual environments are called simulations. The goal of this thesis is to survey the most suitable simulations for off-road vehicles (while not discarding the urban option). Only the simulations which provide labeled output data, are included in this work. The chosen 12 simulations are surveyed based on the information found online. The simulations are then analyzed based on the predefined features and categorized according to their suitability for training machine learning models for off-road vehicles. The results are shown in a table for comparison. The main purpose of this work is to map the seemingly large landscape of simulations and give a compact picture of the situation

Driver lane change intention inference using machine learning methods.

Lane changing manoeuvre on highway is a highly interactive task for human drivers. The intelligent vehicles and the advanced driver assistance systems (ADAS) need to have proper awareness of the traffic context as well as the driver. The ADAS also need to understand the driver potential intent correctly since it shares the control authority with the human driver. This study provides a research on the driver intention inference, particular focus on the lane change manoeuvre on highways. This report is organised in a paper basis, where each chapter corresponding to a publication, which is submitted or to be submitted. Part Ⅰ introduce the motivation and general methodology framework for this thesis. Part Ⅱ includes the literature survey and the state-of-art of driver intention inference. Part Ⅲ contains the techniques for traffic context perception that focus on the lane detection. A literature review on lane detection techniques and its integration with parallel driving framework is proposed. Next, a novel integrated lane detection system is designed. Part Ⅳ contains two parts, which provides the driver behaviour monitoring system for normal driving and secondary tasks detection. The first part is based on the conventional feature selection methods while the second part introduces an end-to-end deep learning framework. The design and analysis of driver lane change intention inference system for the lane change manoeuvre is proposed in Part Ⅴ. Finally, discussions and conclusions are made in Part Ⅵ. A major contribution of this project is to propose novel algorithms which accurately model the driver intention inference process. Lane change intention will be recognised based on machine learning (ML) methods due to its good reasoning and generalizing characteristics. Sensors in the vehicle are used to capture context traffic information, vehicle dynamics, and driver behaviours information. Machine learning and image processing are the techniques to recognise human driver behaviour.PhD in Transpor

Evaluation and testing system for automotive LiDAR sensors

The world is facing a great technological transformation towards fully autonomous vehicles, where optimists predict that by 2030 autonomous vehicles will be sufficiently reliable, affordable, and common to displace most human driving. To cope with these trends, reliable perception systems must enable vehicles to hear and see all their surroundings, with light detection and ranging (LiDAR) sensors being a key instrument for recreating a 3D visualization of the world in real time. However, perception systems must rely on accurate measurements of the environment. Thus, these intelligent sensors must be calibrated and benchmarked before being placed on the market or assembled in a car. This article presents an Evaluation and Testing Platform for Automotive LiDAR sensors, with the main goal of testing both commercially available sensors and new sensor prototypes currently under development in Bosch Car Multimedia Portugal. The testing system can benchmark any LiDAR sensor under different conditions, recreating the expected driving environment in which such devices normally operate. To characterize and validate the sensor under test, the platform evaluates several parameters, such as the field of view (FoV), angular resolution, sensor’s range, etc., based only on the point cloud output. This project is the result of a partnership between the University of Minho and Bosch Car Multimedia Portugal.This work was supported by the European Structural and Investment Funds in the FEDER component through the Operational Competitiveness and Internationalization Programme (COM-PETE 2020), Project nº 037902, Funding Reference POCI-01-0247-FEDER-037902

Exploring the challenges and opportunities of image processing and sensor fusion in autonomous vehicles: A comprehensive review

Autonomous vehicles are at the forefront of future transportation solutions, but their success hinges on reliable perception. This review paper surveys image processing and sensor fusion techniques vital for ensuring vehicle safety and efficiency. The paper focuses on object detection, recognition, tracking, and scene comprehension via computer vision and machine learning methodologies. In addition, the paper explores challenges within the field, such as robustness in adverse weather conditions, the demand for real-time processing, and the integration of complex sensor data. Furthermore, we examine localization techniques specific to autonomous vehicles. The results show that while substantial progress has been made in each subfield, there are persistent limitations. These include a shortage of comprehensive large-scale testing, the absence of diverse and robust datasets, and occasional inaccuracies in certain studies. These issues impede the seamless deployment of this technology in real-world scenarios. This comprehensive literature review contributes to a deeper understanding of the current state and future directions of image processing and sensor fusion in autonomous vehicles, aiding researchers and practitioners in advancing the development of reliable autonomous driving systems