A Review of Sensor Technologies for Perception in Automated Driving

After more than 20 years of research, ADAS are common in modern vehicles available in the market. Automated Driving systems, still in research phase and limited in their capabilities, are starting early commercial tests in public roads. These systems rely on the information provided by on-board sensors, which allow to describe the state of the vehicle, its environment and other actors. Selection and arrangement of sensors represent a key factor in the design of the system. This survey reviews existing, novel and upcoming sensor technologies, applied to common perception tasks for ADAS and Automated Driving. They are put in context making a historical review of the most relevant demonstrations on Automated Driving, focused on their sensing setup. Finally, the article presents a snapshot of the future challenges for sensing technologies and perception, finishing with an overview of the commercial initiatives and manufacturers alliances that will show future market trends in sensors technologies for Automated Vehicles.This work has been partly supported by ECSEL Project ENABLE- S3 (with grant agreement number 692455-2), by the Spanish Government through CICYT projects (TRA2015- 63708-R and TRA2016-78886-C3-1-R)

자율 주행 시스템의 차량 안전을 위한 적응형 관심 영역 기반 효율적 환경 인지

학위논문(박사)--서울대학교 대학원 :공과대학 기계항공공학부,2020. 2. 이경수.전 세계적으로 자동차 사고로 120 만 명이 사망하기 때문에 교통 사고에 대한 기본적인 예방 조치에 대한 논의가 진행되고 있다. 통계 자료에 따르면 교통 사고의 94 %가 인적 오류에 기인한다. 도로 안전 확보의 관점에서 자율 주행 기술은 이러한 심각한 문제를 해결하는 방법으로써 관심이 높아졌으며, 연구 개발을 통해 단계적 상용화가 이루어지고 있다. 주요 자동차 제조업체는 이미 차선 유지 보조장치 (LKAS: Lane Keeping Assistant System), 적응형 순항 제어 시스템(ACC: Adaptive Cruise Control), 주차 보조 시스템 (PAS: Parking Assistance System), 자동 긴급 제동장치 (AEB: Automated Emergency Braking) 등의 첨단 운전자 보조 시스템 (ADAS)을 개발하고 상용화하였다. 또한 Audi의 Audi AI Traffic Jam Pilot, Tesla의 Autopilot, Mercedes-Benz의 Distronic Plus, 현대자동차의 Highway Driving Assist 및 BMW의 Driving Assistant Plus 와 같은 부분 자율 주행 시스템이 출시되고 있다. 이러한 부분 자율 주행 시스템은 여전히 운전자의 주의가 수반되어야 함에도 불구하고, 안전성을 크게 향상시키는 데 효과적이기 때문에 지속적으로 그 수요가 증가하고 있다. 최근 몇 년간 많은 수의 자율주행 사고가 발생하였으며, 그 빈도수가 빠르게 증가하여 사회적으로 주목받고 있다. 차량 사고는 인명 사고와 직접적으로 연관되기 때문에 자율 주행 차량의 사고들은 자율 주행 기술 신뢰성의 저하를 야기하여 사회적인 불안감을 키운다. 최근 자율 주행 관련 사고들로 인해, 자율주행 차량의 안전성의 보장이 더욱 강조되고 있다. 따라서 본 연구에서는 자율 주행 차량의 거동 제어를 고려하여 자율 주행 시스템 관점에서 차량의 안전성을 우선적으로 확보하는 접근 방식을 제안한다. 또한 자율주행 기술 개발은 단순하게 운전을 대체하는 기술이 아니라, 첨단기술의 집약 체로써 산업적으로 매우 큰 파급력을 가진다고 전망된다. 현재 자율주행 시스템은 기존 자동차 산업의 고전적인 틀에서 확장되어, 다양한 분야의 관점에서 주도적으로 개발이 진행되고 있다. 자율 주행은 다양한 기술의 복합적인 결합으로 구성되기 때문에, 현재 각기 다른 다양한 환경에서 개발이 진행 중이며, 아직 표준화되어 있지 않은 실정이다. 대부분 각 모듈 단위의 지엽적인 성능향상을 추구하는 경향이 있으며, 구성 모듈 간 관계가 고려된 전체 시스템 단위의 접근방식은 미흡한 실정이다. 세부 모듈 단위의 지엽적인 연구 개발은 통합 시, 모듈 간 상호작용으로 인한 영향으로 시스템 관점에서 적절한 성능을 확보하기 어려울 수 있다. 각 모듈의 성능만을 고려한 일방적인 방향의 연구는 한계가 명확하며, 연관된 모듈들의 특성을 고려하여 반영할 필요가 있다. 따라서 자율주행 전체 시스템의 관점에서, 차량 안전을 우선적으로 확보하고 전체 성능을 극대화하는 효과적인 접근 방식을 본 연구에서 제안하고자 한다. 본 연구는 자율 주행 시스템의 안정적이고 높은 성능을 확보하기 위해 전체 시스템 작동 측면에서 구성된 모듈 간의 상호 작용을 고려하여 효율적인 환경 인식 알고리즘을 개발하는데 중점을 둔다. 실질적인 관점에서 효과적인 정보 처리를 수행하고 차량 안전을 확보하기 위해 적응형 관심 영역 (ROI) 기반 계산 부하 관리 전략을 제안한다. 차량의 거동 특성, 도로 설계 표준, 추월 및 차선 변경과 같은 주변 차량의 주행 특성이 적응형 ROI 설계 및 주행 상황에 따른 영역 확장에 반영된다. 또한, 자율 주행 차량의 실질적인 안전을 보장하기 위해 ROI 설계에서 자율 주행 제어를 위한 거동 계획 결과가 고려된다. 보다 넓은 주변 영역에 대한 환경 정보를 확보하기 위해 라이다 데이터는 설계된 ROI별로 분류되며, 영역별 중요도에 따라 연산 과정이 분리되어 수행된다. 목표 시스템을 구성하는 모듈 별 연산 시간이 측정된 데이터 기반으로 통계적으로 분석된다. 운전자의 반응 시간, 산업 표준, 대상 하드웨어 사양 및 센서 성능을 기반으로 결정된 시스템 성능 조건을 고려하여, 안전성을 확보하기 위한 자율 주행 시스템의 적절한 샘플링 주기가 정의된다. 데이터 기반 다중 선형 회귀 분석은 인식 모듈을 구성하는 함수 별 실행 시간을 예측하기 위해 적용되며, 안정적인 실시간 성능을 보장하기 위해 적응형 ROI를 기반으로 자율 주행 안전에 필요한 데이터를 선택적으로 분류하여 연산 부하가 감축된다. 연산 부하 평가 관리에서 환경 인지 모듈과 전체 시스템의 연산 부하가 대상 환경에서의 적절성을 평가하고, 연산 부하 관리에 문제가 있을 때 자율 주행 차량의 거동을 제한하여 시스템 안정성을 유지함으로써 차량 안전성을 확보한다. 제안된 자율주행 인지 전략 및 알고리즘의 성능은 데이터 기반 시뮬레이션 및 실차 실험을 통해 검증되었다. 실험 결과를 통해 제안된 환경 인식 알고리즘은 자율 주행 시스템을 구성하는 모듈 간의 상호 작용을 고려하여 도심 도로 환경에서 자율 주행 차량의 안전성과 시스템의 안정적인 성능을 보장할 수 있음을 확인하였다.Since annually 1.2 million people die from car crashes worldwide, discussions about fundamental preventive measures for traffic accidents are taking place. According to the statistical survey, 94 percent of all traffic accidents are caused by human error. From the perspective of securing road safety, automated driving technology became interesting as a way to solve this serious problem, and its commercialization was considered through a step-by-step application through research and development. Major carmakers already have developed and commercialized advanced driver assistance systems (ADAS), such as lane keeping assistance system (LKAS), adaptive cruise control (ACC), parking assistance system (PAS), automated emergency braking (AEB), and so on. Furthermore, partially automated driving systems are being installed in vehicles and released by carmakers. Audi AI Traffic Jam Pilot (Audi), Autopilot (Tesla), Distronic Plus (Mercedes-Benz), Highway Driving Assist (Hyundai Motor Company), and Driving Assistant Plus (BMW) are typical released examples of the partially automated driving system. These released partially automated driving systems are still must be accompanied by driver attention. Nevertheless, it is proving to be effective in significantly improving safety. In recent years, several automated driving accidents have occurred, and the frequency is rapidly increasing and attracting social attention. Since vehicle accidents are directly related to human casualty, accidents of automated vehicles cause social insecurity by causing a decrease in the reliability of automated driving technology. Due to recent automated driving-related accidents, the safety of the automated vehicle has been emphasized more. Therefore, in this study, we propose an approach to secure vehicle safety in terms of the entire system in consideration of the behavior control of the automated driving vehicle. In addition, the development of automated driving is not merely a replacement technology for driving, but it is expected to have an industrial assembly as integration of high technology. Currently, automated driving systems have been extended from the conventional framework of the existing automotive industry, and are being developed in various fields. Since automated driving is composed of a complex combination of various technologies, development is currently underway in various conditions and has not been standardized yet. Most developments tend to pursue local performance improvement in each module unit, and the overall system unit approaches considering the relationship between component modules is insufficient. Local research and development at the submodule level can be challenging to achieve adequate performance from a system-level due to the effects of module interaction in terms of system integration perspective. The one-way approach that considers only the performance of each module has its limitations. To overcome this problem, it is necessary to consider the characteristics of the modules involved. This dissertation focuses on developing an efficient environment perception algorithm by considering the interaction between configured modules in terms of entire system operation to secure the stable and high performance of an automated driving system. In order to perform effective information processing and secure vehicle safety from a practical perspective, we propose an adaptive ROI based computational load management strategy. The motion characteristics of the subject vehicle, road design standards, and driving tasks of the surrounding vehicles, such as overtaking, and lane change, are reflected in the design of adaptive ROI, and the expansion of the area according to the driving task is considered. Additionally, motion planning results for automated driving are considered in the ROI design in order to guarantee the practical safety of the automated vehicle. In order to secure reasonable and appropriate environment information for the wider areas, lidar sensor data is classified by the designed ROI, and separated processing is conducted according to area importance. Based on the driving data, the calculation time of each module constituting the target system is statistically analyzed. In consideration of the system performance constraint determined by using human reaction time and industry standards, target hardware specification and the performance of sensor, the appropriate sampling time for automated driving system is defined to enhance safety. The data-based multiple linear regression is applied to predict the computation time by each function constituting perception module, and the computational load reduction is applied sequentially by selecting the data essential for automated driving safety based on adaptive ROI to secure the stable real-time execution performance of the system. In computational load assessment, it evaluates whether the computational load of the environmental perception module and entire system are appropriate and restricts the vehicle behavior when there is a problem in the computational load management to ensure vehicle safety by maintaining system stability. The performance of the proposed strategy and algorithms is evaluated through driving data-based simulation and actual vehicle tests. Test results show that the proposed environment recognition algorithm, which considers the interactions between the modules that make up the automated driving system, guarantees the safety of automated vehicle and reliable performance of system in an urban environment scenario.Chapter 1 Introduction 1 1.1. Background and Motivation 1 1.2. Previous Researches 6 1.3. Thesis Objectives 11 1.4. Thesis Outline 13 Chapter 2 Overall Architecture 14 2.1. Automated Driving Architecture 14 2.2. Test Vehicle Configuration 19 Chapter 3 Design of Adaptive ROI and Processing 21 3.1. ROI Definition 25 3.1.1. ROI Design for Normal Driving Condition 30 3.1.2. ROI Design for Lane Change 50 3.1.3. ROI Design for Intersection 56 3.2. Data Processing based on Adaptive ROI 62 3.2.1. Point Cloud Categorization by Adaptive ROI 63 3.2.2. Separated Voxelization 66 3.2.3. Separated Clustering 70 Chapter 4 Environment Perception Algorithm for Automated Driving 75 4.1. Time Delay Compensation of Environment Sensor 77 4.1.1. Algorithm Structure of Time Delay Estimation and Compensation 78 4.1.2. Time Delay Compensation Algorithm 79 4.1.3. Analysis of Processing Delay 84 4.1.4. Test Data based Open-loop Simulation 91 4.2. Environment Representation 96 4.2.1. Static Obstacle Map Construction 98 4.2.2. Lane and Road Boundary Detection 100 4.3. Multiple Object State Estimation and Tracking based on Geometric Model-Free Approach 107 4.3.1. Prediction of Geometric Model-Free Approach 109 4.3.2. Track Management 111 4.3.3. Measurement Update 112 4.3.4. Performance Evaluation via vehicle test 114 Chapter 5 Computational Load Management 117 5.1. Processing Time Analysis of Driving Data 121 5.2. Processing Time Estimation based on Multiple Linear Regression 128 5.2.1. Clustering Processing Time Estimation 129 5.2.2. Multi Object Tracking (MOT) Processing Time Estimation 138 5.2.3. Validation through Data-based Simulation 146 5.3. Computational Load Management 149 5.3.1. Sequential Processing to Computation Load Reduction 151 5.3.2. Restriction of Driving Control 154 5.3.3. Validation through Data-based Simulation 159 Chapter 6 Vehicle Tests based Performance Evaluation 163 6.1. Test-data based Simulation 164 6.2. Vehicle Tests: Urban Automated Driving 171 6.2.1. Test Configuration 171 6.2.2. Motion Planning and Vehicle Control 172 6.2.3. Vehicle Tests Results 174 Chapter 7 Conclusions and Future Works 184 Bibliography 188 Abstract in Korean 200Docto

Vision-based localization methods under GPS-denied conditions

This paper reviews vision-based localization methods in GPS-denied environments and classifies the mainstream methods into Relative Vision Localization (RVL) and Absolute Vision Localization (AVL). For RVL, we discuss the broad application of optical flow in feature extraction-based Visual Odometry (VO) solutions and introduce advanced optical flow estimation methods. For AVL, we review recent advances in Visual Simultaneous Localization and Mapping (VSLAM) techniques, from optimization-based methods to Extended Kalman Filter (EKF) based methods. We also introduce the application of offline map registration and lane vision detection schemes to achieve Absolute Visual Localization. This paper compares the performance and applications of mainstream methods for visual localization and provides suggestions for future studies.Comment: 32 pages, 15 figure

Event-Based Obstacle Detection with Commercial Lidar

Computerized obstacle detection for moving vehicles is becoming more important as vehicle manufacturers make their systems more autonomous and safe. However, obstacle detection must operate quickly in dynamic environments such as driving at highway speeds. A unique obstacle detection system using 3D changes in the environment is proposed. Furthermore, these 3D changes are shown to contain sufficient information for avoiding obstacles. To make the system easy to integrate onto a vehicle, additional processing is implemented to remove unnecessary dependencies. This system provides a method for obstacle detection that breaks away from typical systems to be more efficient

Performance of Sensor Fusion for Vehicular Applications

Sensor fusion is a key system in Advanced Driver Assistance Systems, ADAS. The perfor-mance of the sensor fusion depends on many factors such as the sensors used, the kinematicmodel used in the Extended Kalman Filter, EKF, the motion of the vehicles, the type ofroad, the density of vehicles, and the gating methods. The interactions between parametersand the extent to which individual parameters contribute to the overall accuracy of a sensorfusion system can be difficult to assess.In this study, a full-factorial experimental evaluation of a sensor fusion system basedon a real vehicle was performed. The experimental results for different driving scenariosand parameters are discussed and the factors that make the most impact are identified.The performance of sensor fusion depends on many factors such as the sensors used, thekinematic model used in the Extended Kalman Filter (EKF) motion of the vehicles, type ofroad, density of vehicles, and gating methods.This study identified that the distance between the vehicles has the largest impact on theestimation error because the vision sensor performs poorly with increased distance. In addi-tion, it was identified that the kinematic models had no significant impact on the estimation.Last but not least, the ellipsoid gates performed better than rectangular gates.In addition, we propose a new gating algorithm called an angular gate. This algorithmis based on the observation that the data for each target lies in the direction of that target.Therefore, the angle and the range can be used for setting up a two-level gating approachthat is both more intuitive and computationally faster than ellipsoid gates. The angulargates can achieve a speedup factor of up to 2.27 compared to ellipsoid gates.Furthermore, we provide time complexity analysis of angular gates, ellipsoid gates, andrectangular gates demonstrating the theoretical reasons why angular gates perform better.Last, we evaluated the performance of the Munkres algorithm using a full factorial designand identified that narrower gates can speedup the running time of the Munkres algorithmand, surprisingly, even improve the RMSE in some cases.The low target maneuvering index of vehicular systems was identified as the reason whythe kinematic models do not have an impact on the estimation. This finding supports the useof simpler and computationally inexpensive filters instead of complex Interacting MultipleModel filters. The angular gates also improve the computational efficiency of the overallsensor fusion system making them suitable for vehicular application as well as for embeddedsystems and robotics

Advances in Automated Driving Systems

Electrification, automation of vehicle control, digitalization and new mobility are the mega-trends in automotive engineering, and they are strongly connected. While many demonstrations for highly automated vehicles have been made worldwide, many challenges remain in bringing automated vehicles to the market for private and commercial use. The main challenges are as follows: reliable machine perception; accepted standards for vehicle-type approval and homologation; verification and validation of the functional safety, especially at SAE level 3+ systems; legal and ethical implications; acceptance of vehicle automation by occupants and society; interaction between automated and human-controlled vehicles in mixed traffic; human–machine interaction and usability; manipulation, misuse and cyber-security; the system costs of hard- and software and development efforts. This Special Issue was prepared in the years 2021 and 2022 and includes 15 papers with original research related to recent advances in the aforementioned challenges. The topics of this Special Issue cover: Machine perception for SAE L3+ driving automation; Trajectory planning and decision-making in complex traffic situations; X-by-Wire system components; Verification and validation of SAE L3+ systems; Misuse, manipulation and cybersecurity; Human–machine interactions, driver monitoring and driver-intention recognition; Road infrastructure measures for the introduction of SAE L3+ systems; Solutions for interactions between human- and machine-controlled vehicles in mixed traffic

Design and validation of decision and control systems in automated driving

xxvi, 148 p.En la última década ha surgido una tendencia creciente hacia la automatización de los vehículos, generando un cambio significativo en la movilidad, que afectará profundamente el modo de vida de las personas, la logística de mercancías y otros sectores dependientes del transporte. En el desarrollo de la conducción automatizada en entornos estructurados, la seguridad y el confort, como parte de las nuevas funcionalidades de la conducción, aún no se describen de forma estandarizada. Dado que los métodos de prueba utilizan cada vez más las técnicas de simulación, los desarrollos existentes deben adaptarse a este proceso. Por ejemplo, dado que las tecnologías de seguimiento de trayectorias son habilitadores esenciales, se deben aplicar verificaciones exhaustivas en aplicaciones relacionadas como el control de movimiento del vehículo y la estimación de parámetros. Además, las tecnologías en el vehículo deben ser lo suficientemente robustas para cumplir con los requisitos de seguridad, mejorando la redundancia y respaldar una operación a prueba de fallos. Considerando las premisas mencionadas, esta Tesis Doctoral tiene como objetivo el diseño y la implementación de un marco para lograr Sistemas de Conducción Automatizados (ADS) considerando aspectos cruciales, como la ejecución en tiempo real, la robustez, el rango operativo y el ajuste sencillo de parámetros. Para desarrollar las aportaciones relacionadas con este trabajo, se lleva a cabo un estudio del estado del arte actual en tecnologías de alta automatización de conducción. Luego, se propone un método de dos pasos que aborda la validación de ambos modelos de vehículos de simulación y ADS. Se introducen nuevas formulaciones predictivas basadas en modelos para mejorar la seguridad y el confort en el proceso de seguimiento de trayectorias. Por último, se evalúan escenarios de mal funcionamiento para mejorar la seguridad en entornos urbanos, proponiendo una estrategia alternativa de estimación de posicionamiento para minimizar las condiciones de riesgo