O₂SAT: Object-Oriented-Segmentation-Guided Spatial-Attention Network for 3D Object Detection in Autonomous Vehicles

Autonomous vehicles (AVs) strive to adapt to the specific characteristics of sustainable urban environments. Accurate 3D object detection with LiDAR is paramount for autonomous driving. However, existing research predominantly relies on the 3D object-based assumption, which overlooks the complexity of real-world road environments. Consequently, current methods experience performance degradation when targeting only local features and overlooking the intersection of objects and road features, especially in uneven road conditions. This study proposes a 3D Object-Oriented-Segmentation Spatial-Attention (O2SAT) approach to distinguish object points from road points and enhance the keypoint feature learning by a channel-wise spatial attention mechanism. O2SAT consists of three modules: Object-Oriented Segmentation (OOS), Spatial-Attention Feature Reweighting (SFR), and Road-Aware 3D Detection Head (R3D). OOS distinguishes object and road points and performs object-aware downsampling to augment data by learning to identify the hidden connection between landscape and object; SFR performs weight augmentation to learn crucial neighboring relationships and dynamically adjust feature weights through spatial attention mechanisms, which enhances the long-range interactions and contextual feature discrimination for noise suppression, improving overall detection performance; and R3D utilizes refined object segmentation and optimized feature representations. Our system forecasts prediction confidence into existing point-backbones. Our method’s effectiveness and robustness across diverse datasets (KITTI) has been demonstrated through vast experiments. The proposed modules seamlessly integrate into existing point-based frameworks, following a plug-and-play approach

Deep Learning for Safe Autonomous Driving: Current Challenges and Future Directions

[EN] Advances in information and signal processing technologies have a significant impact on autonomous driving (AD), improving driving safety while minimizing the efforts of human drivers with the help of advanced artificial intelligence (AI) techniques. Recently, deep learning (DL) approaches have solved several real-world problems of complex nature. However, their strengths in terms of control processes for AD have not been deeply investigated and highlighted yet. This survey highlights the power of DL architectures in terms of reliability and efficient real-time performance and overviews state-of-the-art strategies for safe AD, with their major achievements and limitations. Furthermore, it covers major embodiments of DL along the AD pipeline including measurement, analysis, and execution, with a focus on road, lane, vehicle, pedestrian, drowsiness detection, collision avoidance, and traffic sign detection through sensing and vision-based DL methods. In addition, we discuss on the performance of several reviewed methods by using different evaluation metrics, with critics on their pros and cons. Finally, this survey highlights the current issues of safe DL-based AD with a prospect of recommendations for future research, rounding up a reference material for newcomers and researchers willing to join this vibrant area of Intelligent Transportation Systems.This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) Grant funded by the Korea Government (MSIT) (2019-0-00136, Development of AI-Convergence Technologies for Smart City Industry Productivity Innovation); The work of Javier Del Ser was supported by the Basque Government through the EMAITEK and ELKARTEK Programs, as well as by the Department of Education of this institution (Consolidated Research Group MATHMODE, IT1294-19); VHCA received support from the Brazilian National Council for Research and Development (CNPq, Grant #304315/2017-6 and #430274/2018-1).Muhammad, K.; Ullah, A.; Lloret, J.; Del Ser, J.; De Albuquerque, VHC. (2021). Deep Learning for Safe Autonomous Driving: Current Challenges and Future Directions. IEEE Transactions on Intelligent Transportation Systems. 22(7):4316-4336. https://doi.org/10.1109/TITS.2020.30322274316433622

Detección y tracking de objetos utilizando visión y láser 3D para vehículos inteligentes

La detección y el seguimiento de los objetos presentes en un entorno son dos factores imprescindibles para el guiado de vehículos autónomos. Con este trabajo se pretenden estudiar y aplicar diversas técnicas para detectar los elementos que rodean a un vehículo con el objetivo de poder determinar sus características y posicionarlos en el espacio. Tras la detección se realizará un seguimiento de cada uno de los objetos detectados que permitirá identificarlos en una secuencia de tiempo y predecir su comportamiento. Se utilizará la información de las imágenes provenientes de las cámaras del vehículo y de las nubes de puntos que nos entregará el LIDAR situado en el mismo.Object detection and tracking are essential for autonomous and self-driving vehicles. The objective of this paper is to study and apply different techniques and approaches intended to detect the elements surrounding a vehicle in order to determine their features and location in the space. Once the objects have been detected they will be tracked, which will allow for their identification in a sequence of time and to predict their behavior. The main data sources that are going to be used are the images taken by the cameras placed in the vehicle and the point clouds provided by the vehicle LIDAR.Grado en Ingeniería Telemátic

자율 주행 시스템의 차량 안전을 위한 적응형 관심 영역 기반 효율적 환경 인지

학위논문(박사)--서울대학교 대학원 :공과대학 기계항공공학부,2020. 2. 이경수.전 세계적으로 자동차 사고로 120 만 명이 사망하기 때문에 교통 사고에 대한 기본적인 예방 조치에 대한 논의가 진행되고 있다. 통계 자료에 따르면 교통 사고의 94 %가 인적 오류에 기인한다. 도로 안전 확보의 관점에서 자율 주행 기술은 이러한 심각한 문제를 해결하는 방법으로써 관심이 높아졌으며, 연구 개발을 통해 단계적 상용화가 이루어지고 있다. 주요 자동차 제조업체는 이미 차선 유지 보조장치 (LKAS: Lane Keeping Assistant System), 적응형 순항 제어 시스템(ACC: Adaptive Cruise Control), 주차 보조 시스템 (PAS: Parking Assistance System), 자동 긴급 제동장치 (AEB: Automated Emergency Braking) 등의 첨단 운전자 보조 시스템 (ADAS)을 개발하고 상용화하였다. 또한 Audi의 Audi AI Traffic Jam Pilot, Tesla의 Autopilot, Mercedes-Benz의 Distronic Plus, 현대자동차의 Highway Driving Assist 및 BMW의 Driving Assistant Plus 와 같은 부분 자율 주행 시스템이 출시되고 있다. 이러한 부분 자율 주행 시스템은 여전히 운전자의 주의가 수반되어야 함에도 불구하고, 안전성을 크게 향상시키는 데 효과적이기 때문에 지속적으로 그 수요가 증가하고 있다. 최근 몇 년간 많은 수의 자율주행 사고가 발생하였으며, 그 빈도수가 빠르게 증가하여 사회적으로 주목받고 있다. 차량 사고는 인명 사고와 직접적으로 연관되기 때문에 자율 주행 차량의 사고들은 자율 주행 기술 신뢰성의 저하를 야기하여 사회적인 불안감을 키운다. 최근 자율 주행 관련 사고들로 인해, 자율주행 차량의 안전성의 보장이 더욱 강조되고 있다. 따라서 본 연구에서는 자율 주행 차량의 거동 제어를 고려하여 자율 주행 시스템 관점에서 차량의 안전성을 우선적으로 확보하는 접근 방식을 제안한다. 또한 자율주행 기술 개발은 단순하게 운전을 대체하는 기술이 아니라, 첨단기술의 집약 체로써 산업적으로 매우 큰 파급력을 가진다고 전망된다. 현재 자율주행 시스템은 기존 자동차 산업의 고전적인 틀에서 확장되어, 다양한 분야의 관점에서 주도적으로 개발이 진행되고 있다. 자율 주행은 다양한 기술의 복합적인 결합으로 구성되기 때문에, 현재 각기 다른 다양한 환경에서 개발이 진행 중이며, 아직 표준화되어 있지 않은 실정이다. 대부분 각 모듈 단위의 지엽적인 성능향상을 추구하는 경향이 있으며, 구성 모듈 간 관계가 고려된 전체 시스템 단위의 접근방식은 미흡한 실정이다. 세부 모듈 단위의 지엽적인 연구 개발은 통합 시, 모듈 간 상호작용으로 인한 영향으로 시스템 관점에서 적절한 성능을 확보하기 어려울 수 있다. 각 모듈의 성능만을 고려한 일방적인 방향의 연구는 한계가 명확하며, 연관된 모듈들의 특성을 고려하여 반영할 필요가 있다. 따라서 자율주행 전체 시스템의 관점에서, 차량 안전을 우선적으로 확보하고 전체 성능을 극대화하는 효과적인 접근 방식을 본 연구에서 제안하고자 한다. 본 연구는 자율 주행 시스템의 안정적이고 높은 성능을 확보하기 위해 전체 시스템 작동 측면에서 구성된 모듈 간의 상호 작용을 고려하여 효율적인 환경 인식 알고리즘을 개발하는데 중점을 둔다. 실질적인 관점에서 효과적인 정보 처리를 수행하고 차량 안전을 확보하기 위해 적응형 관심 영역 (ROI) 기반 계산 부하 관리 전략을 제안한다. 차량의 거동 특성, 도로 설계 표준, 추월 및 차선 변경과 같은 주변 차량의 주행 특성이 적응형 ROI 설계 및 주행 상황에 따른 영역 확장에 반영된다. 또한, 자율 주행 차량의 실질적인 안전을 보장하기 위해 ROI 설계에서 자율 주행 제어를 위한 거동 계획 결과가 고려된다. 보다 넓은 주변 영역에 대한 환경 정보를 확보하기 위해 라이다 데이터는 설계된 ROI별로 분류되며, 영역별 중요도에 따라 연산 과정이 분리되어 수행된다. 목표 시스템을 구성하는 모듈 별 연산 시간이 측정된 데이터 기반으로 통계적으로 분석된다. 운전자의 반응 시간, 산업 표준, 대상 하드웨어 사양 및 센서 성능을 기반으로 결정된 시스템 성능 조건을 고려하여, 안전성을 확보하기 위한 자율 주행 시스템의 적절한 샘플링 주기가 정의된다. 데이터 기반 다중 선형 회귀 분석은 인식 모듈을 구성하는 함수 별 실행 시간을 예측하기 위해 적용되며, 안정적인 실시간 성능을 보장하기 위해 적응형 ROI를 기반으로 자율 주행 안전에 필요한 데이터를 선택적으로 분류하여 연산 부하가 감축된다. 연산 부하 평가 관리에서 환경 인지 모듈과 전체 시스템의 연산 부하가 대상 환경에서의 적절성을 평가하고, 연산 부하 관리에 문제가 있을 때 자율 주행 차량의 거동을 제한하여 시스템 안정성을 유지함으로써 차량 안전성을 확보한다. 제안된 자율주행 인지 전략 및 알고리즘의 성능은 데이터 기반 시뮬레이션 및 실차 실험을 통해 검증되었다. 실험 결과를 통해 제안된 환경 인식 알고리즘은 자율 주행 시스템을 구성하는 모듈 간의 상호 작용을 고려하여 도심 도로 환경에서 자율 주행 차량의 안전성과 시스템의 안정적인 성능을 보장할 수 있음을 확인하였다.Since annually 1.2 million people die from car crashes worldwide, discussions about fundamental preventive measures for traffic accidents are taking place. According to the statistical survey, 94 percent of all traffic accidents are caused by human error. From the perspective of securing road safety, automated driving technology became interesting as a way to solve this serious problem, and its commercialization was considered through a step-by-step application through research and development. Major carmakers already have developed and commercialized advanced driver assistance systems (ADAS), such as lane keeping assistance system (LKAS), adaptive cruise control (ACC), parking assistance system (PAS), automated emergency braking (AEB), and so on. Furthermore, partially automated driving systems are being installed in vehicles and released by carmakers. Audi AI Traffic Jam Pilot (Audi), Autopilot (Tesla), Distronic Plus (Mercedes-Benz), Highway Driving Assist (Hyundai Motor Company), and Driving Assistant Plus (BMW) are typical released examples of the partially automated driving system. These released partially automated driving systems are still must be accompanied by driver attention. Nevertheless, it is proving to be effective in significantly improving safety. In recent years, several automated driving accidents have occurred, and the frequency is rapidly increasing and attracting social attention. Since vehicle accidents are directly related to human casualty, accidents of automated vehicles cause social insecurity by causing a decrease in the reliability of automated driving technology. Due to recent automated driving-related accidents, the safety of the automated vehicle has been emphasized more. Therefore, in this study, we propose an approach to secure vehicle safety in terms of the entire system in consideration of the behavior control of the automated driving vehicle. In addition, the development of automated driving is not merely a replacement technology for driving, but it is expected to have an industrial assembly as integration of high technology. Currently, automated driving systems have been extended from the conventional framework of the existing automotive industry, and are being developed in various fields. Since automated driving is composed of a complex combination of various technologies, development is currently underway in various conditions and has not been standardized yet. Most developments tend to pursue local performance improvement in each module unit, and the overall system unit approaches considering the relationship between component modules is insufficient. Local research and development at the submodule level can be challenging to achieve adequate performance from a system-level due to the effects of module interaction in terms of system integration perspective. The one-way approach that considers only the performance of each module has its limitations. To overcome this problem, it is necessary to consider the characteristics of the modules involved. This dissertation focuses on developing an efficient environment perception algorithm by considering the interaction between configured modules in terms of entire system operation to secure the stable and high performance of an automated driving system. In order to perform effective information processing and secure vehicle safety from a practical perspective, we propose an adaptive ROI based computational load management strategy. The motion characteristics of the subject vehicle, road design standards, and driving tasks of the surrounding vehicles, such as overtaking, and lane change, are reflected in the design of adaptive ROI, and the expansion of the area according to the driving task is considered. Additionally, motion planning results for automated driving are considered in the ROI design in order to guarantee the practical safety of the automated vehicle. In order to secure reasonable and appropriate environment information for the wider areas, lidar sensor data is classified by the designed ROI, and separated processing is conducted according to area importance. Based on the driving data, the calculation time of each module constituting the target system is statistically analyzed. In consideration of the system performance constraint determined by using human reaction time and industry standards, target hardware specification and the performance of sensor, the appropriate sampling time for automated driving system is defined to enhance safety. The data-based multiple linear regression is applied to predict the computation time by each function constituting perception module, and the computational load reduction is applied sequentially by selecting the data essential for automated driving safety based on adaptive ROI to secure the stable real-time execution performance of the system. In computational load assessment, it evaluates whether the computational load of the environmental perception module and entire system are appropriate and restricts the vehicle behavior when there is a problem in the computational load management to ensure vehicle safety by maintaining system stability. The performance of the proposed strategy and algorithms is evaluated through driving data-based simulation and actual vehicle tests. Test results show that the proposed environment recognition algorithm, which considers the interactions between the modules that make up the automated driving system, guarantees the safety of automated vehicle and reliable performance of system in an urban environment scenario.Chapter 1 Introduction 1 1.1. Background and Motivation 1 1.2. Previous Researches 6 1.3. Thesis Objectives 11 1.4. Thesis Outline 13 Chapter 2 Overall Architecture 14 2.1. Automated Driving Architecture 14 2.2. Test Vehicle Configuration 19 Chapter 3 Design of Adaptive ROI and Processing 21 3.1. ROI Definition 25 3.1.1. ROI Design for Normal Driving Condition 30 3.1.2. ROI Design for Lane Change 50 3.1.3. ROI Design for Intersection 56 3.2. Data Processing based on Adaptive ROI 62 3.2.1. Point Cloud Categorization by Adaptive ROI 63 3.2.2. Separated Voxelization 66 3.2.3. Separated Clustering 70 Chapter 4 Environment Perception Algorithm for Automated Driving 75 4.1. Time Delay Compensation of Environment Sensor 77 4.1.1. Algorithm Structure of Time Delay Estimation and Compensation 78 4.1.2. Time Delay Compensation Algorithm 79 4.1.3. Analysis of Processing Delay 84 4.1.4. Test Data based Open-loop Simulation 91 4.2. Environment Representation 96 4.2.1. Static Obstacle Map Construction 98 4.2.2. Lane and Road Boundary Detection 100 4.3. Multiple Object State Estimation and Tracking based on Geometric Model-Free Approach 107 4.3.1. Prediction of Geometric Model-Free Approach 109 4.3.2. Track Management 111 4.3.3. Measurement Update 112 4.3.4. Performance Evaluation via vehicle test 114 Chapter 5 Computational Load Management 117 5.1. Processing Time Analysis of Driving Data 121 5.2. Processing Time Estimation based on Multiple Linear Regression 128 5.2.1. Clustering Processing Time Estimation 129 5.2.2. Multi Object Tracking (MOT) Processing Time Estimation 138 5.2.3. Validation through Data-based Simulation 146 5.3. Computational Load Management 149 5.3.1. Sequential Processing to Computation Load Reduction 151 5.3.2. Restriction of Driving Control 154 5.3.3. Validation through Data-based Simulation 159 Chapter 6 Vehicle Tests based Performance Evaluation 163 6.1. Test-data based Simulation 164 6.2. Vehicle Tests: Urban Automated Driving 171 6.2.1. Test Configuration 171 6.2.2. Motion Planning and Vehicle Control 172 6.2.3. Vehicle Tests Results 174 Chapter 7 Conclusions and Future Works 184 Bibliography 188 Abstract in Korean 200Docto