구조 감응형 데이터 증강 기법과 혼합 밀도 신경망을 이용한 라이다 기반 3차원 객체 검출 개선

학위논문(박사) -- 서울대학교대학원 : 융합과학기술대학원 융합과학부, 2023. 2. 곽노준.자율주행자동차, 로봇의 인식 장비로 많이 활용되고있는 라이다 (LiDAR) 는 레이저 펄스를 방출하여 되돌아오는 시간을 계산하여 포인트 클라우드 (point cloud) 형태로 주변 환경을 감지한다. 주변 환경을 감지할때 가장 중 요한 부분은 근처에 어떤 객체가 있는지, 어디에 위치해 있는지를 인식하는 것이고 이러한 작업을 수행하기 위해 포인트 클라우드를 활용하는 3차원 객 체 검출 기술들이 많이 연구되고 있다. 포인트 클라우드 데이터의 전처리 방법에 따라 매우 다양한 구조의 백본 네트워크 (backbone network) 가 연구되고 있다. 고도화된 백본 네트워크들로 인해 인식 성능에 큰 발전을 이루었지만, 이들의 형태가 크게 다르기 때문에 서로 호환성이 부족하여 연구들의 갈래가 많이 나누어지고 있다. 본 논문에 서 풀고자하는 문제는 파편화된 백본 네트워크의 구조들에 구애받지 않고 3차원 객체 검출기의 성능을 향상시킬 방법이 있는가 이다. 이를 위해 본 논문 에서는 포인트 클라우드 데이터 기반의 3차원 객체 검출 기술을 향상시키는 두 가지 방법을 제안한다. 첫 번째는 3차원 경계 상자 (3D bounding box) 의 구조적인 정보의 활용을 최대화하는 구조 감응형 데이터 증강 (PA-AUG) 기법이다. 3차원 경계 상자 라벨은 객체에 딱 맞게 생성되고 방향값을 포함하기 때문에 상자 내에 객체의 구조 정보를 포함하고 있다. 이를 활용하기 위해 우리는 3차원 경계 상자를 구조 감응형 파티션으로 구분하는 방식을 제안하고, 파티션 수준에서 수행되는 새로운 방식의 데이터 증강 기법을 제안한다. PA-AUG는 다양한 형태의 3차원 객체 검출기들의 성능을 강인하게 만들어주고, 학습 데이터를 2.5배 증 강시키는 만큼의 인식 성능 향상 효과를 보여준다. 두 번째는 혼합 밀도 신경망 기반 3차원 객체 검출 (MD3D) 기법이다. MD3D는 가우시간 혼합 모델 (Gaussian Mixture Model) 을 이용해 3차원 경 계 상자 회귀 문제를 밀도 예측 방식으로 재정의한 기법이다. 이러한 방식은 기존의 라벨 할당식의 학습 방법들과 달리 포인트 클라우드 전처리 형태에 구애받지 않고 동일한 학습 방식을 적용할 수 있다. 또한 기존 방식 대비 학습 에 필요한 하이퍼 파라미터가 현저히 적어서 최적화가 용이하여 인식 성능을 크게 높일 수 있을 뿐만 아니라 간단한 구조로 인해 인식 속도도 빨라지게 된다. PA-AUG와 MD3D는 모두 백본 네트워크 구조에 상관없이 다양한 3차원 객체 검출기에 공통적으로 사용될 수 있으며 높은 인식 성능 향상을 보여준다. 뿐만 아니라 두 기법은 검출기의 서로 다른 영역에 적용되는 기법이므로 함께 동시에 사용할 수 있고, 함께 사용했을때 인식 성능이 더욱 크게 향상된다.LiDAR (Light Detection And Ranging), which is widely used as a sensing device for autonomous vehicles and robots, emits laser pulses and calculates the return time to sense the surrounding environment in the form of a point cloud. When recognizing the surrounding environment, the most important part is recognizing what objects are nearby and where they are located, and 3D object detection methods using point clouds have been actively studied to perform these tasks. Various backbone networks for point cloud-based 3D object detection have been proposed according to the preprocessing method of point cloud data. Although advanced backbone networks have made great strides in detection performance, they are largely different in structure, so there is a lack of compatibility with each other. The problem to be solved in this dissertation is How to improve the performance of 3D object detectors regardless of their diverse backbone network structures?. This dissertation proposes two general methods to improve point cloud-based 3D object detectors. First, we propose a part-aware data augmentation (PA-AUG) method which maximizes the utilization of structural information of 3D bounding boxes. Since the 3D bounding box labels fit the objects boundaries and include the orientation value, they contain the structural information of the object in the box. To fully utilize the intra-object structural information, we propose a novel partaware partitioning method which separates 3D bounding boxes with characteristic sub-parts. PA-AUG applies newly proposed data augmentation methods at the partition level. It makes various types of 3D object detectors robust and brings the equivalent effect of increasing the train data by about 2.5×. Second, we propose a mixture-density-based 3D object detection (MD3D). MD3D predicts the distribution of 3D bounding boxes using a Gaussian mixture model (GMM). It reformulates the conventional regression methods as a density estimation problem. Thus, unlike conventional target assignment methods, it can be applied to any 3D object detector regardless of the point cloud preprocessing method. In addition, as it requires significantly fewer hyper-parameters compared to existing methods, it is easy to optimize the detection performance. MD3D also increases the detection speed due to its simple structure. Both PA-AUG and MD3D can be applied to any 3D object detector and shows an impressive increase in detection performance. The two proposed methods cover different stages of the object detection pipeline. Thus, they can be used simultaneously, and the experimental results show they have a synergy effect when applied together.1 Introduction 1 1.1 Problem Definition 3 1.2 Challenges 6 1.3 Contributions 8 1.3.1 Part-Aware Data Augmentation (PA-AUG) 8 1.3.2 Mixture-Density-based 3D Object Detection (MD3D) 9 1.3.3 Combination of PA-AUG and MD3D 10 1.4 Outline 10 2 Related Works 11 2.1 Data augmentation for Object Detection 11 2.1.1 2D Data augmentation 11 2.1.2 3D Data augmentation 12 2.2 LiDAR-based 3D Object Detection 13 2.3 Mixture Density Networks in Computer Vision 15 2.4 Datasets 16 2.4.1 KITTI Dataset 16 2.4.2 Waymo Open Dataset 18 2.5 Evaluation metric 19 2.5.1 Average Precision (AP) 19 2.5.2 Average Orientation Similarity (AOS) 22 2.5.3 Average Precision weighted by Heading (APH) 22 3 Part-Aware Data Augmentation (PA-AUG) 24 3.1 Introduction 24 3.2 Methods 27 3.2.1 Part-Aware Partitioning 27 3.2.2 Part-Aware Data Augmentation 28 3.3 Experiments 33 3.3.1 Results on the KITTI Dataset 33 3.3.2 Robustness Test 36 3.3.3 Data Efficiency Test 38 3.3.4 Ablation Study 40 3.4 Discussion 41 3.5 Conclusion 42 4 Mixture-Density-based 3D Object Detection (MD3D) 43 4.1 Introduction 43 4.2 Methods 47 4.2.1 Modeling Point-cloud-based 3D Object Detection with Mixture Density Network 47 4.2.2 Network Architecture 49 4.2.3 Loss function 52 4.3 Experiments 53 4.3.1 Datasets 53 4.3.2 Experiment Settings 53 4.3.3 Results on the KITTI Dataset 54 4.3.4 Latency of Each Module 56 4.3.5 Results on the Waymo Open Dataset 58 4.3.6 Analyzing Recall by object size 59 4.3.7 Ablation Study 60 4.3.8 Discussion 65 4.4 Conclusion 66 5 Combination of PA-AUG and MD3D 71 5.1 Methods 71 5.2 Experiments 72 5.2.1 Settings 72 5.2.2 Results on the KITTI Dataset 73 5.3 Discussion 76 6 Conclusion 77 6.1 Summary 77 6.2 Limitations and Future works 78 6.2.1 Hyper-parameter-free PA-AUG 78 6.2.2 Redefinition of Part-aware Partitioning 79 6.2.3 Application to other tasks 79 Abstract (In Korean) 94 감사의 글 96박

Multi-View 3D Object Detection Network for Autonomous Driving

This paper aims at high-accuracy 3D object detection in autonomous driving scenario. We propose Multi-View 3D networks (MV3D), a sensory-fusion framework that takes both LIDAR point cloud and RGB images as input and predicts oriented 3D bounding boxes. We encode the sparse 3D point cloud with a compact multi-view representation. The network is composed of two subnetworks: one for 3D object proposal generation and another for multi-view feature fusion. The proposal network generates 3D candidate boxes efficiently from the bird's eye view representation of 3D point cloud. We design a deep fusion scheme to combine region-wise features from multiple views and enable interactions between intermediate layers of different paths. Experiments on the challenging KITTI benchmark show that our approach outperforms the state-of-the-art by around 25% and 30% AP on the tasks of 3D localization and 3D detection. In addition, for 2D detection, our approach obtains 10.3% higher AP than the state-of-the-art on the hard data among the LIDAR-based methods.Comment: To appear in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 201

DecideNet: Counting Varying Density Crowds Through Attention Guided Detection and Density Estimation

In real-world crowd counting applications, the crowd densities vary greatly in spatial and temporal domains. A detection based counting method will estimate crowds accurately in low density scenes, while its reliability in congested areas is downgraded. A regression based approach, on the other hand, captures the general density information in crowded regions. Without knowing the location of each person, it tends to overestimate the count in low density areas. Thus, exclusively using either one of them is not sufficient to handle all kinds of scenes with varying densities. To address this issue, a novel end-to-end crowd counting framework, named DecideNet (DEteCtIon and Density Estimation Network) is proposed. It can adaptively decide the appropriate counting mode for different locations on the image based on its real density conditions. DecideNet starts with estimating the crowd density by generating detection and regression based density maps separately. To capture inevitable variation in densities, it incorporates an attention module, meant to adaptively assess the reliability of the two types of estimations. The final crowd counts are obtained with the guidance of the attention module to adopt suitable estimations from the two kinds of density maps. Experimental results show that our method achieves state-of-the-art performance on three challenging crowd counting datasets.Comment: CVPR 201

Event-Based Motion Segmentation by Motion Compensation

In contrast to traditional cameras, whose pixels have a common exposure time, event-based cameras are novel bio-inspired sensors whose pixels work independently and asynchronously output intensity changes (called "events"), with microsecond resolution. Since events are caused by the apparent motion of objects, event-based cameras sample visual information based on the scene dynamics and are, therefore, a more natural fit than traditional cameras to acquire motion, especially at high speeds, where traditional cameras suffer from motion blur. However, distinguishing between events caused by different moving objects and by the camera's ego-motion is a challenging task. We present the first per-event segmentation method for splitting a scene into independently moving objects. Our method jointly estimates the event-object associations (i.e., segmentation) and the motion parameters of the objects (or the background) by maximization of an objective function, which builds upon recent results on event-based motion-compensation. We provide a thorough evaluation of our method on a public dataset, outperforming the state-of-the-art by as much as 10%. We also show the first quantitative evaluation of a segmentation algorithm for event cameras, yielding around 90% accuracy at 4 pixels relative displacement.Comment: When viewed in Acrobat Reader, several of the figures animate. Video: https://youtu.be/0q6ap_OSBA

Road pollution estimation using static cameras and neural networks

Este artículo presenta una metodología para estimar la contaminación en carreteras mediante el análisis de secuencias de video de tráfico. El objetivo es aprovechar la gran red de cámaras IP existente en el sistema de carreteras de cualquier estado o país para estimar la contaminación en cada área. Esta propuesta utiliza redes neuronales de aprendizaje profundo para la detección de objetos, y un modelo de estimación de contaminación basado en la frecuencia de vehículos y su velocidad. Los experimentos muestran prometedores resultados que sugieren que el sistema se puede usar en solitario o combinado con los sistemas existentes para medir la contaminación en carreteras.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech