LO-Net: Deep Real-time Lidar Odometry

We present a novel deep convolutional network pipeline, LO-Net, for real-time lidar odometry estimation. Unlike most existing lidar odometry (LO) estimations that go through individually designed feature selection, feature matching, and pose estimation pipeline, LO-Net can be trained in an end-to-end manner. With a new mask-weighted geometric constraint loss, LO-Net can effectively learn feature representation for LO estimation, and can implicitly exploit the sequential dependencies and dynamics in the data. We also design a scan-to-map module, which uses the geometric and semantic information learned in LO-Net, to improve the estimation accuracy. Experiments on benchmark datasets demonstrate that LO-Net outperforms existing learning based approaches and has similar accuracy with the state-of-the-art geometry-based approach, LOAM

Enabling Multi-LiDAR Sensing in GNSS-Denied Environments: SLAM Dataset, Benchmark, and UAV Tracking with LiDAR-as-a-camera

The rise of Light Detection and Ranging (LiDAR) sensors has profoundly impacted industries ranging from automotive to urban planning. As these sensors become increasingly affordable and compact, their applications are diversifying, driving precision, and innovation. This thesis delves into LiDAR's advancements in autonomous robotic systems, with a focus on its role in simultaneous localization and mapping (SLAM) methodologies and LiDAR as a camera-based tracking for Unmanned Aerial Vehicles (UAV). Our contributions span two primary domains: the Multi-Modal LiDAR SLAM Benchmark, and the LiDAR-as-a-camera UAV Tracking. In the former, we have expanded our previous multi-modal LiDAR dataset by adding more data sequences from various scenarios. In contrast to the previous dataset, we employ different ground truth-generating approaches. We propose a new multi-modal multi-lidar SLAM-assisted and ICP-based sensor fusion method for generating ground truth maps. Additionally, we also supplement our data with new open road sequences with GNSS-RTK. This enriched dataset, supported by high-resolution LiDAR, provides detailed insights through an evaluation of ten configurations, pairing diverse LiDAR sensors with state-of-the-art SLAM algorithms. In the latter contribution, we leverage a custom YOLOv5 model trained on panoramic low-resolution images from LiDAR reflectivity (LiDAR-as-a-camera) to detect UAVs, demonstrating the superiority of this approach over point cloud or image-only methods. Additionally, we evaluated the real-time performance of our approach on the Nvidia Jetson Nano, a popular mobile computing platform. Overall, our research underscores the transformative potential of integrating advanced LiDAR sensors with autonomous robotics. By bridging the gaps between different technological approaches, we pave the way for more versatile and efficient applications in the future

Advances in Simultaneous Localization and Mapping in Confined Underwater Environments Using Sonar and Optical Imaging.

This thesis reports on the incorporation of surface information into a probabilistic simultaneous localization and mapping (SLAM) framework used on an autonomous underwater vehicle (AUV) designed for underwater inspection. AUVs operating in cluttered underwater environments, such as ship hulls or dams, are commonly equipped with Doppler-based sensors, which---in addition to navigation---provide a sparse representation of the environment in the form of a three-dimensional (3D) point cloud. The goal of this thesis is to develop perceptual algorithms that take full advantage of these sparse observations for correcting navigational drift and building a model of the environment. In particular, we focus on three objectives. First, we introduce a novel representation of this 3D point cloud as collections of planar features arranged in a factor graph. This factor graph representation probabalistically infers the spatial arrangement of each planar segment and can effectively model smooth surfaces (such as a ship hull). Second, we show how this technique can produce 3D models that serve as input to our pipeline that produces the first-ever 3D photomosaics using a two-dimensional (2D) imaging sonar. Finally, we propose a model-assisted bundle adjustment (BA) framework that allows for robust registration between surfaces observed from a Doppler sensor and visual features detected from optical images. Throughout this thesis, we show methods that produce 3D photomosaics using a combination of triangular meshes (derived from our SLAM framework or given a-priori), optical images, and sonar images. Overall, the contributions of this thesis greatly increase the accuracy, reliability, and utility of in-water ship hull inspection with AUVs despite the challenges they face in underwater environments. We provide results using the Hovering Autonomous Underwater Vehicle (HAUV) for autonomous ship hull inspection, which serves as the primary testbed for the algorithms presented in this thesis. The sensor payload of the HAUV consists primarily of: a Doppler velocity log (DVL) for underwater navigation and ranging, monocular and stereo cameras, and---for some applications---an imaging sonar.PhDElectrical Engineering: SystemsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/120750/1/paulozog_1.pd

Scan matching by cross-correlation and differential evolution

Scan matching is an important task, solved in the context of many high-level problems including pose estimation, indoor localization, simultaneous localization and mapping and others. Methods that are accurate and adaptive and at the same time computationally efficient are required to enable location-based services in autonomous mobile devices. Such devices usually have a wide range of high-resolution sensors but only a limited processing power and constrained energy supply. This work introduces a novel high-level scan matching strategy that uses a combination of two advanced algorithms recently used in this field: cross-correlation and differential evolution. The cross-correlation between two laser range scans is used as an efficient measure of scan alignment and the differential evolution algorithm is used to search for the parameters of a transformation that aligns the scans. The proposed method was experimentally validated and showed good ability to match laser range scans taken shortly after each other and an excellent ability to match laser range scans taken with longer time intervals between them.Web of Science88art. no. 85

Learning and Searching Methods for Robust, Real-Time Visual Odometry.

Accurate position estimation provides a critical foundation for mobile robot perception and control. While well-studied, it remains difficult to provide timely, precise, and robust position estimates for applications that operate in uncontrolled environments, such as robotic exploration and autonomous driving. Continuous, high-rate egomotion estimation is possible using cameras and Visual Odometry (VO), which tracks the movement of sparse scene content known as image keypoints or features. However, high update rates, often 30~Hz or greater, leave little computation time per frame, while variability in scene content stresses robustness. Due to these challenges, implementing an accurate and robust visual odometry system remains difficult. This thesis investigates fundamental improvements throughout all stages of a visual odometry system, and has three primary contributions: The first contribution is a machine learning method for feature detector design. This method considers end-to-end motion estimation accuracy during learning. Consequently, accuracy and robustness are improved across multiple challenging datasets in comparison to state of the art alternatives. The second contribution is a proposed feature descriptor, TailoredBRIEF, that builds upon recent advances in the field in fast, low-memory descriptor extraction and matching. TailoredBRIEF is an in-situ descriptor learning method that improves feature matching accuracy by efficiently customizing descriptor structures on a per-feature basis. Further, a common asymmetry in vision system design between reference and query images is described and exploited, enabling approaches that would otherwise exceed runtime constraints. The final contribution is a new algorithm for visual motion estimation: Perspective Alignment Search~(PAS). Many vision systems depend on the unique appearance of features during matching, despite a large quantity of non-unique features in otherwise barren environments. A search-based method, PAS, is proposed to employ features that lack unique appearance through descriptorless matching. This method simplifies visual odometry pipelines, defining one method that subsumes feature matching, outlier rejection, and motion estimation. Throughout this work, evaluations of the proposed methods and systems are carried out on ground-truth datasets, often generated with custom experimental platforms in challenging environments. Particular focus is placed on preserving runtimes compatible with real-time operation, as is necessary for deployment in the field.PhDComputer Science and EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/113365/1/chardson_1.pd

주행계 및 지도 작성을 위한 3차원 확률적 정규분포변환의 정합 방법

학위논문 (박사)-- 서울대학교 대학원 : 공과대학 전기·컴퓨터공학부, 2019. 2. 이범희.로봇은 거리센서를 이용하여 위치한 환경의 공간 정보를 점군(point set) 형태로 수집할 수 있는데, 이렇게 수집한 정보를 환경의 복원에 이용할 수 있다. 또한, 로봇은 점군과 모델을 정합하는 위치를 추정할 수 있다. 거리센서가 수집한 점군이 2차원에서 3차원으로 확장되고 해상도가 높아지면서 점의 개수가 크게 증가하면서, NDT (normal distributions transform)를 이용한 정합이 ICP (iterative closest point)의 대안으로 부상하였다. NDT는 점군을 분포로 변환하여 공간을 표현하는 압축된 공간 표현 방법이다. 분포의 개수가 점의 개수에 비해 월등히 작기 때문에 ICP에 비해 빠른 성능을 가졌다. 그러나 NDT 정합 기반 위치 추정의 성능을 좌우하는 셀의 크기, 셀의 중첩 정도, 셀의 방향, 분포의 스케일, 대응쌍의 비중 등 파라미터를 설정하기가 매우 어렵다. 본 학위 논문에서는 이러한 어려움에 대응하여 NDT 정합 기반 위치 추정의 정확도를 향상할 수 있는 방법을 제안하였다. 본 논문은 표현법과 정합법 2개 파트로 나눌 수 있다. 표현법에 있어 본 논문은 다음 3개 방법을 제안하였다. 첫째, 본 논문에서는 분포의 퇴화를 막기 위해 경험적으로 공분산 행렬의 고유값을 수정하여 공간적 형태의 왜곡을 가져오는 문제점과 고해상도의 NDT를 생성할 때 셀당 점의 개수가 감소하며 구조를 반영하는 분포가 형성되지 않는 문제점을 주목했다. 이를 해결하기 위하여 각 점에 대해 불확실성을 부여하고, 평균과 분산의 기대값으로 수정한 확률적 NDT (PNDT, probabilistic NDT) 표현법을 제안하였다. 공간 정보의 누락 없이 모든 점을 분포로 변환한 NDT를 통해 향상된 정확도를 보인 PNDT는 샘플링을 통한 가을을 가능하도록 하였다. 둘째, 본 논문에서는 정육면체를 셀로 다루며, 셀을 중심좌표와 변의 길이로 정의한다. 또한, 셀들로 이뤄진 격자를 각 셀의 중심점 사이의 간격과 셀의 크기로 정의한다. 이러한 정의를 토대로, 본 논문에서는 셀의 확대를 통하여 셀을 중첩시키는 방법과 셀의 간격 조절을 통하여 셀을 중첩시키는 방법을 제안하였다. 본 논문은 기존 2D NDT에서 사용한 셀의 삽입법을 주목하였다. 단순입방구조를 이루는 기존 방법 외에 면심입방구조와 체심입방구조의 셀로 이뤄진 격자가 생성하였다. 그 다음 해당 격자를 이용하여 NDT를 생성하는 방법을 제안하였다. 또한, 이렇게 생성된 NDT를 정합할 때 많은 시간을 소요하기 때문에 대응쌍 검색 영역을 정의하여 정합 속도를 향상하였다. 셋째, 저사양 로봇들은 점군 지도를 NDT 지도로 압축하여 보관하는 것이 효율적이다. 그러나 로봇 포즈가 갱신되거나, 다개체 로봇간 랑데뷰가 일어나 지도를 공유 및 결합하는 경우 NDT의 분포 형태가 왜곡되는 문제가 발생한다. 이러한 문제를 해결하기 위하여 NDT 재생성 방법을 제안하였다. 정합법에 있어 본 논문은 다음 4개 방법을 제안하였다. 첫째, 점군의 각 점에 대해 대응되는 색상 정보가 제공될 때 색상 hue를 이용한 향상된 NDT 정합으로 각 대응쌍에 대해 hue의 유사도를 비중으로 사용하는 목적함수를 제안하였다. 둘째, 본 논문은은 다양한 크기의 위치 변화량에 대응하기 위한 다중 레이어 NDT 정합 (ML-NDT, multi-layered NDT)의 한계를 극복하기 위하여 키레이어 NDT 정합 (KL-NDT, key-layered NDT)을 제안하였다. KL-NDT는 각 해상도의 셀에서 활성화된 점의 개수 변화량을 척도로 키레이어를 결정한다. 또한 키레이어에서 위치의 추정값이 수렴할 때까지 정합을 수행하는 방식을 취하여 다음 키레이어에 더 좋은 초기값을 제공한다. 셋째, 본 논문은 이산적인 셀로 인해 NDT간 정합 기법인 NDT-D2D (distribution-to-distribution NDT)의 목적 함수가 비선형이며 국소 최저치의 완화를 위한 방법으로 신규 NDT와 모델 NDT에 독립된 스케일을 정의하고 스케일을 변화하며 정합하는 동적 스케일 기반 NDT 정합 (DSF-NDT-D2D, dynamic scaling factor-based NDT-D2D)을 제안하였다. 마지막으로, 본 논문은 소스 NDT와 지도간 증대적 정합을 이용한 주행계 추정 및 지도 작성 방법을 제안하였다. 이 방법은 로봇의 현재 포즈에 대한 초기값을 소스 점군에 적용한 뒤 NDT로 변환하여 지도 상 NDT와 가능한 한 유사한 NDT를 작성한다. 그 다음 로봇 포즈 및 소스 NDT의 GC (Gaussian component)를 고려하여 부분지도를 추출한다. 이렇게 추출한 부분지도와 소스 NDT는 다중 레이어 NDT 정합을 수행하여 정확한 주행계를 추정하고, 추정 포즈로 소스 점군을 회전 및 이동 후 기존 지도를 갱신한다. 이러한 과정을 통해 이 방법은 현재 최고 성능을 가진 LOAM (lidar odometry and mapping)에 비하여 더 높은 정확도와 더 빠른 처리속도를 보였다.The robot is a self-operating device using its intelligence, and autonomous navigation is a critical form of intelligence for a robot. This dissertation focuses on localization and mapping using a 3D range sensor for autonomous navigation. The robot can collect spatial information from the environment using a range sensor. This information can be used to reconstruct the environment. Additionally, the robot can estimate pose variations by registering the source point set with the model. Given that the point set collected by the sensor is expanded in three dimensions and becomes dense, registration using the normal distribution transform (NDT) has emerged as an alternative to the most commonly used iterative closest point (ICP) method. NDT is a compact representation which describes using a set of GCs (GC) converted from a point set. Because the number of GCs is much smaller than the number of points, with regard to the computation time, NDT outperforms ICP. However, the NDT has issues to be resolved, such as the discretization of the point set and the objective function. This dissertation is divided into two parts: representation and registration. For the representation part, first we present the probabilistic NDT (PNDT) to deal with the destruction and degeneration problems caused by the small cell size and the sparse point set. PNDT assigns an uncertainty to each point sample to convert a point set with fewer than four points into a distribution. As a result, PNDT allows for more precise registration using small cells. Second, we present lattice adjustment and cell insertion methods to overlap cells to overcome the discreteness problem of the NDT. In the lattice adjustment method, a lattice is expressed as the distance between the cells and the side length of each cell. In the cell insertion method, simple, face-centered-cubic, and body-centered-cubic lattices are compared. Third, we present a means of regenerating the NDT for the target lattice. A single robot updates its poses using simultaneous localization and mapping (SLAM) and fuses the NDT at each pose to update its NDT map. Moreover, multiple robots share NDT maps built with inconsistent lattices and fuse the maps. Because the simple fusion of the NDT maps can change the centers, shapes, and normal vectors of GCs, the regeneration method subdivides the NDT into truncated GCs using the target lattice and regenerates the NDT. For the registration part, first we present a hue-assisted NDT registration if the robot acquires color information corresponding to each point sample from a vision sensor. Each GC of the NDT has a distribution of the hue and uses the similarity of the hue distributions as the weight in the objective function. Second, we present a key-layered NDT registration (KL-NDT) method. The multi-layered NDT registration (ML-NDT) registers points to the NDT in multiple resolutions of lattices. However, the initial cell size and the number of layers are difficult to determine. KL-NDT determines the key layers in which the registration is performed based on the change of the number of activated points. Third, we present a method involving dynamic scaling factors of the covariance. This method scales the source NDT at zero initially to avoid a negative correlation between the likelihood and rotational alignment. It also scales the target NDT from the maximum scale to the minimum scale. Finally, we present a method of incremental registration of PNDTs which outperforms the state-of-the-art lidar odometry and mapping method.1 Introduction 1 1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3.1 Point Set Registration . . . . . . . . . . . . . . . . . . . . . 7 1.3.2 Incremental Registration for Odometry Estimation . . . . . . 16 1.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.5 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2 Preliminaries 21 2.1 NDT Representation . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2 NDT Registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.3 NDT Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.4 Transformation Matrix and The Parameter Vector . . . . . . . . . . . 27 2.5 Cubic Cell and Lattice . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.6 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.7 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.8 Evaluation of Registration . . . . . . . . . . . . . . . . . . . . . . . 31 2.9 Benchmark Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3 Probabilistic NDT Representation 34 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.2 Uncertainty of Point Based on Sensor Model . . . . . . . . . . . . . . 36 3.3 Probabilistic NDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.4 Generalization of NDT Registration Based on PNDT . . . . . . . . . 40 3.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.5.2 Evaluation of Representation . . . . . . . . . . . . . . . . . . 41 3.5.3 Evaluation of Registration . . . . . . . . . . . . . . . . . . . 46 3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 4 Interpolation for NDT Using Overlapped Regular Cells 51 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.2 Lattice Adjustment . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.3 Crystalline NDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.4.1 Lattice Adjustment . . . . . . . . . . . . . . . . . . . . . . . 56 4.4.2 Performance of Crystalline NDT . . . . . . . . . . . . . . . . 60 4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5 Regeneration of Normal Distributions Transform 65 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 5.2 Mathematical Preliminaries . . . . . . . . . . . . . . . . . . . . . . . 67 5.2.1 Trivariate Normal Distribution . . . . . . . . . . . . . . . . . 67 5.2.2 Truncated Trivariate Normal Distribution . . . . . . . . . . . 67 5.3 Regeneration of NDT . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.3.1 Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.3.2 Subdivision of Gaussian Components . . . . . . . . . . . . . 70 5.3.3 Fusion of Gaussian Components . . . . . . . . . . . . . . . . 72 5.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.4.1 Evaluation Metrics for Representation . . . . . . . . . . . . . 73 5.4.2 Representation Performance of the Regenerated NDT . . . . . 75 5.4.3 Computation Performance of the Regeneration . . . . . . . . 82 5.4.4 Application of Map Fusion . . . . . . . . . . . . . . . . . . . 83 5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 6 Hue-Assisted Registration 91 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.2 Preliminary of the HSV Model . . . . . . . . . . . . . . . . . . . . . 92 6.3 Colored Octree for Subdivision . . . . . . . . . . . . . . . . . . . . . 94 6.4 HA-NDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 6.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.5.1 Evaluation of HA-NDT against nhue . . . . . . . . . . . . . . 97 6.5.2 Evaluation of NDT and HA-NDT . . . . . . . . . . . . . . . 98 6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 7 Key-Layered NDT Registration 103 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 7.2 Key-layered NDT-P2D . . . . . . . . . . . . . . . . . . . . . . . . . 105 7.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 7.3.1 Evaluation of KL-NDT-P2D and ML-NDT-P2D . . . . . . . . 108 7.3.2 Evaluation of KL-NDT-D2D and ML-NDT-D2D . . . . . . . 111 7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 8 Scaled NDT and The Multi-scale Registration 113 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 8.2 Scaled NDT representation and L2 distance . . . . . . . . . . . . . . 114 8.3 NDT-D2D with dynamic scaling factors of covariances . . . . . . . . 116 8.4 Range of scaling factors . . . . . . . . . . . . . . . . . . . . . . . . . 120 8.5 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 8.5.1 Evaluation of the presented method without initial guess . . . 122 8.5.2 Application of odometry estimation . . . . . . . . . . . . . . 125 8.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 9 Scan-to-map Registration 129 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 9.2 Multi-layered PNDT . . . . . . . . . . . . . . . . . . . . . . . . . . 130 9.3 NDT Incremental Registration . . . . . . . . . . . . . . . . . . . . . 132 9.3.1 Initialization of PNDT-Map . . . . . . . . . . . . . . . . . . 133 9.3.2 Generation of Source ML-PNDT . . . . . . . . . . . . . . . . 134 9.3.3 Reconstruction of The Target ML-PNDT . . . . . . . . . . . 134 9.3.4 Pose Estimation Based on Multi-layered Registration . . . . . 135 9.3.5 Update of PNDT-Map . . . . . . . . . . . . . . . . . . . . . 136 9.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 9.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 10 Conclusions 142 Bibliography 145 초록 159 감사의 글 162Docto