Fast and robust 3D feature extraction from sparse point clouds

Matching 3D point clouds, a critical operation in map building and localization, is difficult with Velodyne-type sensors due to the sparse and non-uniform point clouds that they produce. Standard methods from dense 3D point clouds are generally not effective. In this paper, we describe a featurebased approach using Principal Components Analysis (PCA) of neighborhoods of points, which results in mathematically principled line and plane features. The key contribution in this work is to show how this type of feature extraction can be done efficiently and robustly even on non-uniformly sampled point clouds. The resulting detector runs in real-time and can be easily tuned to have a low false positive rate, simplifying data association. We evaluate the performance of our algorithm on an autonomous car at the MCity Test Facility using a Velodyne HDL-32E, and we compare our results against the state-of-theart NARF keypoint detector. © 2016 IEEE

Graph-based Data Modeling and Analysis for Data Fusion in Remote Sensing

Hyperspectral imaging provides the capability of increased sensitivity and discrimination over traditional imaging methods by combining standard digital imaging with spectroscopic methods. For each individual pixel in a hyperspectral image (HSI), a continuous spectrum is sampled as the spectral reflectance/radiance signature to facilitate identification of ground cover and surface material. The abundant spectrum knowledge allows all available information from the data to be mined. The superior qualities within hyperspectral imaging allow wide applications such as mineral exploration, agriculture monitoring, and ecological surveillance, etc. The processing of massive high-dimensional HSI datasets is a challenge since many data processing techniques have a computational complexity that grows exponentially with the dimension. Besides, a HSI dataset may contain a limited number of degrees of freedom due to the high correlations between data points and among the spectra. On the other hand, merely taking advantage of the sampled spectrum of individual HSI data point may produce inaccurate results due to the mixed nature of raw HSI data, such as mixed pixels, optical interferences and etc. Fusion strategies are widely adopted in data processing to achieve better performance, especially in the field of classification and clustering. There are mainly three types of fusion strategies, namely low-level data fusion, intermediate-level feature fusion, and high-level decision fusion. Low-level data fusion combines multi-source data that is expected to be complementary or cooperative. Intermediate-level feature fusion aims at selection and combination of features to remove redundant information. Decision level fusion exploits a set of classifiers to provide more accurate results. The fusion strategies have wide applications including HSI data processing. With the fast development of multiple remote sensing modalities, e.g. Very High Resolution (VHR) optical sensors, LiDAR, etc., fusion of multi-source data can in principal produce more detailed information than each single source. On the other hand, besides the abundant spectral information contained in HSI data, features such as texture and shape may be employed to represent data points from a spatial perspective. Furthermore, feature fusion also includes the strategy of removing redundant and noisy features in the dataset. One of the major problems in machine learning and pattern recognition is to develop appropriate representations for complex nonlinear data. In HSI processing, a particular data point is usually described as a vector with coordinates corresponding to the intensities measured in the spectral bands. This vector representation permits the application of linear and nonlinear transformations with linear algebra to find an alternative representation of the data. More generally, HSI is multi-dimensional in nature and the vector representation may lose the contextual correlations. Tensor representation provides a more sophisticated modeling technique and a higher-order generalization to linear subspace analysis. In graph theory, data points can be generalized as nodes with connectivities measured from the proximity of a local neighborhood. The graph-based framework efficiently characterizes the relationships among the data and allows for convenient mathematical manipulation in many applications, such as data clustering, feature extraction, feature selection and data alignment. In this thesis, graph-based approaches applied in the field of multi-source feature and data fusion in remote sensing area are explored. We will mainly investigate the fusion of spatial, spectral and LiDAR information with linear and multilinear algebra under graph-based framework for data clustering and classification problems

SMURF: Spatial Multi-Representation Fusion for 3D Object Detection with 4D Imaging Radar

The 4D Millimeter wave (mmWave) radar is a promising technology for vehicle sensing due to its cost-effectiveness and operability in adverse weather conditions. However, the adoption of this technology has been hindered by sparsity and noise issues in radar point cloud data. This paper introduces spatial multi-representation fusion (SMURF), a novel approach to 3D object detection using a single 4D imaging radar. SMURF leverages multiple representations of radar detection points, including pillarization and density features of a multi-dimensional Gaussian mixture distribution through kernel density estimation (KDE). KDE effectively mitigates measurement inaccuracy caused by limited angular resolution and multi-path propagation of radar signals. Additionally, KDE helps alleviate point cloud sparsity by capturing density features. Experimental evaluations on View-of-Delft (VoD) and TJ4DRadSet datasets demonstrate the effectiveness and generalization ability of SMURF, outperforming recently proposed 4D imaging radar-based single-representation models. Moreover, while using 4D imaging radar only, SMURF still achieves comparable performance to the state-of-the-art 4D imaging radar and camera fusion-based method, with an increase of 1.22% in the mean average precision on bird's-eye view of TJ4DRadSet dataset and 1.32% in the 3D mean average precision on the entire annotated area of VoD dataset. Our proposed method demonstrates impressive inference time and addresses the challenges of real-time detection, with the inference time no more than 0.05 seconds for most scans on both datasets. This research highlights the benefits of 4D mmWave radar and is a strong benchmark for subsequent works regarding 3D object detection with 4D imaging radar

Scalability of lineament and fracture networks within the crystalline Wiborg Rapakivi Batholith, SE Finland

Multiscale lineament and fracture extraction conducted within the Wiborg Rapakivi Batholith offers insights both into the brittle bedrock structures of the batholith and to the scale-dependence of lineament and fracture analysis results. Multiscale fracture studies from crystalline rocks are sparse even though brittle structures in the crystalline bedrock significantly affect the flow models of fluids, hydrothermal heat and hydrocarbons, and are the main factor controlling the permeability in crystalline rocks. The main goal of this study is to assess the scalability of lineament and fracture networks through statistic characterization of lineament and fracture datasets extracted from four scales of observation using geometric and topological parameters, and by studying the subsequent correlations between the dataset characterizations. The parameters are acquired from both the individual lineaments and fractures and from their respective networks. Brittle bedrock structures were extracted manually using two principle methods: lineament traces were digitized from Light Detection And Ranging (LiDAR) digital elevation models and fracture traces were digitized from drone-based orthophotography of bedrock outcrops. Both extractions result in two-dimensional datasets and, consequently, all characterizations of these datasets along with the scalability analysis results are limited to two dimensions. The crystalline Wiborg Rapakivi Batholith is structurally isotropic and lithologically sufficiently homogeneous so that the effect of both precursor fabrics and lithological variations can be ignored when considering the genesis and emplacement of brittle bedrock structures in the batholith. Scalability analyses conducted within this investigation revealed that the results of lineament and fracture network extractions are always dependent on the scale of observation. Even dimensionless parameters of networks, such as connectivity, were found to follow a scale-dependent trend: The apparent connectivity of a lineament or fracture network decreases as the scale of observation increases. The characterizations of the datasets were used for the interpretation of Wiborg Rapakivi Batholith fracture patterns and paleostresses, which could be compared to Olkiluoto site studies of paleostresses in southern Finland.Viipurin rapakivibatoliitin alueella useassa mittakaavassa tehty lineamenttien ja rakojen kartoitus antaa tietoa sekä batoliitin hauraista kallioperän rakenteista että lineamentti- ja rakokartoituksen tulosten skaalariippuvuudesta. Useassa mittakaavassa tehtävät rakotutkimukset kiteisistä kivistä ovat harvinaisia, vaikka kiteisen kallioperän hauraat rakenteet vaikuttavat vahvasti nesteiden, kaasujen, hydrotermisen lämmön ja hiilivetyjen virtausmalleihin ja ne ovat kiteisen kallioperän permeabiliteetin tärkein kontrolloija. Tämän tutkimuksen tärkein tavoite on lineamentti- ja rakoverkkojen skaalautuvuuden tutkiminen. Tutkiminen tapahtuu ensin karakterisoimalla tilastollisesti lineamentti- ja rakoaineistoja neljästä eri mittakaavasta käyttäen geometrisiä ja topologisia parametrejä, ja sitten tutkimalla aineistojen karakterisointien välisiä korrelaatioita. Parametrit ovat sekä yksittäisten lineamenttien ja rakojen että lineamentti- ja rakoverkkojen parametrejä. Kallioperän hauraat rakenteet kartoitettiin kahdella eri metodilla: lineamenttiviivat digitoitiin laserkeilauskorkeusmalleista (LiDAR DEMs) ja rakoviivat digitoitiin lennokilla otetuista kalliopaljastumien ortomosaiikkikuvista. Molempien kartoitusten tuloksena oli kaksiulotteisia aineistoja, ja tämän takia myös kaikki aineistojen karakterisoinnit ja skaalautuvuusanalyysien tulokset ovat kaksiulotteisia. Kiteinen Viipurin rapakivibatoliitti on rakenteellisesti isotrooppinen ja litologisesti riittävän homogeeninen, jotta sekä edeltävät rakenteet että litologiset vaihtelut voidaan jättää huomioimatta, kun tutkimuksen kohteena on batoliitin hauraiden rakenteiden syntyminen. Tämän tutkimuksen puitteissa tehdyt skaalautuvuusanalyysit osoittivat, että lineamentti- ja rakoverkkokartoitusten tulokset ovat aina riippuvaisia kartoituksen mittakaavasta. Jopa yksiköttömät verkkojen parametrit, kuten verkottuneisuus, seurasi skaalariippuvaista trendiä: Näennäinen lineamentti- tai rakoverkon verkottuneisuus pienenee, kun mittakaava suurenee. Lineamentti- ja rakoaineistojen karakterisointeja käytettiin Viipurin rapakivibatoliitin rakojen muodostamien kuvioiden ja paleostressien tulkintaan. Paleostressitulkintoja voi verrata Olkiluodossa tehtyihin tutkimuksiin paleostresseistä eteläisessä Suomessa

Real-time implementation of 3D LiDAR point cloud semantic segmentation in an FPGA

Dissertação de mestrado em Informatics EngineeringIn the last few years, the automotive industry has relied heavily on deep learning applications for perception solutions. With data-heavy sensors, such as LiDAR, becoming a standard, the task of developing low-power and real-time applications has become increasingly more challenging. To obtain the maximum computational efficiency, no longer can one focus solely on the software aspect of such applications, while disregarding the underlying hardware. In this thesis, a hardware-software co-design approach is used to implement an inference application leveraging the SqueezeSegV3, a LiDAR-based convolutional neural network, on the Versal ACAP VCK190 FPGA. Automotive requirements carefully drive the development of the proposed solution, with real-time performance and low power consumption being the target metrics. A first experiment validates the suitability of Xilinx’s Vitis-AI tool for the deployment of deep convolutional neural networks on FPGAs. Both the ResNet-18 and SqueezeNet neural networks are deployed to the Zynq UltraScale+ MPSoC ZCU104 and Versal ACAP VCK190 FPGAs. The results show that both networks achieve far more than the real-time requirements while consuming low power. Compared to an NVIDIA RTX 3090 GPU, the performance per watt during both network’s inference is 12x and 47.8x higher and 15.1x and 26.6x higher respectively for the Zynq UltraScale+ MPSoC ZCU104 and the Versal ACAP VCK190 FPGA. These results are obtained with no drop in accuracy in the quantization step. A second experiment builds upon the results of the first by deploying a real-time application containing the SqueezeSegV3 model using the Semantic-KITTI dataset. A framerate of 11 Hz is achieved with a peak power consumption of 78 Watts. The quantization step results in a minimal accuracy and IoU degradation of 0.7 and 1.5 points respectively. A smaller version of the same model is also deployed achieving a framerate of 19 Hz and a peak power consumption of 76 Watts. The application performs semantic segmentation over all the point cloud with a field of view of 360°.Nos últimos anos a indústria automóvel tem cada vez mais aplicado deep learning para solucionar problemas de perceção. Dado que os sensores que produzem grandes quantidades de dados, como o LiDAR, se têm tornado standard, a tarefa de desenvolver aplicações de baixo consumo energético e com capacidades de reagir em tempo real tem-se tornado cada vez mais desafiante. Para obter a máxima eficiência computacional, deixou de ser possível focar-se apenas no software aquando do desenvolvimento de uma aplicação deixando de lado o hardware subjacente. Nesta tese, uma abordagem de desenvolvimento simultâneo de hardware e software é usada para implementar uma aplicação de inferência usando o SqueezeSegV3, uma rede neuronal convolucional profunda, na FPGA Versal ACAP VCK190. São os requisitos automotive que guiam o desenvolvimento da solução proposta, sendo a performance em tempo real e o baixo consumo energético, as métricas alvo principais. Uma primeira experiência valida a aptidão da ferramenta Vitis-AI para a implantação de redes neuronais convolucionais profundas em FPGAs. As redes ResNet-18 e SqueezeNet são ambas implantadas nas FPGAs Zynq UltraScale+ MPSoC ZCU104 e Versal ACAP VCK190. Os resultados mostram que ambas as redes ultrapassam os requisitos de tempo real consumindo pouca energia. Comparado com a GPU NVIDIA RTX 3090, a performance por Watt durante a inferência de ambas as redes é superior em 12x e 47.8x e 15.1x e 26.6x respetivamente na Zynq UltraScale+ MPSoC ZCU104 e na Versal ACAP VCK190. Estes resultados foram obtidos sem qualquer perda de accuracy na etapa de quantização. Uma segunda experiência é feita no seguimento dos resultados da primeira, implantando uma aplicação de inferência em tempo real contendo o modelo SqueezeSegV3 e usando o conjunto de dados Semantic-KITTI. Um framerate de 11 Hz é atingido com um pico de consumo energético de 78 Watts. O processo de quantização resulta numa perda mínima de accuracy e IoU com valores de 0.7 e 1.5 pontos respetivamente. Uma versão mais pequena do mesmo modelo é também implantada, atingindo uma framerate de 19 Hz e um pico de consumo energético de 76 Watts. A aplicação desenvolvida executa segmentação semântica sobre a totalidade das nuvens de pontos LiDAR, com um campo de visão de 360°

Point-cloud based 3D object detection and classification methods for self-driving applications: A survey and taxonomy

Autonomous vehicles are becoming central for the future of mobility, supported by advances in deep learning techniques. The performance of aself-driving system is highly dependent on the quality of the perception task. Developments in sensor technologies have led to an increased availability of 3D scanners such as LiDAR, allowing for a more accurate representation of the vehicle's surroundings, leading to safer systems. The rapid development and consequent rise of research studies around self-driving systems since early 2010, resulted in a tremendous increase in the number and novelty of object detection methods. After the first wave of works that essentially tried to expand known techniques from object detection in images, more recently there has been a notable development in newer and more adapted to LiDAR data works. This paper addresses the existing literature on object detection using LiDAR data within the scope of self-driving and brings a systematic way for analysing it. Unlike general object detection surveys, we will focus on point-cloud data, which presents specific challenges, notably its high-dimensional and sparse nature. This work introduces a common object detection pipeline and taxonomy to facilitate a thorough comparison between different techniques and, departing from it, this work will critically examine the representation of data (critical for complexity reduction), feature extraction and finally the object detection models. A comparison between performance results of the different models is included, alongside with some future research challenges.This work is supported by European Structural and Investment Funds in the FEDER component, through the Operational Competitiveness and Internationalization Programme (COMPETE 2020) [Project n. 037902; Funding Reference: POCI-01-0247-FEDER-037902]

A framework for representing, building and reusing novel atate-of-the-art three-dimensional object detection models in point clouds targeting self-driving applications

The rapid development of deep learning has brought novel methodologies for 3D object detection using LiDAR sensing technology. These improvements in precision and inference speed performances lead to notable high performance and real-time inference, which is especially important for self-driving purposes. However, the developments carried by these approaches overwhelm the research process in this area since new methods, technologies and software versions lead to different project necessities, specifications and requirements. Moreover, the improvements brought by the new methods may be due to improvements in newer versions of deep learning frameworks and not just the novelty and innovation of the model architecture. Thus, it has become crucial to create a framework with the same software versions, specifications and requirements that accommodate all these methodologies and allow for the easy introduction of new methods and models. A framework is proposed that abstracts the implementation, reusing and building of novel methods and models. The main idea is to facilitate the representation of state-of-the-art (SoA) approaches and simultaneously encourage the implementation of new approaches by reusing, improving and innovating modules in the proposed framework, which has the same software specifications to allow for a fair comparison. This makes it possible to determine if the key innovation approach outperforms the current SoA by comparing models in a framework with the same software specifications and requirements.This work has been supported by FCT—Fundação para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020 and the project “Integrated and Innovative Solutions for the well-being of people in complex urban centers” within the Project Scope NORTE-01-0145-FEDER 000086. The work of Pedro Oliveira was supported by the doctoral Grant PRT/BD/154311/2022 financed by the Portuguese Foundation for Science and Technology (FCT), and with funds from European Union, under MIT Portugal Program. The work of Paulo Novais and Dalila Durães is supported by National Funds through the Portuguese funding agency, FCT—Fundação para a Ciência e a Tecnologia within project 2022.06822.PTDC