Stress-Testing LiDAR Registration

Point cloud registration (PCR) is an important task in many fields including autonomous driving with LiDAR sensors. PCR algorithms have improved significantly in recent years, by combining deep-learned features with robust estimation methods. These algorithms succeed in scenarios such as indoor scenes and object models registration. However, testing in the automotive LiDAR setting, which presents its own challenges, has been limited. The standard benchmark for this setting, KITTI-10m, has essentially been saturated by recent algorithms: many of them achieve near-perfect recall. In this work, we stress-test recent PCR techniques with LiDAR data. We propose a method for selecting balanced registration sets, which are challenging sets of frame-pairs from LiDAR datasets. They contain a balanced representation of the different relative motions that appear in a dataset, i.e. small and large rotations, small and large offsets in space and time, and various combinations of these. We perform a thorough comparison of accuracy and run-time on these benchmarks. Perhaps unexpectedly, we find that the fastest and simultaneously most accurate approach is a version of advanced RANSAC. We further improve results with a novel pre-filtering method

Efficient High-Resolution Template Matching with Vector Quantized Nearest Neighbour Fields

Template matching is a fundamental problem in computer vision and has applications in various fields, such as object detection, image registration, and object tracking. The current state-of-the-art methods rely on nearest-neighbour (NN) matching in which the query feature space is converted to NN space by representing each query pixel with its NN in the template pixels. The NN-based methods have been shown to perform better in occlusions, changes in appearance, illumination variations, and non-rigid transformations. However, NN matching scales poorly with high-resolution data and high feature dimensions. In this work, we present an NN-based template-matching method which efficiently reduces the NN computations and introduces filtering in the NN fields to consider deformations. A vector quantization step first represents the template with $k$ features, then filtering compares the template and query distributions over the $k$ features. We show that state-of-the-art performance was achieved in low-resolution data, and our method outperforms previous methods at higher resolution showing the robustness and scalability of the approach

Neural Semantic Surface Maps

We present an automated technique for computing a map between two genus-zero shapes, which matches semantically corresponding regions to one another. Lack of annotated data prohibits direct inference of 3D semantic priors; instead, current State-of-the-art methods predominantly optimize geometric properties or require varying amounts of manual annotation. To overcome the lack of annotated training data, we distill semantic matches from pre-trained vision models: our method renders the pair of 3D shapes from multiple viewpoints; the resulting renders are then fed into an off-the-shelf image-matching method which leverages a pretrained visual model to produce feature points. This yields semantic correspondences, which can be projected back to the 3D shapes, producing a raw matching that is inaccurate and inconsistent between different viewpoints. These correspondences are refined and distilled into an inter-surface map by a dedicated optimization scheme, which promotes bijectivity and continuity of the output map. We illustrate that our approach can generate semantic surface-to-surface maps, eliminating manual annotations or any 3D training data requirement. Furthermore, it proves effective in scenarios with high semantic complexity, where objects are non-isometrically related, as well as in situations where they are nearly isometric

Report on shape analysis and matching and on semantic matching

In GRAVITATE, two disparate specialities will come together in one working platform for the archaeologist: the fields of shape analysis, and of metadata search. These fields are relatively disjoint at the moment, and the research and development challenge of GRAVITATE is precisely to merge them for our chosen tasks. As shown in chapter 7 the small amount of literature that already attempts join 3D geometry and semantics is not related to the cultural heritage domain. Therefore, after the project is done, there should be a clear ‘before-GRAVITATE’ and ‘after-GRAVITATE’ split in how these two aspects of a cultural heritage artefact are treated.This state of the art report (SOTA) is ‘before-GRAVITATE’. Shape analysis and metadata description are described separately, as currently in the literature and we end the report with common recommendations in chapter 8 on possible or plausible cross-connections that suggest themselves. These considerations will be refined for the Roadmap for Research deliverable.Within the project, a jargon is developing in which ‘geometry’ stands for the physical properties of an artefact (not only its shape, but also its colour and material) and ‘metadata’ is used as a general shorthand for the semantic description of the provenance, location, ownership, classification, use etc. of the artefact. As we proceed in the project, we will find a need to refine those broad divisions, and find intermediate classes (such as a semantic description of certain colour patterns), but for now the terminology is convenient – not least because it highlights the interesting area where both aspects meet.On the ‘geometry’ side, the GRAVITATE partners are UVA, Technion, CNR/IMATI; on the metadata side, IT Innovation, British Museum and Cyprus Institute; the latter two of course also playing the role of internal users, and representatives of the Cultural Heritage (CH) data and target user’s group. CNR/IMATI’s experience in shape analysis and similarity will be an important bridge between the two worlds for geometry and metadata. The authorship and styles of this SOTA reflect these specialisms: the first part (chapters 3 and 4) purely by the geometry partners (mostly IMATI and UVA), the second part (chapters 5 and 6) by the metadata partners, especially IT Innovation while the joint overview on 3D geometry and semantics is mainly by IT Innovation and IMATI. The common section on Perspectives was written with the contribution of all

Consistent Correspondences for Shape and Image Problems

Establish consistent correspondences between different objects is a classic problem in computer science/vision. It helps to match highly similar objects in both 3D and 2D domain. Inthe 3D domain, finding consistent correspondences has been studying for more than 20 yearsand it is still a hot topic. In 2D domain, consistent correspondences can also help in puzzlesolving. However, only a few works are focused on this approach. In this thesis, we focuson finding consistent correspondences and extend to develop robust matching techniques inboth 3D shape segments and 2D puzzle solving. In the 3D domain, segment-wise matching isan important research problem that supports higher-level understanding of shapes in geometryprocessing. Many existing segment-wise matching techniques assume perfect input segmentation and would suffer from imperfect or over-segmented input. To handle this shortcoming,we propose multi-layer graphs (MLGs) to represent possible arrangements of partially mergedsegments of input shapes. We then adapt the diffusion pruning technique on the MLGs to findconsistent segment-wise matching. To obtain high-quality matching, we develop our own voting step which is able to remove inconsistent results, for finding hierarchically consistent correspondences as final output. We evaluate our technique with both quantitative and qualitativeexperiments on both man-made and deformable shapes. Experimental results demonstrate theeffectiveness of our technique when compared to two state-of-art methods. In the 2D domain,solving jigsaw puzzles is also a classic problem in computer vision with various applications.Over the past decades, many useful approaches have been introduced. Most existing worksuse edge-wise similarity measures for assembling puzzles with square pieces of the same size, and recent work innovates to use the loop constraint to improve efficiency and accuracy. Weobserve that most existing techniques cannot be easily extended to puzzles with rectangularpieces of arbitrary sizes, and no existing loop constraints can be used to model such challenging scenarios. We propose new matching approaches based on sub-edges/corners, modelledusing the MatchLift or diffusion framework to solve square puzzles with cycle consistency.We demonstrate the robustness of our approaches by comparing our methods with state-of-artmethods. We also show how puzzles with rectangular pieces of arbitrary sizes, or puzzles withtriangular and square pieces can be solved by our techniques