Dimensional discoveries: unveiling the potential of 3D heritage point clouds with a robust ontology framework

3D point clouds feature valuable geometric and, often, radiometric and semantic information to support studies, analyses and understanding of the surveyed scene. Due to their generally large size, the use and interpretation of point clouds could be problematic. User-friendly and quick approaches for querying these valuable datasets and retrieving information could surely support end-users, in particular in the heritage sector. This work presents an ontology-based approach to facilitate the query and use of 3D heritage point clouds by means of sets of rules in order to infer properties and characteristics of the surveyed scene. Our approach is focused on linking together 3D spatial data and expert knowledge, in a way that the ontology can elaborate, represent, enrich and query a given point cloud. Results show how different queries can be set-up and how the procedure can be replicated to various queries and datasets

Dimensional discoveries: Unveiling the potential of 3d heritage point clouds with a robust ontology framework

3D point clouds feature valuable geometric and, often, radiometric and semantic information to support studies, analyses and understanding of the surveyed scene. Due to their generally large size, the use and interpretation of point clouds could be problematic. User-friendly and quick approaches for querying these valuable datasets and retrieving information could surely support end-users, in particular in the heritage sector. This work presents an ontology-based approach to facilitate the query and use of 3D heritage point clouds by means of sets of rules in order to infer properties and characteristics of the surveyed scene. Our approach is focused on linking together 3D spatial data and expert knowledge, in a way that the ontology can elaborate, represent, enrich and query a given point cloud. Results show how different queries can be set-up and how the procedure can be replicated to various queries and datasets

NERF FOR HERITAGE 3D RECONSTRUCTION

Conventional or learning-based 3D reconstruction methods from images have clearly shown their potential for 3D heritage documentation. Nevertheless, Neural Radiance Field (NeRF) approaches are recently revolutionising the way a scene can be rendered or reconstructed in 3D from a set of oriented images. Therefore the paper wants to review some of the last NeRF methods applied to various cultural heritage datasets collected with smartphone videos, touristic approaches or reflex cameras. Firstly several NeRF methods are evaluated. It turned out that Instant-NGP and Nerfacto methods achieved the best outcomes, outperforming all other methods significantly. Successively qualitative and quantitative analyses are performed on various datasets, revealing the good performances of NeRF methods, in particular for areas with uniform texture or shining surfaces, as well as for small datasets of lost artefacts. This is for sure opening new frontiers for 3D documentation, visualization and communication purposes of digital heritage

Combining image and point cloud segmentation to improve heritage understanding

Current 2D and 3D semantic segmentation frameworks are developed and trained on specific benchmark datasets, often rich of synthetic data, and when they are applied to complex and real-world heritage scenarios they offer much lower accuracy than expected. In this work, we present and demonstrate an early and late fusion of methods for semantic segmentation in cultural heritage applications. We rely on image datasets, point clouds and BIM models. The early fusion utilizes multi-view rendering to generate RGBD imagery of the scene. In contrast, the late fusion approach merges image-based segmentation with a Point Transformer applied to point clouds. Two scenarios are considered and inference results show that predictions are primarily influenced by whether the scene has a predominantly geometric or texture-based signature, underscoring the necessity of fusion methods

Combining image and point cloud segmentation to improve heritage understanding

Current 2D and 3D semantic segmentation frameworks are developed and trained on specific benchmark datasets, often rich of synthetic data, and when they are applied to complex and real-world heritage scenarios they offer much lower accuracy than expected. In this work, we present and demonstrate an early and late fusion of methods for semantic segmentation in cultural heritage applications. We rely on image datasets, point clouds and BIM models. The early fusion utilizes multi-view rendering to generate RGBD imagery of the scene. In contrast, the late fusion approach merges image-based segmentation with a Point Transformer applied to point clouds. Two scenarios are considered and inference results show that predictions are primarily influenced by whether the scene has a predominantly geometric or texture-based signature, underscoring the necessity of fusion methods

A Critical Analysis of NeRF-Based 3D Reconstruction

This paper presents a critical analysis of image-based 3D reconstruction using neural radiance fields (NeRFs), with a focus on quantitative comparisons with respect to traditional photogrammetry. The aim is, therefore, to objectively evaluate the strengths and weaknesses of NeRFs and provide insights into their applicability to different real-life scenarios, from small objects to heritage and industrial scenes. After a comprehensive overview of photogrammetry and NeRF methods, highlighting their respective advantages and disadvantages, various NeRF methods are compared using diverse objects with varying sizes and surface characteristics, including texture-less, metallic, translucent, and transparent surfaces. We evaluated the quality of the resulting 3D reconstructions using multiple criteria, such as noise level, geometric accuracy, and the number of required images (i.e., image baselines). The results show that NeRFs exhibit superior performance over photogrammetry in terms of non-collaborative objects with texture-less, reflective, and refractive surfaces. Conversely, photogrammetry outperforms NeRFs in cases where the object’s surface possesses cooperative texture. Such complementarity should be further exploited in future works

NERFBK: A HOLISTIC DATASET FOR BENCHMARKING NERF-BASED 3D RECONSTRUCTION

Neural Radiance Field methods are innovative solutions to derive 3D data from a set of oriented images. This paper introduces new real and synthetic image datasets - called NeRFBK - specifically designed for testing and comparing NeRF-based 3D reconstruction algorithms. More and more reconstruction algorithms and techniques are available nowadays, raising the need to evaluate and compare the quality of derived 3D products currently used in various domains and applications. However, gathering diverse data with precise ground truth is challenging and may not encompass all relevant applications. The NeRFBK dataset addresses this issue by providing multi-scale, indoor and outdoor datasets with high-resolution images and videos and camera parameters for testing and comparing NeRF-based algorithms. This paper presents the design and creation of the NeRFBK set of data, various examples and application scenarios, and highlights its potential for advancing the field of 3D reconstruction

Deep-image-matching: A toolbox for multiview image matching of complex scenarios

Finding corresponding points between images is a fundamental step in photogrammetry and computer vision tasks. Traditionally, image matching has relied on hand-crafted algorithms such as SIFT or ORB. However, these algorithms face challenges when dealing with multi-Temporal images, varying radiometry and contents as well as significant viewpoint differences. Recently, the computer vision community has proposed several deep learning-based approaches that are trained for challenging illumination and wide viewing angle scenarios. However, they suffer from certain limitations, such as rotations, and they are not applicable to high resolution images due to computational constraints. In addition, they are not widely used by the photogrammetric community due to limited integration with standard photogrammetric software packages. To overcome these challenges, this paper introduces Deep-Image-Matching, an opensource toolbox designed to match images using different matching strategies, ranging from traditional hand-crafted to deep-learning methods (https://github.com/3DOM-FBK/deep-image-matching). The toolbox accommodates high-resolution datasets, e.g. data acquired with full-frame or aerial sensors, and addresses known rotation-related problems of the learned features. The toolbox provides image correspondences outcomes that are directly compatible with commercial and open-source software packages, such as COLMAP and openMVG, for a bundle adjustment. The paper includes also a series of cultural heritage case studies that present challenging conditions where traditional hand-crafted approaches typically fail

Nerf for heritage 3d reconstruction

Conventional or learning-based 3D reconstruction methods from images have clearly shown their potential for 3D heritage documentation. Nevertheless, Neural Radiance Field (NeRF) approaches are recently revolutionising the way a scene can be rendered or reconstructed in 3D from a set of oriented images. Therefore the paper wants to review some of the last NeRF methods applied to various cultural heritage datasets collected with smartphone videos, touristic approaches or reflex cameras. Firstly several NeRF methods are evaluated. It turned out that Instant-NGP and Nerfacto methods achieved the best outcomes, outperforming all other methods significantly. Successively qualitative and quantitative analyses are performed on various datasets, revealing the good performances of NeRF methods, in particular for areas with uniform texture or shining surfaces, as well as for small datasets of lost artefacts. This is for sure opening new frontiers for 3D documentation, visualization and communication purposes of digital heritage

NERFBK: A HOLISTIC DATASET FOR BENCHMARKING NERF-BASED 3D RECONSTRUCTION

Neural Radiance Field methods are innovative solutions to derive 3D data from a set of oriented images. This paper introduces new real and synthetic image datasets - called NeRFBK - specifically designed for testing and comparing NeRF-based 3D reconstruction algorithms. More and more reconstruction algorithms and techniques are available nowadays, raising the need to evaluate and compare the quality of derived 3D products currently used in various domains and applications. However, gathering diverse data with precise ground truth is challenging and may not encompass all relevant applications. The NeRFBK dataset addresses this issue by providing multi-scale, indoor and outdoor datasets with high-resolution images and videos and camera parameters for testing and comparing NeRF-based algorithms. This paper presents the design and creation of the NeRFBK set of data, various examples and application scenarios, and highlights its potential for advancing the field of 3D reconstruction