Deep learning for semantic segmentation of 3D point cloud.

Cultural Heritage is a testimony of past human activity, and, as such, its objects exhibit great variety in their nature, size and complexity; from small artefacts and museum items to cultural landscapes, from historical building and ancient monuments to city centers and archaeological sites. Cultural Heritage around the globe suffers from wars, natural disasters and human negligence. The importance of digital documentation is well recognized and there is an increasing pressure to document our heritage both nationally and internationally. For this reason, the three-dimensional scanning and modeling of sites and artifacts of cultural heritage have remarkably increased in recent years. The semantic segmentation of point clouds is an essential step of the entire pipeline; in fact, it allows to decompose complex architectures in single elements, which are then enriched with meaningful information within Building Information Modelling software. Notwithstanding, this step is very time consuming and completely entrusted on the manual work of domain experts, far from being automatized. This work describes a method to label and cluster automatically a point cloud based on a supervised Deep Learning approach, using a state-of-the-art Neural Network called PointNet++. Despite other methods are known, we have choose PointNet++ as it reached significant results for classifying and segmenting 3D point clouds. PointNet++ has been tested and improved, by training the network with annotated point clouds coming from a real survey and to evaluate how performance changes according to the input training data. It can result of great interest for the research community dealing with the point cloud semantic segmentation, since it makes public a labelled dataset of CH elements for further tests

PetroSurf3D - A Dataset for high-resolution 3D Surface Segmentation

The development of powerful 3D scanning hardware and reconstruction algorithms has strongly promoted the generation of 3D surface reconstructions in different domains. An area of special interest for such 3D reconstructions is the cultural heritage domain, where surface reconstructions are generated to digitally preserve historical artifacts. While reconstruction quality nowadays is sufficient in many cases, the robust analysis (e.g. segmentation, matching, and classification) of reconstructed 3D data is still an open topic. In this paper, we target the automatic and interactive segmentation of high-resolution 3D surface reconstructions from the archaeological domain. To foster research in this field, we introduce a fully annotated and publicly available large-scale 3D surface dataset including high-resolution meshes, depth maps and point clouds as a novel benchmark dataset to the community. We provide baseline results for our existing random forest-based approach and for the first time investigate segmentation with convolutional neural networks (CNNs) on the data. Results show that both approaches have complementary strengths and weaknesses and that the provided dataset represents a challenge for future research.Comment: CBMI Submission; Dataset and more information can be found at http://lrs.icg.tugraz.at/research/petroglyphsegmentation

Robots for Exploration, Digital Preservation and Visualization of Archeological Sites

Monitoring and conservation of archaeological sites are important activities necessary to prevent damage or to perform restoration on cultural heritage. Standard techniques, like mapping and digitizing, are typically used to document the status of such sites. While these task are normally accomplished manually by humans, this is not possible when dealing with hard-to-access areas. For example, due to the possibility of structural collapses, underground tunnels like catacombs are considered highly unstable environments. Moreover, they are full of radioactive gas radon that limits the presence of people only for few minutes. The progress recently made in the artificial intelligence and robotics field opened new possibilities for mobile robots to be used in locations where humans are not allowed to enter. The ROVINA project aims at developing autonomous mobile robots to make faster, cheaper and safer the monitoring of archaeological sites. ROVINA will be evaluated on the catacombs of Priscilla (in Rome) and S. Gennaro (in Naples)

3D Data Processing Toward Maintenance and Conservation. The Integrated Digital Documentation of Casa de Vidro

During the last decade, 3D integrated surveys and BIM modelling procedures have greatly improved the overall knowledge on some Brazilian Modernist buildings. In this framework, the Casa de Vidro 3D survey carried out by DIAPReM centre at Ferrara University, beside the important outputs, analysis and researches achieved from the point cloud database processing, was also useful to test several awareness increasing activities in cooperation with local stakeholders. The first digital documentation test of the Casa de Vidro allowed verifying the feasibility of a full survey on the building towards the restoration and possible placement of new architectures into the garden as an archive-museum of the Lina Bo and P.M. Bardi Foundation. Later, full 3D integrated survey and diagnostic analysis were carried out to achieve the total digital documentation of the house sponsored by the Keeping it Modern initiative of Getty Foundation (Los Angeles). Following its characteristics, the survey had to take into consideration the different architectural features, up to the relationship of architecture and nature. These 3D documentation activities and the point cloud processing allowed several analysis in a multidisciplinary framework

Transferencia de técnicas de aprendizaje y mejora del rendimiento en la segmentación semántica profunda de nubes de puntos del patrimonio construido

[EN] The growing availability of three-dimensional (3D) data, such as point clouds, coming from Light Detection and Ranging (LiDAR), Mobile Mapping Systems (MMSs) or Unmanned Aerial Vehicles (UAVs), provides the opportunity to rapidly generate 3D models to support the restoration, conservation, and safeguarding activities of cultural heritage (CH). The so-called scan-to-BIM process can, in fact, benefit from such data, and they can themselves be a source for further analyses or activities on the archaeological and built heritage. There are several ways to exploit this type of data, such as Historic Building Information Modelling (HBIM), mesh creation, rasterisation, classification, and semantic segmentation. The latter, referring to point clouds, is a trending topic not only in the CH domain but also in other fields like autonomous navigation, medicine or retail. Precisely in these sectors, the task of semantic segmentation has been mainly exploited and developed with artificial intelligence techniques. In particular, machine learning (ML) algorithms, and their deep learning (DL) subset, are increasingly applied and have established a solid state-of-the-art in the last half-decade. However, applications of DL techniques on heritage point clouds are still scarce; therefore, we propose to tackle this framework within the built heritage field. Starting from some previous tests with the Dynamic Graph Convolutional Neural Network (DGCNN), in this contribution close attention is paid to: i) the investigation of fine-tuned models, used as a transfer learning technique, ii) the combination of external classifiers, such as Random Forest (RF), with the artificial neural network, and iii) the evaluation of the data augmentation results for the domain-specific ArCH dataset. Finally, after taking into account the main advantages and criticalities, considerations are made on the possibility to profit by this methodology also for non-programming or domain experts.[ES] La creciente disponibilidad de datos tridimensionales (3D), como nubes de puntos, provenientes de la detección de la luz y distancia (LiDAR), sistemas de mapeado móvil (MMS) o vehículos aéreos no tripulados (UAV), brinda la oportunidad de generar rápidamente modelos 3D para apoyar las actividades de restauración, conservación y salvaguardia del patrimonio cultural (CH). El llamado proceso de escaneado-a-BIM puede, de hecho, beneficiarse de dichos datos, y ellos mismos pueden ser una fuente para futuros análisis o actividades sobre el patrimonio arqueológico y el construido. Hay varias formas de explotar este tipo de datos, como el modelado de información de edificios históricos (HBIM), la creación de mallas, la rasterización, la clasificación y la segmentación semántica. Este último, referido a las nubes de puntos, es un tema de máxima actualidad no solo en el dominio del PC sino también en otros campos como la navegación autónoma, la medicina o el comercio minorista. Precisamente en estos sectores, la tarea de la segmentación semántica se ha explotado y desarrollado principalmente con técnicas de inteligencia artificial. En particular, los algoritmos de aprendizaje automático (AA) y su subconjunto de aprendizaje profundo (AP) se aplican cada vez más y han establecido un sólido estado de la técnica en la última media década. Sin embargo, las aplicaciones de las técnicas de AP en las nubes de puntos tradicionales son todavía escasas; por tanto, nos proponemos abordar este marco dentro del ámbito del patrimonio construido. Partiendo de algunas pruebas anteriores con la Red Neural Convolucional de Gráfico Dinámico (DGCNN), en esta contribución se presta atención a: i) la investigación de modelos afinados, utilizados como técnica de aprendizaje por transferencia, ii) la combinación de clasificadores externos, como Random Forest (RF), con la red neuronal artificial, y iii) la evaluación de los resultados de aumentación de datos para el conjunto de datos específico del dominio ArCH. Finalmente, después de tener en cuenta las principales ventajas y criticidades, se hace una consideración sobre la posibilidad de beneficiarse de esta metodología también a expertos no programadores o del campo.Matrone, F.; Martini, M. (2021). Transfer learning and performance enhancement techniques for deep semantic segmentation of built heritage point clouds. Virtual Archaeology Review. 12(25):73-84. https://doi.org/10.4995/var.2021.15318OJS73841225Armeni, I., Sener, O., Zamir, A. R., Jiang, H., Brilakis, I., Fischer, M., & Savarese, S. (2016). 3D semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1534-1543. https://doi.org/10.1109/CVPR.2016.170Baraldi, L., Cornia, M., Grana, C., & Cucchiara, R. (2018). Aligning text and document illustrations: towards visually explainable digital humanities. In 24th International Conference on Pattern Recognition (ICPR), 1097-1102. IEEE. https://doi.org/10.1109/ICPR.2018.8545064Bassier, M., Yousefzadeh, M., & Vergauwen, M. (2020). Comparison of 2D and 3D wall reconstruction algorithms from point cloud data for as-built BIM. Journal of Information Technology in Construction (ITcon), 25(11), 173-192. https://doi.org/10.36680/j.itcon.2020.011Boulch, A., Guerry, J., Le Saux, B., & Audebert, N. (2018). SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks. Computers & Graphics, 71, 189-198. https://doi.org/10.1016/j.cag.2017.11.010Chadwick, J., (2020). Google launches hieroglyphics translator that uses AI to decipher images of Ancient Egyptian script. Available at https://www.dailymail.co.uk/sciencetech/article-8540329/Google-launches-hieroglyphics-translator-uses-AI-decipher-Ancient-Egyptian-script.html Last access 24/11/2020Fiorucci, M., Khoroshiltseva, M., Pontil, M., Traviglia, A., Del Bue, A., & James, S. (2020). Machine learning for cultural heritage: a survey. Pattern Recognition Letters, 133, 102-108. https://doi.org/10.1016/j.patrec.2020.02.017Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013). Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, 32(11), 1231-1237. https://doi.org/10.1177/0278364913491297Grilli, E., & Remondino, F. (2019). Classification of 3D digital heritage. Remote Sensing, 11(7), 847. https://doi.org/10.3390/rs11070847Grilli, E., & Remondino, F. (2020). Machine learning generalisation across different 3D architectural heritage. ISPRS International Journal of Geo-Information, 9(6), 379. https://doi.org/10.3390/ijgi9060379Grilli, E., Özdemir, E., & Remondino, F. (2019a). Application Of Machine And Deep Learning Strategies For The Classification Of Heritage Point Clouds. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-4/W18, 447-454, 2019. https://doi.org/10.5194/isprs-archives-XLII-4-W18-447-2019Grilli, E., Farella, E. M., Torresani, A., & Remondino, F. (2019b). Geometric features analysis for the classification of cultural heritage point clouds. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 541-548, 2019 https://doi.org/10.5194/isprs-archives-XLII-2-W15-541-2019Hackel, T., Savinov, N., Ladicky, L., Wegner, J. D., Schindler, K., & Pollefeys, M. (2017). Semantic3d.net: A new large-scale point cloud classification benchmark. arXiv:1704.03847He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770-778. arXiv:1512.03385Korc, F., & Förstner, W. (2009). eTRIMS Image Database for interpreting images of man-made scenes. Dept. of Photogrammetry, University of Bonn, Tech. Rep. TR-IGG-P-2009-01.Landrieu, L., & Simonovsky, M. (2018). Large-scale point cloud semantic segmentation with superpoint graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4558-4567. arXiv:1711.09869Llamas, J., M Lerones, P., Medina, R., Zalama, E., & Gómez-García-Bermejo, J. (2017). Classification of architectural heritage images using deep learning techniques. Applied Sciences, 7(10), 992. https://doi.org/10.3390/app7100992Mathias, M., Martinovic, A., Weissenberg, J., Haegler, S., & VanGool, L. (2011). Automatic architectural style recognition. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XXXVIII-5/W16, 171-176 3. https://doi.org/10.3390/app7100992Matrone, F., Grilli, E., Martini, M., Paolanti, M., Pierdicca, R., & Remondino, F. (2020a). Comparing machine and deep learning methods for large 3D heritage semantic segmentation. ISPRS International Journal of Geo-Information, 9(9), 535. https://doi.org/10.3390/ijgi9090535Matrone, F., Lingua, A., Pierdicca, R., Malinverni, E. S., Paolanti, M., Grilli, E., Remondino, F., Murtiyoso, A., & Landes, T. (2020b). A benchmark for large-scale heritage point cloud semantic segmentation. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLIII-B2-2020, 1419-1426. https://doi.org/10.5194/isprs-archives-XLIII-B2-2020-1419-2020Murtiyoso, A., & Grussenmeyer, P. (2019a). Automatic heritage building point cloud segmentation and classification using geometrical rules. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 821-827. https://doi.org/10.5194/isprs-archives-XLII-2-W15-821-2019Murtiyoso, A., & Grussenmeyer, P. (2019b). Point cloud segmentation and semantic annotation aided by GIS data for heritage complexes. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W9, 523-528, 2019. https://doi.org/10.5194/isprs-archives-XLII-2-W9-523-2019Oses, N., Dornaika, F., & Moujahid, A. (2014). Image-based delineation and classification of built heritage masonry. Remote Sensing, 6(3), 1863-1889. https://doi.org/10.3390/rs6031863Park, Y., & Guldmann, J. M. (2019). Creating 3D city models with building footprints and LIDAR point cloud classification: A machine learning approach. Computers, Environment and Urban Systems, 75, 76-89. https://doi.org/10.1016/j.compenvurbsys.2019.01.004Pierdicca, R., Paolanti, M., Matrone, F., Martini, M., Morbidoni, C., Malinverni, E. S. & Lingua, A. M. (2020). Point cloud semantic segmentation using a deep learning framework for cultural heritage. Remote Sensing, 12(6), 1005. https://doi.org/10.3390/rs12061005Qi, C. R., Su, H., Mo, K., & Guibas, L. J. (2017). Pointnet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 652-660. arXiv:1612.00593Sharafi, S., Fouladvand, S., Simpson, I., & Alvarez, J. A. B. (2016). Application of pattern recognition in detection of buried archaeological sites based on analysing environmental variables, Khorramabad Plain, West Iran. Journal of Archaeological Science: Reports, 8, 206-215. https://doi.org/10.1016/j.jasrep.2016.06.024Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556Stathopoulou, E. K., & Remondino, F. (2019). Semantic photogrammetry: boosting image-based 3D reconstruction with semantic labeling. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 42(2), W9. https://doi.org/10.5194/isprs-archives-XLII-2-W9-685-2019Teboul, O., Kokkinos, I., Simon, L., Koutsourakis, P., & Paragios, N. (2012). Parsing facades with shape grammars and reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 35(7), 1744-1756. https://doi.org/10.1109/TPAMI.2012.252.Teruggi, S., Grilli, E., Russo, M., Fassi, F., & Remondino, F. (2020). A hierarchical machine learning approach for multi-level and multi-resolution 3D point cloud classification. Remote Sensing, 12(16), 2598. https://doi.org/10.3390/rs12162598Tyleček, R., & Šára, R. (2013). Spatial pattern templates for recognition of objects with regular structure. In German Conference on Pattern Recognition, Springer, Berlin, Heidelberg, 364-374. https://doi.org/10.1007/978-3-642-40602-7_39Verschoof-van der Vaart, W. B., & Lambers, K. (2019). Learning to Look at LiDAR: the use of R-CNN in the automated detection of archaeological objects in LiDAR data from the Netherlands. Journal of Computer Applications in Archaeology, 2(1). https://doi.org/10.5334/jcaa.32Wang, Y., Sun, Y., Liu, Z., Sarma, S. E., Bronstein, M. M., &Solomon, J. M. (2019). Dynamic graph CNN for learning on point clouds. ACM Transactions On Graphics, 38(5), 1-12. arXiv:1801.07829Weinmann, M., Jutzi, B., Hinz, S., & Mallet, C. (2015). Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers. ISPRS Journal of Photogrammetry and Remote Sensing, 105, 286-304. https://doi.org/10.1016/j.isprsjprs.2015.01.016Xie, Y., Tian, J., & Zhu, X. X. (2019). Linking points with labels in 3D: a review of point cloud semantic segmentation. arXiv:1908.08854Yan, H., Ding, Y., Li, P., Wang, Q., Xu, Y., & Zuo, W. (2017). Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 2272-2281). arXiv:1705.00609 https://doi.org/10.1109/CVPR.2017.10

Volumetric models from 3D point clouds: The case study of sarcophagi cargo from a 2nd/3rd century AD Roman shipwreck near Sutivan on island Brac, Croatia

Multi-image photogrammetry can in favorable conditions even under water generate large clouds of 3D points which can be used for visualization of sunken heritage. For analysis of under-water archeological sites and comparison of artifacts, more compact shape models must be reconstructed from 3D points, where each object or a part of it is modeled individually. Volumetric models and superquadric models in particular are good candidates for such modeling since automated methods for their reconstruction and segmentation from 3D points exist. For the study case we use an underwater wreck site of a Roman ship from 2nd/3rd century AD located near Sutivan on island Brac in Croatia. We demonstrate how super- quadric models of sarcophagi and other stone blocks can be reconstructed from an unsegmented cloud of 3D points obtained by multi-image photogrammetry. We compare the dimensions of stone objects measured directly on the corresponding 3D point cloud with dimensions of the reconstructed super- quadric models and discuss other advantages of these volumetric models. The average difference be- tween point-to-point measurements of stone blocks and the dimensions of the corresponding superquadric model is on the order of few centimeters

THE RECOGNITION OF HERITAGE QUALITIES FROM FEATURE-BASED DIGITAL PROCEDURES IN THE ANALYSIS OF HISTORICAL URBAN CONTEXTS

Abstract. The documentation of historical architectural heritage in urban contexts involves the consideration of planning adaptations of settlements and landscape, related to the identification of formal and semantic qualities. In particular, the identification of cultural significance of Heritage building units can find correspondence in geometrical features that are documented within the urban asset. In this way, urban monitoring, in an increasingly automated way, can support the identification and characterization of semantic elements also regarding Heritage objects, observing the invariance and conservation of formal constants in urban dynamic assets.Considering the experimental case study of Solikamsk historical center, belonging to Upper Kama route (Russia), a multi-instrumental strategy of spatial survey is applied, evaluating data coverages and resolutions. This analysis defines a preliminary framework to develop further processes of 3D triangulation and reality-based meshing. The morpho-metric detail of final models constitutes the basis for the computing test of feature-based procedures, including regions recognition and mesh segmentation, which can be calibrated for shape qualities and scales, reaching a preliminary modeling classification of Heritage and urban building units

As-Built 3D Heritage City Modelling to Support Numerical Structural Analysis: Application to the Assessment of an Archaeological Remain

Terrestrial laser scanning is a widely used technology to digitise archaeological, architectural and cultural heritage. This allows for modelling the assets’ real condition in comparison with traditional data acquisition methods. This paper, based on the case study of the basilica in the Baelo Claudia archaeological ensemble (Tarifa, Spain), justifies the need of accurate heritage modelling against excessively simplified approaches in order to support structural safety analysis. To do this, after validating the 3Dmeshing process frompoint cloud data, the semi-automatic digital reconstitution of the basilica columns is performed. Next, a geometric analysis is conducted to calculate the structural alterations of the columns. In order to determine the structural performance, focusing both on the accuracy and suitability of the geometric models, static and modal analyses are carried out by means of the finite element method (FEM) on three different models for the most unfavourable column in terms of structural damage: (1) as-built (2) simplified and (3) ideal model without deformations. Finally, the outcomes show that the as-built modelling enhances the conservation status analysis of the 3D heritage city (in terms of realistic compliance factor values), although further automation still needs to be implemented in the modelling process

Mapping and classification of ecologically sensitive marine habitats using unmanned aerial vehicle (UAV) imagery and object-based image analysis (OBIA)

Nowadays, emerging technologies, such as long-range transmitters, increasingly miniaturized components for positioning, and enhanced imaging sensors, have led to an upsurge in the availability of new ecological applications for remote sensing based on unmanned aerial vehicles (UAVs), sometimes referred to as “drones”. In fact, structure-from-motion (SfM) photogrammetry coupled with imagery acquired by UAVs offers a rapid and inexpensive tool to produce high-resolution orthomosaics, giving ecologists a new way for responsive, timely, and cost-effective monitoring of ecological processes. Here, we adopted a lightweight quadcopter as an aerial survey tool and object-based image analysis (OBIA) workflow to demonstrate the strength of such methods in producing very high spatial resolution maps of sensitive marine habitats. Therefore, three different coastal environments were mapped using the autonomous flight capability of a lightweight UAV equipped with a fully stabilized consumer-grade RGB digital camera. In particular we investigated a Posidonia oceanica seagrass meadow, a rocky coast with nurseries for juvenile fish, and two sandy areas showing biogenic reefs of Sabelleria alveolata. We adopted, for the first time, UAV-based raster thematic maps of these key coastal habitats, produced after OBIA classification, as a new method for fine-scale, low-cost, and time saving characterization of sensitive marine environments which may lead to a more effective and efficient monitoring and management of natural resource