Architectural heritage images classification using deep learning with CNN

© 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Digital documentation of cultural heritage images has emerged as an important topic in data analysis. Increasing the size and number of images to be processed making the task of categorizing them a challenging task and may take an inordinate amount of time. This research paper proposes a solution to the mentioned challenges by classifying the subject of the image of the study using Convolutional Neural Network. Classification of available images leads to improve the management of the images dataset and enhance the search of a specific item, which helps in the tasks of studying and analysis the proper heritage object. Deep learning for architectural heritage images classification has been employed during the course of this study. The pre-trained convolutional neural networks GoogLeNet, resnet18 and resnet50 proposed to be applied on public dataset Cultural Heritage images. Experimental results have shown promising outcomes with an accuracy of “87.91”, “95.47” and “95.57” respectively

Between images and built form: Automating the recognition of standardised building components using deep learning

Building on the richness of recent contributions in the field, this paper presents a state-of-the-art CNN analysis method for automatingthe recognition of standardised building components in modern heritage buildings. At the turn of the twentieth century manufacturedbuilding components became widely advertised for specification by architects. Consequently, a form of standardisation across varioustypologies began to take place. During this era of rapid economic and industrialised growth, many forms of public building wereerected. This paper seeks to demonstrate a method for informing the recognition of such elements using deep learning to recognise'families' of elements across a range of buildings in order to retrieve and recognise their technical specifications from the contemporarytrade literature. The method is illustrated through the case of Carnegie Public Libraries in the UK, which provides a unique butubiquitous platform from which to explore the potential for the automated recognition of manufactured standard architecturalcomponents. The aim of enhancing this knowledge base is to use the degree to which these were standardised originally as a means toinform and so support their ongoing care but also that of many other contemporary buildings. Although these libraries are numerous,they are maintained at a local level and as such, their shared challenges for maintenance remain unknown to one another. Additionally,this paper presents a methodology to indirectly retrieve useful indicators and semantics, relating to emerging HBIM families, byapplying deep learning to a varied range of architectural imagery

Application of Artificial Intelligence in Digital Architecture to Identify Traditional Javanese Buildings

Traditional buildings have a cultural philosophy and characterize the culture of an area. The occurrence of environmental changes, population growth, and the growth of modern buildings impact traditional buildings. Therefore, preserving those traditional buildings is needed to avoid extinction and make as cultural assets. The research aims to develop an application to help architects quantitatively measure the content of traditional architectural styles in their designs. This study uses the Artificial Intelligence (AI) method to identify buildings' similarities, acquiring traditional building data in roofs and ornaments images as a dataset totaling 650 images of roofs and 7,180 ornaments. Data processing was carried out by making architectural models, training, testing accuracy, and creating application interfaces. The algorithm used to identify similarities between building types was the Convolutional Naural Network (CNN) and the Support Vector Machine (SVM). The results of the accuracy-test using the Confusion matrix method reached an accuracy value of 99.5% in identifying building similarities and 85% in classifying building types

A BENCHMARK FOR LARGE-SCALE HERITAGE POINT CLOUD SEMANTIC SEGMENTATION

The lack of benchmarking data for the semantic segmentation of digital heritage scenarios is hampering the development of automatic classification solutions in this field. Heritage 3D data feature complex structures and uncommon classes that prevent the simple deployment of available methods developed in other fields and for other types of data. The semantic classification of heritage 3D data would support the community in better understanding and analysing digital twins, facilitate restoration and conservation work, etc. In this paper, we present the first benchmark with millions of manually labelled 3D points belonging to heritage scenarios, realised to facilitate the development, training, testing and evaluation of machine and deep learning methods and algorithms in the heritage field. The proposed benchmark, available at http://archdataset.polito.it/, comprises datasets and classification results for better comparisons and insights into the strengths and weaknesses of different machine and deep learning approaches for heritage point cloud semantic segmentation, in addition to promoting a form of crowdsourcing to enrich the already annotated databas

TOWARDS DEEP LEARNING FOR ARCHITECTURE: A MONUMENT RECOGNITION MOBILE APP

Abstract. In recent years, the diffusion of large image datasets and an unprecedented computational power have boosted the development of a class of artificial intelligence (AI) algorithms referred to as deep learning (DL). Among DL methods, convolutional neural networks (CNNs) have proven particularly effective in computer vision, finding applications in many disciplines. This paper introduces a project aimed at studying CNN techniques in the field of architectural heritage, a still to be developed research stream. The first steps and results in the development of a mobile app to recognize monuments are discussed. While AI is just beginning to interact with the built environment through mobile devices, heritage technologies have long been producing and exploring digital models and spatial archives. The interaction between DL algorithms and state-of-the-art information modeling is addressed, as an opportunity to both exploit heritage collections and optimize new object recognition techniques.</p

DEEP CONVOLUTIONAL NEURAL NETWORKS FOR SENTIMENT ANALYSIS OF CULTURAL HERITAGE

Abstract. The promotion of Cultural Heritage (CH) goods has become a major challenges over the last years. CH goods promote economic development, notably through cultural and creative industries and tourism. Thus, an effective planning of archaeological, cultural, artistic and architectural sites within the territory make CH goods easily accessible. A way of adding value to these services is making them capable of providing, using new technologies, a more immersive and stimulating fruition of information. In this light, an effective contribution can be provided by sentiment analysis. The sentiment related to a monument can be used for its evaluation considering that if it is positive, it influences its public image by increasing its value. This work introduces an approach to estimate the sentiment of Social Media pictures CH related. The sentiment of a picture is identified by an especially trained Deep Convolutional Neural Network (DCNN); aftewards, we compared the performance of three DCNNs: VGG16, ResNet and InceptionResNet. It is interesting to observe how these three different architectures are able to correctly evaluate the sentiment of an image referred to a ancient monument, historical buildings, archaeological sites, museum objects, and more. Our approach has been applied to a newly collected dataset of pictures from Instagram, which shows CH goods included in the UNESCO list of World Heritage properties.</p

Machine Learning and Deep Learning for the Built Heritage Analysis: Laser Scanning and UAV-Based Surveying Applications on a Complex Spatial Grid Structure

The reconstruction of 3D geometries starting from reality-based data is challenging and time-consuming due to the difficulties involved in modeling existing structures and the complex nature of built heritage. This paper presents a methodological approach for the automated segmentation and classification of surveying outputs to improve the interpretation and building information modeling from laser scanning and photogrammetric data. The research focused on the surveying of reticular, space grid structures of the late 19th–20th–21st centuries, as part of our architectural heritage, which might require monitoring maintenance activities, and relied on artificial intelligence (machine learning and deep learning) for: (i) the classification of 3D architectural components at multiple levels of detail and (ii) automated masking in standard photogrammetric processing. Focusing on the case study of the grid structure in steel named La Vela in Bologna, the work raises many critical issues in space grid structures in terms of data accuracy, geometric and spatial complexity, semantic classification, and component recognition

Surgical Phase Recognition of Short Video Shots Based on Temporal Modeling of Deep Features

Recognizing the phases of a laparoscopic surgery (LS) operation form its video constitutes a fundamental step for efficient content representation, indexing and retrieval in surgical video databases. In the literature, most techniques focus on phase segmentation of the entire LS video using hand-crafted visual features, instrument usage signals, and recently convolutional neural networks (CNNs). In this paper we address the problem of phase recognition of short video shots (10s) of the operation, without utilizing information about the preceding/forthcoming video frames, their phase labels or the instruments used. We investigate four state-of-the-art CNN architectures (Alexnet, VGG19, GoogleNet, and ResNet101), for feature extraction via transfer learning. Visual saliency was employed for selecting the most informative region of the image as input to the CNN. Video shot representation was based on two temporal pooling mechanisms. Most importantly, we investigate the role of 'elapsed time' (from the beginning of the operation), and we show that inclusion of this feature can increase performance dramatically (69% vs. 75% mean accuracy). Finally, a long short-term memory (LSTM) network was trained for video shot classification based on the fusion of CNN features with 'elapsed time', increasing the accuracy to 86%. Our results highlight the prominent role of visual saliency, long-range temporal recursion and 'elapsed time' (a feature so far ignored), for surgical phase recognition.Comment: 6 pages, 4 figures, 6 table