Classification of geometric forms in mosaics using deep neural network

The paper addresses an image processing problem in the field of fine arts. In particular, a deep learning-based technique to classify geometric forms of artworks, such as paintings and mosaics, is presented. We proposed and tested a convolutional neural network (CNN)-based framework that autonomously quantifies the feature map and classifies it. Convolution, pooling and dense layers are three distinct categories of levels that generate attributes from the dataset images by introducing certain specified filters. As a case study, a Roman mosaic is considered, which is digitally reconstructed by close-range photogrammetry based on standard photos. During the digital transformation from a 2D perspective view of the mosaic into an orthophoto, each photo is rectified (i.e., it is an orthogonal projection of the real photo on the plane of the mosaic). Image samples of the geometric forms, e.g., triangles, squares, circles, octagons and leaves, even if they are partially deformed, were extracted from both the original and the rectified photos and originated the dataset for testing the CNN-based approach. The proposed method has proved to be robust enough to analyze the mosaic geometric forms, with an accuracy higher than 97%. Furthermore, the performance of the proposed method was compared with standard deep learning frameworks. Due to the promising results, this method can be applied to many other pattern identification problems related to artworks

A combined non-invasive approach to the study of a mosaic model: First laboratory experimental results

This paper presents first laboratory results of a combined approach carried out by the use of three different portable non-invasive electromagnetic methods: Digital holographic speckle pattern interferometry (DHSPI), stimulated infrared thermography (SIRT) and holographic subsurface radar (HSR), proposed for the analysis of a custom-built wall mosaic model. The model reproduces a series of defects (e.g., cracks, voids, detachments), simulating common deteriorated, restored or reshuffled areas in wall mosaics. DHSPI and SIRT, already well known in the field of non-destructive (NDT) methods, are full-field contactless techniques, providing complementary information on the subsurface hidden discontinuities. The use of DHSPI, based on optical imaging and interferometry, provides remote control and visualization of surface micro-deformation after induced thermal stress, while the use of SIRT allows visualization of thermal energy diffusion in the surface upon the induced thermal stress. DHSPI and SIRT data are complemented by the use of HSR, a contact method that provides localized information about the distribution of contrasts in dielectric permittivity and related possible anomalies. The experimental results, made by the combined use of these methods to the identification of the known anomalies in the mosaic model, are presented and discussed here as a contribution in the development of an efficient non-invasive approach to the in-situ subsurface analysis of ancient wall mosaics

Unmanned Aerial Vehicle Photography: Exploring the Medieval City of Merv, on the Silk Roads of Central Asia

The Ancient Merv Project is a collaboration between the Turkmenistan Ministry of Culture, the Ancient Merv State Park and the UCL Institute of Archaeology. It aims to research, protect and conserve the remains of one of the great historic cities of the Silk Roads. This paper explores a new survey of the Islamic city using an Unmanned Aerial Vehicle to take comprehensive and systematic vertical photographs to assist in the analysis of the medieval cityscape. The background to the research and the application of the technology are presented, together with our initial conclusions

Melite Civitas Romana in 3D: Virtualization Project of the Archaeological Park and Museum of the Domus Romana of Rabat, Malta

Abstract The archaeological site of the Domus Romana in Rabat, Malta was excavated almost 100 years ago yielding artefacts from the various phases of the site. The Melite Civitas Romana project was designed to investigate the domus, which may have been the home of a Roman Senator, and its many phases of use. Pending planned archaeological excavations designed to investigate the various phases of the site, a team from the Institute for Digital Exploration from the University of South Florida carried out a digitization campaign in the summer of 2019 using terrestrial laser scanning and aerial digital photogrammetry to document the current state of the site to provide a baseline of documentation and plan the coming excavations. In parallel, structured light scanning and photogrammetry were used to digitize 128 artefacts in the museum of the Domus Romana to aid in off-site research and create a virtual museum platform for global dissemination

Mo.Se.: Segmentación de mosaico de imágenes basado en aprendizaje profundo en cascada

[EN] Mosaic is an ancient type of art used to create decorative images or patterns combining small components. A digital version of a mosaic can be useful for archaeologists, scholars and restorers who are interested in studying, comparing and preserving mosaics. Nowadays, archaeologists base their studies mainly on manual operation and visual observation that, although still fundamental, should be supported by an automatized procedure of information extraction. In this context, this research explains improvements which can change the manual and time-consuming procedure of mosaic tesserae drawing. More specifically, this paper analyses the advantages of using Mo.Se. (Mosaic Segmentation), an algorithm that exploits deep learning and image segmentation techniques; the methodology combines U-Net 3 Network with the Watershed algorithm. The final purpose is to define a workflow which establishes the steps to perform a robust segmentation and obtain a digital (vector) representation of a mosaic. The detailed approach is presented, and theoretical justifications are provided, building various connections with other models, thus making the workflow both theoretically valuable and practically scalable for medium or large datasets. The automatic segmentation process was tested with the high-resolution orthoimage of an ancient mosaic by following a close-range photogrammetry procedure. Our approach has been tested in the pavement of St. Stephen's Church in Umm ar-Rasas, a Jordan archaeological site, located 30 km southeast of the city of Madaba (Jordan). Experimental results show that this generalized framework yields good performances, obtaining higher accuracy compared with other state-of-the-art approaches. Mo.Se. has been validated using publicly available datasets as a benchmark, demonstrating that the combination of learning-based methods with procedural ones enhances segmentation performance in terms of overall accuracy, which is almost 10% higher. This study’s ambitious aim is to provide archaeologists with a tool which accelerates their work of automatically extracting ancient geometric mosaics.Highlights:A Mo.Se. (Mosaic Segmentation) algorithm is described with the purpose to perform robust image segmentation to automatically detect tesserae in ancient mosaics.This research aims to overcome manual and time-consuming procedure of tesserae segmentation by proposing an approach that uses deep learning and image processing techniques, obtaining a digital replica of a mosaic.Extensive experiments show that the proposed framework outperforms state-of-the-art methods with higher accuracy, even compared with publicly available datasets.[ES] El mosaico es un tipo de arte antiguo utilizado para crear imágenes decorativas o patrones de pequeños componentes. Una versión digital de un mosaico puede ser útil a los arqueólogos, estudiosos y restauradores que están interesados en el estudio, la comparación y la preservación de los mosaicos. Hoy en día, los arqueólogos basan sus estudios principalmente en la operación manual y la observación visual que, aunque sigue siendo fundamental, debe ser apoyada con la ayuda de un procedimiento automatizado de extracción de la información. En este contexto, esta investigación tiene la intención de superar el procedimiento manual y lento del dibujo de teselas en mosaico proponiendo Mo.Se. (Mosaic Segmentation), un algoritmo que explota técnicas de aprendizaje profundo y segmentación de imagen; específicamente, la metodología combina la red U-Net 3 con el algoritmo Watershed. El propósito final es definir un flujo de trabajo que establezca los pasos para realizar una segmentación robusta y obtener una representación digital (vectorial) de un mosaico. Se presenta el procedimiento detallado y se proporcionan justificaciones teóricas, construyendo varias conexiones con otros modelos, haciendo que el flujo de trabajo sea teóricamente valioso y prácticamente escalable en conjuntos de datos medianos o grandes. El proceso de segmentación automática se probó con la ortoimagen de alta resolución de un mosaico antiguo, siguiendo un procedimiento de fotogrametría de objeto cercano. Nuestro enfoque se ha probado en el pavimento de la Iglesia de San Esteban en Umm ar-Rasas, un sitio arqueológico de Jordania, ubicado a 30 km al sureste de la ciudad de Madaba (Jordania). Los resultados experimentales muestran que este marco generalizado produce buenos rendimientos, obteniendo una mayor precisión en comparación con otros enfoques de vanguardia. Mo.Se. se ha validado utilizando conjuntos de datos disponibles públicamente como punto de referencia, lo que demuestra que la combinación de métodos basadosen el aprendizaje con métodos procedimentales mejora el rendimiento de la segmentación en casi un 10% en términos de exactitud en general. El ambicioso objetivo de este estudio es proporcionar a los arqueólogos una herramienta que acelere su trabajo de extracción automática de mosaicos geométricos antiguos.This work was partially found within the framework of the project Innovative technologies and training activities for the conservation and enhancement of the archaeological site of Umm er-Rasas (Jordan) funded by Ministero degli Affari Esteri e della Cooperazione Internazionale. The authors would like to express their gratitude to the ISPC CNR and in particular to Dott. Roberto Gabrielli (project leader) and Alessandra Albiero for providing the dataset.Felicetti, A.; Paolanti, M.; Zingaretti, P.; Pierdicca, R.; Malinverni, ES. (2021). Mo.Se.: Mosaic image segmentation based on deep cascading learning. Virtual Archaeology Review. 12(24):25-38. https://doi.org/10.4995/var.2021.14179OJS25381224Bartoli, A., Fenu, G., Medvet, E., Pellegrino, F. A., & Timeus, N. (2016, November). Segmentation of Mosaic Images Based on Deformable Models Using Genetic Algorithms. In International Conference on Smart Objects and Technologies for Social Good (pp. 233-242). Springer, Cham. https://doi.org/10.1007/978-3-319-61949-1_25Battiato, S., Di Blasi, G., Farinella, G. M., & Gallo, G. (2007, December). Digital mosaic frameworks‐an overview. In computer graphics forum (Vol. 26, No. 4, pp. 794-812). Oxford, UK: Blackwell Publishing Ltd. https://doi.org/10.1111/j.1467-8659.2007.01021.xBeucher, S., & Lantuéjoul, C. (1979). Use of watersheds in contour detection. International workshop on image processing: Real-time edge and motion detection/estimation. Rennes, France.Benyoussef, L., & Derrode, S. (2011). Analysis of ancient mosaic images for dedicated applications. Digital Imaging for Cultural Heritage Preservation: Analysis, Restoration, and Reconstruction of Ancient Artworks, 385.Bonfigli, R., Felicetti, A., Principi, E., Fagiani, M., Squartini, S., & Piazza, F. (2018). Denoising autoencoders for non-intrusive load monitoring: improvements and comparative evaluation. Energy and Buildings, 158. https://doi.org/10.1016/j.enbuild.2017.11.054Bordoni, L., & Mele, F. (Eds.). (2016). Artificial intelligence for cultural heritage. Cambridge Scholars Publishing.Bourke, P. (2014, December). Novel imaging of heritage objects and sites. In 2014 International Conference on Virtual Systems & Multimedia (VSMM) (pp. 25-30). IEEE. 10.1109/VSMM.2014.7136666Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., & Ronneberger, O. (2016, October). 3D U-Net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention (pp. 424-432). Springer, Cham. https://doi.org/10.1007/978-3-319-46723-8_49Cipriani, L., & Fantini, F. (2017). Digitalization culture VS archaeological visualization: integration of pipelines and open issues. The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 42, 195. https://doi.org/10.5194/isprs-archives-XLII-2-W3-195-2017Djibril, M. O., & Thami, R. O. H. (2008). Islamic geometrical patterns indexing and classification using discrete symmetry groups. Journal on Computing and Cultural Heritage (JOCCH), 1(2), 1-14. https://doi.org/10.1145/1434763.1434767Djibril, M. O., Thami, R. O. H., Benslimane, R., & Daoudi, M. (2005). Une nouvelle technique pour l'indexation des arabesques basée sur la dimension fractale. Univ. Mohamed V, Maroc.Falk, T., Mai, D., Bensch, R., Çiçek, Ö., Abdulkadir, A., Marrakchi, Y., Böhm, A., Deubner, J., Jäckel, Z., Seiwald, K., & Dovzhenko, A. (2019). U-Net: deep learning for cell counting, detection, and morphometry. Nature methods, 16(1), 67-70. https://doi.org/10.1038/s41592-018-0261-2Felicetti, A., Albiero, A., Gabrielli, R., Pierdicca, R., Paolanti, M., Zingaretti, P., & Malinverni, E. S. (2018). Automatic Mosaic Digitalization: a Deep Learning approach to tessera segmentation. In METROARCHEO, IEEE International Conference on Metrology for Archaeology and Cultural Heritage. Cassino. https://doi.org/10.1109/MetroArchaeo43810.2018.13606Fenu, G., Jain, N., Medvet, E., Pellegrino, F. A., & Namer, M. P. (2015, March). On the Assessment of Segmentation Methods for Images of Mosaics. In VISAPP (3) (pp. 130-137). https://doi.org/10.13140/RG.2.1.3025.6489Fenu, G., Medvet, E., Panfilo, D., & Pellegrino, F. A. (2020). Mosaic Images Segmentation using U-net. In International Conference on Pattern Recognition Applications and Methods (pp. 485-492). Scitepress. http://dx.doi.org/10.5220/0008967404850492Fontanella, F., Molinara, M., Gallozzi, A., Cigola, M., Senatore, L. J., Florio, R., Clini, P., & Celis, F. (2019, June). HeritageGO (HeGO) A Social Media Based Project for Cultural Heritage Valorization. In Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization (pp. 377-382). https://doi.org/10.1145/3314183.3323863Gil, F. A., Gomis, J. M., & Pérez, M. (2009). Reconstruction Techniques for Image Analysis of Ancient Islamic Mosaics. International Journal of Virtual Reality, 8(3), 5-12. https://doi.org/10.20870/IJVR.2009.8.3.2735Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.Kohl, S., Romera-Paredes, B., Meyer, C., De Fauw, J., Ledsam, J. R., Maier-Hein, K., Eslami, S.M.A, Rezende, D.J., & Ronneberger, O. (2018). A probabilistic u-net for segmentation of ambiguous images. In Advances in Neural Information Processing Systems (pp. 6965-6975). https://arxiv.org/abs/1806.05034Liciotti, D., Paolanti, M., Pietrini, R., Frontoni, E., & Zingaretti, P. (2018, August). Convolutional networks for semantic heads segmentation using top-view depth data in crowded environment. In 2018 24th international conference on pattern recognition (ICPR) IEEE. https://doi.org/10.1109/ICPR.2018.8545397Maghrebi, W., Ammar, A. B., Alimi, A. M., & Khabou, M. A. (2013). An Intelligent mutli-object retrieval system for historical mosaics. Editorial Preface, 4(4). https://doi.org/10.14569/IJACSA.2013.040417Maghrebi, W., Baccour, L., Khabou, M. A., & Alimi, A. M. (2007, November). An indexing and retrieval system of historic art images based on fuzzy shape similarity. In Mexican International Conference on Artificial Intelligence (pp. 623-633). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-76631-5_59Maghrebi, W., Borchani, A., Khabou, M. A., & Alimi, A. M. (2007, September). A system for historic document image indexing and retrieval based on xml database conforming to mpeg7 standard. In International Workshop on Graphics Recognition (pp. 114-125). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-88188-9_12Malinverni, E. S., Pierdicca, R., Di Stefano, F., Gabrielli, R., & Albiero, A. (2019). Virtual museum enriched by GIS data to share science and culture. Church of Saint Stephen in Umm Ar-Rasas (Jordan). Virtual Archaeology Review, 10(21). https://doi.org/10.4995/var.2019.11919M'hedhbi, M., Mezhoud, R., M'hiri, S., & Ghorbel, F. (2006, April). A new content-based image indexing and retrieval system of mosaic images. In 2006 2nd International Conference on Information & Communication Technologies (Vol. 1, pp. 1715-1719). IEEE. https://doi.org/10.1109/ICTTA.2006.1684644Pierdicca, R., Frontoni, E., Malinverni, E. S., Colosi, F., & Orazi, R. (2016). Virtual reconstruction of archaeological heritage using a combination of photogrammetric techniques: Huaca Arco Iris, Chan Chan, Peru. Digital Applications in Archaeology and Cultural Heritage, 3(3). https://doi.org/10.1016/j.daach.2016.06.002Pierdicca, R., Frontoni, E., Zingaretti, P., Malinverni, E. S., Colosi, F., & Orazi, R. (2015, August). Making visible the invisible. Augmented reality visualization for 3D reconstructions of archaeological sites. In International Conference on Augmented and Virtual Reality (Blinded for peer review). Springer, Cham. https://doi.org/10.1007/978-3-319-22888-4_3Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham. https://doi.org/10.1007/978-3-319-24574-4_28Vincent, L., & Soille, P. (1991). Watersheds in digital spaces: an efficient algorithm based on immersion simulations. IEEE Transactions on Pattern Analysis & Machine Intelligence, (6), 583-598. https://doi.org/10.1109/34.87344Youssef, L. B., & Derrode, S. (2008). Tessella-oriented segmentation and guidelines estimation of ancient mosaic images. Journal of Electronic Imaging, 17(4), 043014. https://doi.org/10.1117/1.3013543Zarghili, A., Gadi, N., Benslimane, R., & Bouatouch, K. (2001). Arabo-Moresque decor image retrieval system based on mosaic representations. Journal of Cultural Heritage, 2(2), 149-154. https://doi.org/10.1016/S1296-2074(01)01116-5Zarghili, A., Kharroubi, J., & Benslimane, R. (2008). Arabo-Moresque decor images retrieval system based on spatial relationships indexing. Journal of cultural heritage, 9(3), 317-325. https://doi.org/10.1016/j.culher.2007.10.008Zitová, B., Flusser, J., & Šroubek, F. (2004). An application of image processing in the medieval mosaic conservation. Pattern analysis and applications, 7(1), 18-25. https://doi.org/10.1007/s10044-003-0200-

Mobile spectroscopic instrumentation in archaeometry research

Mobile instrumentation is of growing importance to archaeometry research. Equipment is utilized in the field or at museums, thus avoiding transportation or risk of damage to valuable artifacts. Many spectroscopic techniques are nondestructive and micro-destructive in nature, which preserves the cultural heritage objects themselves. This review includes over 160 references pertaining to the use of mobile spectroscopy for archaeometry. Following a discussion of terminology related to mobile instrumental methods, results of a literature survey on their applications for cultural heritage objects is presented. Sections devoted to specific techniques are then provided: Raman spectroscopy, X-ray fluorescence spectrometry, Fourier transform infrared spectroscopy, laser-induced breakdown spectroscopy, and less frequently used techniques. The review closes with a discussion of combined instrumental approaches

Thermographic Imaging in Cultural Heritage: A Short Review

Over the recent period, there has been an increasing interest in the use of pulsed infrared thermography (PT) for the non-destructive evaluation of Cultural Heritage (CH). Unlike other techniques that are commonly employed in the same field, PT enables the depth-resolved detection of different kinds of subsurface features, thus providing helpful information for both scholars and restorers. Due to this reason, several research activities are currently underway to further improve the PT effectiveness. In this manuscript, the specific use of PT for the analysis of three different types of CH, namely documentary materials, panel paintings–marquetery, and mosaics, will be reviewed. In the latter case, i.e., mosaics, passive thermography combined with ground penetrating radar (GPR) and digital microscopy (DM) have also been deepened, considering their suitability in the open field. Such items have been selected because they are characterized by quite distinct physical and structural properties and, therefore, different PT (and, in some cases, verification) approaches have been employed for their investigations

Identity and connections within medieval heritage: color in the illuminated manuscript through the eyes of the molecular sciences and humanities

The characterization and identification of organic dyes is still a challenge within the field of Conservation Science. By exploring the potentialities of microspectrofluorimetry combined with chemometrics, this doctoral project provides the identification of red organic colorants and a comprehensive knowledge of the making of medieval paints. Microspectrofluorimetry (in the visible) allows the simultaneous acquisition of excitation and emission spectra, offering high sensitivity and selectivity combined with good spatial resolution and the possibility of in-depth profiling, which facilitates an accurate identification of dyes and lake pigments. Although lacking the fingerprint capability of vibrational spectra, it offers valuable knowledge into the paint formulation. Recipes’ specificities can provide insight into chronological and location particularities, such as scriptoria, enabling a better understanding of the making of the artists’ materials. The first part focuses on the development and testing of modeling strategies applied to i) a database of historically accurate reproductions of four natural red colorants namely brazilwood, cochineal, kermes and lac dye, used during the Middle Ages; ii) data from artworks, to address the difficulty of analyzing centuries old paints. The first confirmed the potential for microspectrofluorimetry in the assessment of the chromophore’s environment, i.e., the paint formulation, while the second explored the intricacies of the ‘original’ colors and the effectiveness of this methodology to explore similarities between naturally aged paints. This project proves the ability of microspectrofluorimetry as a powerful technique for the characterization of dyes and lake pigments. The historical reconstructions database allowed to pinpoint the main recipes of cochineal lake pigments from the 19th century Winsor & Newton’s database. The artworks database allowed to better understand recipe specificities and for the first time, we could pinpoint a formulation in which lac dye and brazilwood chromophores are admixed, in manuscripts from the Alcobaça scriptorium. In the second part, the methodology developed was tested in two case studies: the Ajuda Songbook and a group of Islamic manuscripts. The first, the oldest of the surviving Galician-Portuguese songbooks, is an unfinished illuminated manuscript, of which there is no knowledge of the circumstances of its production or the reason why it was never finished. The combination of a multi-analytical approach with the methodology developed in this project enabled the complete molecular characterization of the paint colors. It was shown the skillful construction of the paint layers and the richness of the chromatic palette, which demonstrates the desire and the resources to produce a luxurious manuscript. The methodology allowed to propose a production date for the Ajuda Songbook, in which the presence of brazilwood lake pigment and mosaic gold indicates a 14th century date, while the use of orpiment yellow pushes the date back into the 13th century. The second case study is a group of Islamic manuscripts (12th – 15th c.), from Timbuktu, Mali, which due to their rescue and conservation have allowed the study of their materials and techniques. For the first time, the richness and specificities of the paint formulations used were disclosed. It was possible not only to provide an unequivocal molecular characterization of the red colorant, lac dye but also to ascribe specific recipes to the different paint colors. Interestingly, the comparison with the artworks’ database has shown similarities with lac dye formulations found in Portuguese medieval illuminations. Moreover, the full characterization of the paint materials has also revealed severe degradation of the binding media. This approach will allow for better informed decision-making in the conservation process of these manuscripts. The synergy between the multi-analytical approach for the analysis of medieval manuscripts and the new methodology for the study of organic colorants was essential to the study of both case studies. The confocal spectrofluorimetry set-up used, as well as the expertise in the characterization of artworks, enabled in-depth knowledge into the construction of color paints, well beyond the identification of the single fluorophore

Chapter The Chapel of Sant’Agata in Pisa. 3D surveying, Artificial Intelligence and archival heritage

The 43rd UID conference, held in Genova, takes up the theme of ‘Dialogues’ as practice and debate on many fundamental topics in our social life, especially in these complex and not yet resolved times. The city of Genova offers the opportunity to ponder on the value of comparison and on the possibilities for the community, naturally focused on the aspects that concern us, as professors, researchers, disseminators of knowledge, or on all the possibile meanings of the discipline of representation and its dialogue with ‘others’, which we have broadly catalogued in three macro areas: History, Semiotics, Science / Technology. Therefore, “dialogue” as a profitable exchange based on a common language, without which it is impossible to comprehend and understand one another; and the graphic sign that connotes the conference is the precise transcription of this concept: the title ‘translated’ into signs, derived from the visual alphabet designed for the visual identity of the UID since 2017. There are many topics which refer to three macro sessions: - Witnessing (signs and history) - Communicating (signs and semiotics) - Experimenting (signs and sciences) Thanks to the different points of view, an exceptional resource of our disciplinary area, we want to try to outline the prevailing theoretical-operational synergies, the collaborative lines of an instrumental nature, the recent updates of the repertoires of images that attest and nourish the relations among representation, history, semiotics, sciences