Mo.Se.: Segmentación de mosaico de imágenes basado en aprendizaje profundo en cascada

[EN] Mosaic is an ancient type of art used to create decorative images or patterns combining small components. A digital version of a mosaic can be useful for archaeologists, scholars and restorers who are interested in studying, comparing and preserving mosaics. Nowadays, archaeologists base their studies mainly on manual operation and visual observation that, although still fundamental, should be supported by an automatized procedure of information extraction. In this context, this research explains improvements which can change the manual and time-consuming procedure of mosaic tesserae drawing. More specifically, this paper analyses the advantages of using Mo.Se. (Mosaic Segmentation), an algorithm that exploits deep learning and image segmentation techniques; the methodology combines U-Net 3 Network with the Watershed algorithm. The final purpose is to define a workflow which establishes the steps to perform a robust segmentation and obtain a digital (vector) representation of a mosaic. The detailed approach is presented, and theoretical justifications are provided, building various connections with other models, thus making the workflow both theoretically valuable and practically scalable for medium or large datasets. The automatic segmentation process was tested with the high-resolution orthoimage of an ancient mosaic by following a close-range photogrammetry procedure. Our approach has been tested in the pavement of St. Stephen's Church in Umm ar-Rasas, a Jordan archaeological site, located 30 km southeast of the city of Madaba (Jordan). Experimental results show that this generalized framework yields good performances, obtaining higher accuracy compared with other state-of-the-art approaches. Mo.Se. has been validated using publicly available datasets as a benchmark, demonstrating that the combination of learning-based methods with procedural ones enhances segmentation performance in terms of overall accuracy, which is almost 10% higher. This study’s ambitious aim is to provide archaeologists with a tool which accelerates their work of automatically extracting ancient geometric mosaics.Highlights:A Mo.Se. (Mosaic Segmentation) algorithm is described with the purpose to perform robust image segmentation to automatically detect tesserae in ancient mosaics.This research aims to overcome manual and time-consuming procedure of tesserae segmentation by proposing an approach that uses deep learning and image processing techniques, obtaining a digital replica of a mosaic.Extensive experiments show that the proposed framework outperforms state-of-the-art methods with higher accuracy, even compared with publicly available datasets.[ES] El mosaico es un tipo de arte antiguo utilizado para crear imágenes decorativas o patrones de pequeños componentes. Una versión digital de un mosaico puede ser útil a los arqueólogos, estudiosos y restauradores que están interesados en el estudio, la comparación y la preservación de los mosaicos. Hoy en día, los arqueólogos basan sus estudios principalmente en la operación manual y la observación visual que, aunque sigue siendo fundamental, debe ser apoyada con la ayuda de un procedimiento automatizado de extracción de la información. En este contexto, esta investigación tiene la intención de superar el procedimiento manual y lento del dibujo de teselas en mosaico proponiendo Mo.Se. (Mosaic Segmentation), un algoritmo que explota técnicas de aprendizaje profundo y segmentación de imagen; específicamente, la metodología combina la red U-Net 3 con el algoritmo Watershed. El propósito final es definir un flujo de trabajo que establezca los pasos para realizar una segmentación robusta y obtener una representación digital (vectorial) de un mosaico. Se presenta el procedimiento detallado y se proporcionan justificaciones teóricas, construyendo varias conexiones con otros modelos, haciendo que el flujo de trabajo sea teóricamente valioso y prácticamente escalable en conjuntos de datos medianos o grandes. El proceso de segmentación automática se probó con la ortoimagen de alta resolución de un mosaico antiguo, siguiendo un procedimiento de fotogrametría de objeto cercano. Nuestro enfoque se ha probado en el pavimento de la Iglesia de San Esteban en Umm ar-Rasas, un sitio arqueológico de Jordania, ubicado a 30 km al sureste de la ciudad de Madaba (Jordania). Los resultados experimentales muestran que este marco generalizado produce buenos rendimientos, obteniendo una mayor precisión en comparación con otros enfoques de vanguardia. Mo.Se. se ha validado utilizando conjuntos de datos disponibles públicamente como punto de referencia, lo que demuestra que la combinación de métodos basadosen el aprendizaje con métodos procedimentales mejora el rendimiento de la segmentación en casi un 10% en términos de exactitud en general. El ambicioso objetivo de este estudio es proporcionar a los arqueólogos una herramienta que acelere su trabajo de extracción automática de mosaicos geométricos antiguos.This work was partially found within the framework of the project Innovative technologies and training activities for the conservation and enhancement of the archaeological site of Umm er-Rasas (Jordan) funded by Ministero degli Affari Esteri e della Cooperazione Internazionale. The authors would like to express their gratitude to the ISPC CNR and in particular to Dott. Roberto Gabrielli (project leader) and Alessandra Albiero for providing the dataset.Felicetti, A.; Paolanti, M.; Zingaretti, P.; Pierdicca, R.; Malinverni, ES. (2021). Mo.Se.: Mosaic image segmentation based on deep cascading learning. Virtual Archaeology Review. 12(24):25-38. https://doi.org/10.4995/var.2021.14179OJS25381224Bartoli, A., Fenu, G., Medvet, E., Pellegrino, F. A., & Timeus, N. (2016, November). Segmentation of Mosaic Images Based on Deformable Models Using Genetic Algorithms. In International Conference on Smart Objects and Technologies for Social Good (pp. 233-242). Springer, Cham. https://doi.org/10.1007/978-3-319-61949-1_25Battiato, S., Di Blasi, G., Farinella, G. M., & Gallo, G. (2007, December). Digital mosaic frameworks‐an overview. In computer graphics forum (Vol. 26, No. 4, pp. 794-812). Oxford, UK: Blackwell Publishing Ltd. https://doi.org/10.1111/j.1467-8659.2007.01021.xBeucher, S., & Lantuéjoul, C. (1979). Use of watersheds in contour detection. International workshop on image processing: Real-time edge and motion detection/estimation. Rennes, France.Benyoussef, L., & Derrode, S. (2011). Analysis of ancient mosaic images for dedicated applications. Digital Imaging for Cultural Heritage Preservation: Analysis, Restoration, and Reconstruction of Ancient Artworks, 385.Bonfigli, R., Felicetti, A., Principi, E., Fagiani, M., Squartini, S., & Piazza, F. (2018). Denoising autoencoders for non-intrusive load monitoring: improvements and comparative evaluation. Energy and Buildings, 158. https://doi.org/10.1016/j.enbuild.2017.11.054Bordoni, L., & Mele, F. (Eds.). (2016). Artificial intelligence for cultural heritage. Cambridge Scholars Publishing.Bourke, P. (2014, December). Novel imaging of heritage objects and sites. In 2014 International Conference on Virtual Systems & Multimedia (VSMM) (pp. 25-30). IEEE. 10.1109/VSMM.2014.7136666Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., & Ronneberger, O. (2016, October). 3D U-Net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention (pp. 424-432). Springer, Cham. https://doi.org/10.1007/978-3-319-46723-8_49Cipriani, L., & Fantini, F. (2017). Digitalization culture VS archaeological visualization: integration of pipelines and open issues. The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 42, 195. https://doi.org/10.5194/isprs-archives-XLII-2-W3-195-2017Djibril, M. O., & Thami, R. O. H. (2008). Islamic geometrical patterns indexing and classification using discrete symmetry groups. Journal on Computing and Cultural Heritage (JOCCH), 1(2), 1-14. https://doi.org/10.1145/1434763.1434767Djibril, M. O., Thami, R. O. H., Benslimane, R., & Daoudi, M. (2005). Une nouvelle technique pour l'indexation des arabesques basée sur la dimension fractale. Univ. Mohamed V, Maroc.Falk, T., Mai, D., Bensch, R., Çiçek, Ö., Abdulkadir, A., Marrakchi, Y., Böhm, A., Deubner, J., Jäckel, Z., Seiwald, K., & Dovzhenko, A. (2019). U-Net: deep learning for cell counting, detection, and morphometry. Nature methods, 16(1), 67-70. https://doi.org/10.1038/s41592-018-0261-2Felicetti, A., Albiero, A., Gabrielli, R., Pierdicca, R., Paolanti, M., Zingaretti, P., & Malinverni, E. S. (2018). Automatic Mosaic Digitalization: a Deep Learning approach to tessera segmentation. In METROARCHEO, IEEE International Conference on Metrology for Archaeology and Cultural Heritage. Cassino. https://doi.org/10.1109/MetroArchaeo43810.2018.13606Fenu, G., Jain, N., Medvet, E., Pellegrino, F. A., & Namer, M. P. (2015, March). On the Assessment of Segmentation Methods for Images of Mosaics. In VISAPP (3) (pp. 130-137). https://doi.org/10.13140/RG.2.1.3025.6489Fenu, G., Medvet, E., Panfilo, D., & Pellegrino, F. A. (2020). Mosaic Images Segmentation using U-net. In International Conference on Pattern Recognition Applications and Methods (pp. 485-492). Scitepress. http://dx.doi.org/10.5220/0008967404850492Fontanella, F., Molinara, M., Gallozzi, A., Cigola, M., Senatore, L. J., Florio, R., Clini, P., & Celis, F. (2019, June). HeritageGO (HeGO) A Social Media Based Project for Cultural Heritage Valorization. In Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization (pp. 377-382). https://doi.org/10.1145/3314183.3323863Gil, F. A., Gomis, J. M., & Pérez, M. (2009). Reconstruction Techniques for Image Analysis of Ancient Islamic Mosaics. International Journal of Virtual Reality, 8(3), 5-12. https://doi.org/10.20870/IJVR.2009.8.3.2735Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.Kohl, S., Romera-Paredes, B., Meyer, C., De Fauw, J., Ledsam, J. R., Maier-Hein, K., Eslami, S.M.A, Rezende, D.J., & Ronneberger, O. (2018). A probabilistic u-net for segmentation of ambiguous images. In Advances in Neural Information Processing Systems (pp. 6965-6975). https://arxiv.org/abs/1806.05034Liciotti, D., Paolanti, M., Pietrini, R., Frontoni, E., & Zingaretti, P. (2018, August). Convolutional networks for semantic heads segmentation using top-view depth data in crowded environment. In 2018 24th international conference on pattern recognition (ICPR) IEEE. https://doi.org/10.1109/ICPR.2018.8545397Maghrebi, W., Ammar, A. B., Alimi, A. M., & Khabou, M. A. (2013). An Intelligent mutli-object retrieval system for historical mosaics. Editorial Preface, 4(4). https://doi.org/10.14569/IJACSA.2013.040417Maghrebi, W., Baccour, L., Khabou, M. A., & Alimi, A. M. (2007, November). An indexing and retrieval system of historic art images based on fuzzy shape similarity. In Mexican International Conference on Artificial Intelligence (pp. 623-633). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-76631-5_59Maghrebi, W., Borchani, A., Khabou, M. A., & Alimi, A. M. (2007, September). A system for historic document image indexing and retrieval based on xml database conforming to mpeg7 standard. In International Workshop on Graphics Recognition (pp. 114-125). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-88188-9_12Malinverni, E. S., Pierdicca, R., Di Stefano, F., Gabrielli, R., & Albiero, A. (2019). Virtual museum enriched by GIS data to share science and culture. Church of Saint Stephen in Umm Ar-Rasas (Jordan). Virtual Archaeology Review, 10(21). https://doi.org/10.4995/var.2019.11919M'hedhbi, M., Mezhoud, R., M'hiri, S., & Ghorbel, F. (2006, April). A new content-based image indexing and retrieval system of mosaic images. In 2006 2nd International Conference on Information & Communication Technologies (Vol. 1, pp. 1715-1719). IEEE. https://doi.org/10.1109/ICTTA.2006.1684644Pierdicca, R., Frontoni, E., Malinverni, E. S., Colosi, F., & Orazi, R. (2016). Virtual reconstruction of archaeological heritage using a combination of photogrammetric techniques: Huaca Arco Iris, Chan Chan, Peru. Digital Applications in Archaeology and Cultural Heritage, 3(3). https://doi.org/10.1016/j.daach.2016.06.002Pierdicca, R., Frontoni, E., Zingaretti, P., Malinverni, E. S., Colosi, F., & Orazi, R. (2015, August). Making visible the invisible. Augmented reality visualization for 3D reconstructions of archaeological sites. In International Conference on Augmented and Virtual Reality (Blinded for peer review). Springer, Cham. https://doi.org/10.1007/978-3-319-22888-4_3Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham. https://doi.org/10.1007/978-3-319-24574-4_28Vincent, L., & Soille, P. (1991). Watersheds in digital spaces: an efficient algorithm based on immersion simulations. IEEE Transactions on Pattern Analysis & Machine Intelligence, (6), 583-598. https://doi.org/10.1109/34.87344Youssef, L. B., & Derrode, S. (2008). Tessella-oriented segmentation and guidelines estimation of ancient mosaic images. Journal of Electronic Imaging, 17(4), 043014. https://doi.org/10.1117/1.3013543Zarghili, A., Gadi, N., Benslimane, R., & Bouatouch, K. (2001). Arabo-Moresque decor image retrieval system based on mosaic representations. Journal of Cultural Heritage, 2(2), 149-154. https://doi.org/10.1016/S1296-2074(01)01116-5Zarghili, A., Kharroubi, J., & Benslimane, R. (2008). Arabo-Moresque decor images retrieval system based on spatial relationships indexing. Journal of cultural heritage, 9(3), 317-325. https://doi.org/10.1016/j.culher.2007.10.008Zitová, B., Flusser, J., & Šroubek, F. (2004). An application of image processing in the medieval mosaic conservation. Pattern analysis and applications, 7(1), 18-25. https://doi.org/10.1007/s10044-003-0200-

Image registration and visualization of in situ gene expression images.

In the age of high-throughput molecular biology techniques, scientists have incorporated the methodology of in-situ hybridization to map spatial patterns of gene expression. In order to compare expression patterns within a common tissue structure, these images need to be registered or organized into a common coordinate system for alignment to a reference or atlas images. We use three different image registration methodologies (manual; correlation based; mutual information based) to determine the common coordinate system for the reference and in-situ hybridization images. All three methodologies are incorporated into a Matlab tool to visualize the results in a user friendly way and save them for future work. Our results suggest that the user-defined landmark method is best when considering images from different modalities; automated landmark detection is best when the images are expected to have a high degree of consistency; and the mutual information methodology is useful when the images are from the same modality

TESSERAE3D : Une comparaison pour la segmentation sémantique des tesselles dans les nuages de points 3D

3D point cloud of mosaic tesserae is used by heritage researchers, restorers, and archaeologists for digital investigations. Information extraction, pattern analysis, and semantic assignment are necessary to complement geometric information. Automated processes that can speed up the task are highly sought after, especially new supervised approaches. However, the availability of labeled data necessary for training supervised learning models is a significant constraint. This paper introduces Tesserae3D, a 3D point cloud benchmark dataset for training and evaluating machine learning models, applied to mosaic tesserae segmentation. It is a publicly available, very high density and colored dataset, accompanied by a standard multi-class semantic segmentation baseline. It consists of about 502 million points and contains 11 semantic classes covering a wide range of tesserae types. We propose a semantic segmentation baseline building on radiometric and covariance features fed to ensemble learning methods. The results delineate an achievable 89% F1 score and are made available under https://github.com/akharroubi/Tesserae3D, providing a simple interface to improve the score based on feedback from the research community

ACTIVE: Towards Highly Transferable 3D Physical Camouflage for Universal and Robust Vehicle Evasion

Adversarial camouflage has garnered attention for its ability to attack object detectors from any viewpoint by covering the entire object's surface. However, universality and robustness in existing methods often fall short as the transferability aspect is often overlooked, thus restricting their application only to a specific target with limited performance. To address these challenges, we present Adversarial Camouflage for Transferable and Intensive Vehicle Evasion (ACTIVE), a state-of-the-art physical camouflage attack framework designed to generate universal and robust adversarial camouflage capable of concealing any 3D vehicle from detectors. Our framework incorporates innovative techniques to enhance universality and robustness, including a refined texture rendering that enables common texture application to different vehicles without being constrained to a specific texture map, a novel stealth loss that renders the vehicle undetectable, and a smooth and camouflage loss to enhance the naturalness of the adversarial camouflage. Our extensive experiments on 15 different models show that ACTIVE consistently outperforms existing works on various public detectors, including the latest YOLOv7. Notably, our universality evaluations reveal promising transferability to other vehicle classes, tasks (segmentation models), and the real world, not just other vehicles.Comment: Accepted for ICCV 2023. Main Paper with Supplementary Material. Project Page: https://islab-ai.github.io/active-iccv2023

Astigmatism and Pseudoaccommodation in Pseudophakic Eyes

noAdvanced IOLs with circumferential zones of different power provide pseudoaccommodation. We investigated the potential for power variation with meridian, namely astigmatism, to provide pseudo-accommodation. With appropriate power and axis orientations, acceptable pseudo-accommodation can be achieved

Endoscopic image analysis of aberrant crypt foci

Tese de Mestrado Integrado. Bioengenharia. Faculdade de Engenharia. Universidade do Porto. 201