Large-scale interactive retrieval in art collections using multi-style feature aggregation

Finding objects and motifs across artworks is of great importance for art history as it helps to understand individual works and analyze relations between them. The advent of digitization has produced extensive digital art collections with many research opportunities. However, manual approaches are inadequate to handle this amount of data, and it requires appropriate computer-based methods to analyze them. This article presents a visual search algorithm and user interface to support art historians to find objects and motifs in extensive datasets. Artistic image collections are subject to significant domain shifts induced by large variations in styles, artistic media, and materials. This poses new challenges to most computer vision models which are trained on photographs. To alleviate this problem, we introduce a multi-style feature aggregation that projects images into the same distribution, leading to more accurate and style-invariant search results. Our retrieval system is based on a voting procedure combined with fast nearest-neighbor search and enables finding and localizing motifs within an extensive image collection in seconds. The presented approach significantly improves the state-of-the-art in terms of accuracy and search time on various datasets and applies to large and inhomogeneous collections. In addition to the search algorithm, we introduce a user interface that allows art historians to apply our algorithm in practice. The interface enables users to search for single regions, multiple regions regarding different connection types and holds an interactive feedback system to improve retrieval results further. With our methodological contribution and easy-to-use user interface, this work manifests further progress towards a computer-based analysis of visual art

Introducing an Automated Pipeline for a Browser-based, City-scale Mobile 4D VR Application Based on Historical Images

The process for automatically creating 3D city models from contemporary photographs and visualizing them on mobile devices is now well established, but historical 4D city models are more challenging. The fourth dimension here is time. This contribution describes an automated VR pipeline based on historical photographs and resulting in an interactive browser-based device-rendered 4D visualization and information system for mobile devices. Since the pipeline shown is currently still under development, initial results for stages of the process will be shown and assessed for feasibility

Making large art historical photo archives searchable

In recent years, museums, archives and other cultural institutions have initiated important programs to digitize their collections. Millions of artefacts (paintings, engravings, drawings, ancient photographs) are now represented in digital photographic format. Furthermore, through progress in standardization, a growing portion of these images are now available online, in an easily accessible manner. This thesis studies how such large-scale art history collection can be made searchable using new deep learning approaches for processing and comparing images. It takes as a case study the processing of the photo archive of the Foundation Giorgio Cini, where more than 300'000 images have been digitized. We demonstrate how a generic processing pipeline can reliably extract the visual and textual content of scanned images, opening up ways to efficiently digitize large photo-collections. Then, by leveraging an annotated graph of visual connections, a metric is learnt that allows clustering and searching through artwork reproductions independently of their medium, effectively solving a difficult problem of cross-domain image search. Finally, the thesis studies how a complex Web Interface allows users to perform different searches based on this metric. We also evaluate the process by which users can annotate elements of interest during their navigation to be added to the database, allowing the system to be trained further and give better results. By documenting a complete approach on how to go from a physical photo-archive to a state-of-the-art navigation system, this thesis paves the way for a global search engine across the world's photo archives

CICHMKG: a large-scale and comprehensive Chinese intangible cultural heritage multimodal knowledge graph

Intangible Cultural Heritage (ICH) witnesses human creativity and wisdom in long histories, composed of a variety of immaterial manifestations. The rapid development of digital technologies accelerates the record of ICH, generating a sheer number of heterogenous data but in a state of fragmentation. To resolve that, existing studies mainly adopt approaches of knowledge graphs (KGs) which can provide rich knowledge representation. However, most KGs are text-based and text-derived, and incapable to give related images and empower downstream multimodal tasks, which is also unbeneficial for the public to establish the visual perception and comprehend ICH completely especially when they do not have the related ICH knowledge. Hence, aimed at that, we propose to, taking the Chinese nation-level ICH list as an example, construct a large-scale and comprehensive Multimodal Knowledge Graph (CICHMKG) combining text and image entities from multiple data sources and give a practical construction framework. Additionally, in this paper, to select representative images for ICH entities, we propose a method composed of the denoising algorithm (CNIFA) and a series of criteria, utilizing global and local visual features of images and textual features of captions. Extensive empirical experiments demonstrate its effectiveness. Lastly, we construct the CICHMKG, consisting of 1,774,005 triples, and visualize it to facilitate the interactions and help the public dive into ICH deeply

Fine Art Pattern Extraction and Recognition

This is a reprint of articles from the Special Issue published online in the open access journal Journal of Imaging (ISSN 2313-433X) (available at: https://www.mdpi.com/journal/jimaging/special issues/faper2020)

A 3D Digital Approach to the Stylistic and Typo-Technological Study of Small Figurines from Ayia Irini, Cyprus

The thesis aims to develop a 3D digital approach to the stylistic and typo-technological study of coroplastic, focusing on small figurines. The case study to test the method is a sample of terracotta statuettes from an assemblage of approximately 2000 statues and figurines found at the beginning of the 20th century in a rural open-air sanctuary at Ayia Irini (Cyprus) by the archaeologists of the Swedish Cyprus Expedition. The excavators identified continuity of worship at the sanctuary from the Late Cypriot III (circa 1200 BC) to the end of the Cypro-Archaic II period (ca. 475 BC). They attributed the small figurines to the Cypro-Archaic I-II. Although the excavation was one of the first performed through the newly established stratigraphic method, the archaeologists studied the site and its material following a traditional, merely qualitative approach. Theanalysis of the published results identified a classification of the material with no-clear-cut criteria, and their overlap between types highlights ambiguities in creating groups and classes. Similarly, stratigraphic arguments and different opinions among archaeologists highlight the need for revising. Moreover, pastlegislation allowed the excavators to export half of the excavated antiquities, creating a dispersion of the assemblage. Today, the assemblage is still partly exhibited at the Cyprus Museum in Nicosia and in four different museums in Sweden. Such a setting prevents to study, analyse and interpret the assemblageholistically. This research proposes a 3D chaîne opératoire methodology to study the collection’s small terracotta figurines, aiming to understand the context’s function and social role as reflected by the classification obtained with the 3D digital approach. The integration proposed in this research of traditional archaeological studies, and computer-assisted investigation based on quantitative criteria, identified and defined with 3D measurements and analytical investigations, is adopted as a solution to the biases of a solely qualitative approach. The 3D geometric analysis of the figurines focuses on the objects’ shape and components, mode of manufacture, level of expertise, specialisation or skills of the craftsman and production techniques. The analysis leads to the creation of classes of artefacts which allow archaeologists to formulate hypotheses on the production process, identify a common production (e.g., same hand, same workshop) and establish a relative chronological sequence. 3D reconstruction of the excavation’s area contributes to the virtual re-unification of the assemblage for its holistic study, the relative chronological dating of the figurines and the interpretation of their social and ritual purposes. The results obtained from the selected sample prove the efficacy of the proposed 3D approach and support the expansion of the analysis to the whole assemblage, and possibly initiate quantitative and systematic studies on Cypriot coroplastic production

Special Libraries, December 1974

Volume 65, Issue 12https://scholarworks.sjsu.edu/sla_sl_1974/1008/thumbnail.jp

Context-based multimedia semantics modelling and representation

The evolution of the World Wide Web, increase in processing power, and more network bandwidth have contributed to the proliferation of digital multimedia data. Since multimedia data has become a critical resource in many organisations, there is an increasing need to gain efficient access to data, in order to share, extract knowledge, and ultimately use the knowledge to inform business decisions. Existing methods for multimedia semantic understanding are limited to the computable low-level features; which raises the question of how to identify and represent the high-level semantic knowledge in multimedia resources.In order to bridge the semantic gap between multimedia low-level features and high-level human perception, this thesis seeks to identify the possible contextual dimensions in multimedia resources to help in semantic understanding and organisation. This thesis investigates the use of contextual knowledge to organise and represent the semantics of multimedia data aimed at efficient and effective multimedia content-based semantic retrieval.A mixed methods research approach incorporating both Design Science Research and Formal Methods for investigation and evaluation was adopted. A critical review of current approaches for multimedia semantic retrieval was undertaken and various shortcomings identified. The objectives for a solution were defined which led to the design, development, and formalisation of a context-based model for multimedia semantic understanding and organisation. The model relies on the identification of different contextual dimensions in multimedia resources to aggregate meaning and facilitate semantic representation, knowledge sharing and reuse. A prototype system for multimedia annotation, CONMAN was built to demonstrate aspects of the model and validate the research hypothesis, H₁.Towards providing richer and clearer semantic representation of multimedia content, the original contributions of this thesis to Information Science include: (a) a novel framework and formalised model for organising and representing the semantics of heterogeneous visual data; and (b) a novel S-Space model that is aimed at visual information semantic organisation and discovery, and forms the foundations for automatic video semantic understanding

Geographic Information Science (GIScience) and Geospatial Approaches for the Analysis of Historical Visual Sources and Cartographic Material

This book focuses on the use of GIScience in conjunction with historical visual sources to resolve past scenarios. The themes, knowledge gained and methodologies conducted might be of interest to a variety of scholars from the social science and humanities disciplines