379 research outputs found
A Data Set and a Convolutional Model for Iconography Classification in Paintings
Iconography in art is the discipline that studies the visual content of
artworks to determine their motifs and themes andto characterize the way these
are represented. It is a subject of active research for a variety of purposes,
including the interpretation of meaning, the investigation of the origin and
diffusion in time and space of representations, and the study of influences
across artists and art works. With the proliferation of digital archives of art
images, the possibility arises of applying Computer Vision techniques to the
analysis of art images at an unprecedented scale, which may support iconography
research and education. In this paper we introduce a novel paintings data set
for iconography classification and present the quantitativeand qualitative
results of applying a Convolutional Neural Network (CNN) classifier to the
recognition of the iconography of artworks. The proposed classifier achieves
good performances (71.17% Precision, 70.89% Recall, 70.25% F1-Score and 72.73%
Average Precision) in the task of identifying saints in Christian religious
paintings, a task made difficult by the presence of classes with very similar
visual features. Qualitative analysis of the results shows that the CNN focuses
on the traditional iconic motifs that characterize the representation of each
saint and exploits such hints to attain correct identification. The ultimate
goal of our work is to enable the automatic extraction, decomposition, and
comparison of iconography elements to support iconographic studies and
automatic art work annotation.Comment: Published at ACM Journal on Computing and Cultural Heritage (JOCCH)
https://doi.org/10.1145/345888
DEArt: Dataset of European Art
Large datasets that were made publicly available to the research community
over the last 20 years have been a key enabling factor for the advances in deep
learning algorithms for NLP or computer vision. These datasets are generally
pairs of aligned image / manually annotated metadata, where images are
photographs of everyday life. Scholarly and historical content, on the other
hand, treat subjects that are not necessarily popular to a general audience,
they may not always contain a large number of data points, and new data may be
difficult or impossible to collect. Some exceptions do exist, for instance,
scientific or health data, but this is not the case for cultural heritage (CH).
The poor performance of the best models in computer vision - when tested over
artworks - coupled with the lack of extensively annotated datasets for CH, and
the fact that artwork images depict objects and actions not captured by
photographs, indicate that a CH-specific dataset would be highly valuable for
this community. We propose DEArt, at this point primarily an object detection
and pose classification dataset meant to be a reference for paintings between
the XIIth and the XVIIIth centuries. It contains more than 15000 images, about
80% non-iconic, aligned with manual annotations for the bounding boxes
identifying all instances of 69 classes as well as 12 possible poses for boxes
identifying human-like objects. Of these, more than 50 classes are CH-specific
and thus do not appear in other datasets; these reflect imaginary beings,
symbolic entities and other categories related to art. Additionally, existing
datasets do not include pose annotations. Our results show that object
detectors for the cultural heritage domain can achieve a level of precision
comparable to state-of-art models for generic images via transfer learning.Comment: VISART VI. Workshop at the European Conference of Computer Vision
(ECCV
Machine Learning for Cultural Heritage: A Survey
The application of Machine Learning (ML) to Cultural Heritage (CH) has evolved since basic statistical approaches such as Linear Regression to complex Deep Learning models. The question remains how much of this actively improves on the underlying algorithm versus using it within a ‘black box’ setting. We survey across ML and CH literature to identify the theoretical changes which contribute to the algorithm and in turn them suitable for CH applications. Alternatively, and most commonly, when there are no changes, we review the CH applications, features and pre/post-processing which make the algorithm suitable for its use. We analyse the dominant divides within ML, Supervised, Semi-supervised and Unsupervised, and reflect on a variety of algorithms that have been extensively used. From such an analysis, we give a critical look at the use of ML in CH and consider why CH has only limited adoption of ML
Deep learning approaches to pattern extraction and recognition in paintings and drawings: an overview
This paper provides an overview of some of the most relevant deep learning approaches to pattern extraction and recognition in visual arts, particularly painting and drawing. Recent advances in deep learning and computer vision, coupled with the growing availability of large digitized visual art collections, have opened new opportunities for computer science researchers to assist the art community with automatic tools to analyse and further understand visual arts. Among other benefits, a deeper understanding of visual arts has the potential to make them more accessible to a wider population, ultimately supporting the spread of culture
Linking Art through Human Poses
We address the discovery of composition transfer in artworks based on their
visual content. Automated analysis of large art collections, which are growing
as a result of art digitization among museums and galleries, is an important
tool for art history and assists cultural heritage preservation. Modern image
retrieval systems offer good performance on visually similar artworks, but fail
in the cases of more abstract composition transfer. The proposed approach links
artworks through a pose similarity of human figures depicted in images. Human
figures are the subject of a large fraction of visual art from middle ages to
modernity and their distinctive poses were often a source of inspiration among
artists. The method consists of two steps -- fast pose matching and robust
spatial verification. We experimentally show that explicit human pose matching
is superior to standard content-based image retrieval methods on a manually
annotated art composition transfer dataset
A deep learning approach to clustering visual arts
Clustering artworks is difficult for several reasons. On the one hand,
recognizing meaningful patterns based on domain knowledge and visual perception
is extremely hard. On the other hand, applying traditional clustering and
feature reduction techniques to the highly dimensional pixel space can be
ineffective. To address these issues, in this paper we propose DELIUS: a DEep
learning approach to cLustering vIsUal artS. The method uses a pre-trained
convolutional network to extract features and then feeds these features into a
deep embedded clustering model, where the task of mapping the raw input data to
a latent space is jointly optimized with the task of finding a set of cluster
centroids in this latent space. Quantitative and qualitative experimental
results show the effectiveness of the proposed method. DELIUS can be useful for
several tasks related to art analysis, in particular visual link retrieval and
historical knowledge discovery in painting datasets.Comment: Submitted to IJC
Fine Art Pattern Extraction and Recognition
This is a reprint of articles from the Special Issue published online in the open access journal Journal of Imaging (ISSN 2313-433X) (available at: https://www.mdpi.com/journal/jimaging/special issues/faper2020)
Deep convolutional embedding for digitized painting clustering
Clustering artworks is difficult because of several reasons. On one hand,
recognizing meaningful patterns in accordance with domain knowledge and visual
perception is extremely hard. On the other hand, the application of traditional
clustering and feature reduction techniques to the highly dimensional pixel
space can be ineffective. To address these issues, we propose a deep
convolutional embedding model for clustering digital paintings, in which the
task of mapping the input raw data to an abstract, latent space is optimized
jointly with the task of finding a set of cluster centroids in this latent
feature space. Quantitative and qualitative experimental results show the
effectiveness of the proposed method. The model is also able to outperform
other state-of-the-art deep clustering approaches to the same problem. The
proposed method may be beneficial to several art-related tasks, particularly
visual link retrieval and historical knowledge discovery in painting datasets
- …