A multimodal turn in Digital Humanities. Using contrastive machine learning models to explore, enrich, and analyze digital visual historical collections

Until recently, most research in the Digital Humanities (DH) was monomodal, meaning that the object of analysis was either textual or visual. Seeking to integrate multimodality theory into the DH, this article demonstrates that recently developed multimodal deep learning models, such as Contrastive Language Image Pre-training (CLIP), offer new possibilities to explore and analyze image–text combinations at scale. These models, which are trained on image and text pairs, can be applied to a wide range of text-to-image, image-to-image, and image-to-text prediction tasks. Moreover, multimodal models show high accuracy in zero-shot classification, i.e. predicting unseen categories across heterogeneous datasets. Based on three exploratory case studies, we argue that this zero-shot capability opens up the way for a multimodal turn in DH research. Moreover, multimodal models allow scholars to move past the artificial separation of text and images that was dominant in the field and analyze multimodal meaning at scale. However, we also need to be aware of the specific (historical) bias of multimodal deep learning that stems from biases in the training data used to train these models.</p

Large-scale interactive retrieval in art collections using multi-style feature aggregation

Finding objects and motifs across artworks is of great importance for art history as it helps to understand individual works and analyze relations between them. The advent of digitization has produced extensive digital art collections with many research opportunities. However, manual approaches are inadequate to handle this amount of data, and it requires appropriate computer-based methods to analyze them. This article presents a visual search algorithm and user interface to support art historians to find objects and motifs in extensive datasets. Artistic image collections are subject to significant domain shifts induced by large variations in styles, artistic media, and materials. This poses new challenges to most computer vision models which are trained on photographs. To alleviate this problem, we introduce a multi-style feature aggregation that projects images into the same distribution, leading to more accurate and style-invariant search results. Our retrieval system is based on a voting procedure combined with fast nearest-neighbor search and enables finding and localizing motifs within an extensive image collection in seconds. The presented approach significantly improves the state-of-the-art in terms of accuracy and search time on various datasets and applies to large and inhomogeneous collections. In addition to the search algorithm, we introduce a user interface that allows art historians to apply our algorithm in practice. The interface enables users to search for single regions, multiple regions regarding different connection types and holds an interactive feedback system to improve retrieval results further. With our methodological contribution and easy-to-use user interface, this work manifests further progress towards a computer-based analysis of visual art

Cultural analysis situs

The disciplines of complex network science, of art and cultural history, and of computation have a common ancestor in the 'analysis situs' of Gottfried Wilhelm Leibniz. Unfortunately, this shared conceptual origin remains hidden so far within a history of science that is tragically bifurcated, due to the branching evolution of disciplinary focus, due to changes in language, and due to sometimes forced scholarly migration. This chapter breaks the mutual tear lines of citation between disciplines to enable a common future. What lies at stake is the surprisingly deep-rooted and shared foundation of the emerging enterprise of a systematic science of art and culture. This enterprise currently flourishes mainly in departments of multidisciplinary information science, network and complexity science, and applications in industry. It promises nothing less than an integration of humanistic inquiry and a physics of cultures and cultural production

Expérience et explication de l’image à l’ère digitale: Une approche critique et utopique des tableaux

peer reviewedThat our relationship with art, both in the museum and in society, is currently undergoing radical transformation; that Art History in particular is destined to be revolutionized by the digital; and that the brand-new Digital Art History is poised to tell us and show us something other than what we have been able to see and know up until now: this can now seem seductively evident. This is then the starting point for the paper, which at the same time aims to return to some preliminary questions : What, on the one hand, is it to see and explain a painting better, and how do we go about it [§1]? How, on the other hand, can research linked to the digitization of images—coming well after the one on languages and texts processing—help us to enhance and sharpen our gaze [§2]? These are the issues addressed here in a frontal and critical manner. The article finds its initial impetus in a certain conception of what experiencing & explaining the picture means, in its fundamental knotting. It is argued that the enterprise of knowledge cannot be cut off from it: hence, by contrast, the gnoseological limits inherent in automatic processing, based on the scanning of large corpora. At the same time, certain key principles of analysis in Art History, Aesthetic theory and Visual Semiotics are reviewed and discussed: motifs (in a critical conception of traditional iconography), diagrams (in close relation to the notion of figurality), and the interweaving of the visual and the verbal (an essential and delicate issue in our relationship to the work of art, which Digital Humanities must by no means sidestep) [§3]. In the light of this assessment, the article proposes a series of original digital devices capable of better tying together, and enhancing, the experience and explanation of the painting [§4]. Drawing on some case studies of Gustave Courbet’s paintings, the aim is to combine the phenomenological demands of a singular encounter with each painting with those of an invention of semiotic value that is fundamentally linked to an active edition & montage of the artworks themselves. In short, we want to show the positive paths to a renewal in which machines could be an essential adjuvant, provided we do not stick to the research directions and to the intervention modalities we have seen celebrated and programmed to date.Que notre rapport à l’art, au musée comme dans la société, connaisse actuellement des transformations radicales ; que l’histoire de l’art en particulier soit destinée à être révolutionnée par le numérique ; et que la toute nouvelle Digital Art History soit en passe de nous dire et montrer autre chose que ce que nous avons pu voir et savoir jusqu’à maintenant : cela paraît désormais une évidence séduisante. Tel est le point de départ de cette étude, qui entend revenir dans le même temps à des questions liminaires : qu’est-ce, d’une part, que mieux voir et mieux expliquer un tableau, et comment s’y prend-on [§ 1] ? de quelle façon, d’autre part, les recherches liées à la numérisation des images — venant bien après celles portant sur le traitement automatique des langues et des textes — pourraient-elles nous aider à augmenter et affiner notre regard [§ 2] ? Tels sont les problèmes travaillés ici d’une manière frontale et critique. L’article trouve son élan initial dans une certaine conception de l’expérience et de l’explication du tableau, ressaisis dans leur nouage fondamental. On soutient que l’entreprise de connaissance ne peut en être coupée : d’où s’ensuivent, par contraste, les limites gnoséologiques propres aux traitements automatiques fondés sur le balayage de grands corpus. Corrélativement, certains principes d’analyse clés en histoire de l’art, théorie esthétique et sémiotique visuelle sont passés en revue et discutés : motifs (dans une conception critique de l’iconographie traditionnelle), diagrammes (en relation étroite avec la notion de figuralité), entrelacements du visuel et du verbal (enjeu essentiel et délicat de notre relation à l’œuvre d’art, que les humanités numériques ne doivent en aucun cas éluder) [§ 3]. À la lumière de ce bilan, l’article propose une série de dispositifs numériques originaux capables de mieux nouer, et augmenter, l’expérience et l’explication du tableau [§ 4]. En s’appuyant à quelques cas d’étude des toiles de Gustave Courbet, on vise finalement à conjuguer les exigences phénoménologiques de la rencontre singulière de chaque tableau avec celles d’une invention de la valeur sémiotique qui soit fondamentalement solidaire d’une mise en montage des œuvres. On voudrait montrer, en somme, les voies positives d’un renouvellement dont les machines pourraient être un adjuvant essentiel, à condition de ne pas en rester aux directions de recherche et aux modalités d’intervention que l’on a vu célébrées et programmées jusqu’à présent