Virtual Cleaning of Works of Art Using Deep Learning Based Approaches

Virtual cleaning of art is a key process that conservators apply to see the likely appearance of the work of art they have aimed to clean, before the process of cleaning. There have been many different approaches to virtually clean artworks but having to physically clean the artwork at a few specific places of specific colors, the need to have pure black and white paint on the painting and their low accuracy are only a few of their shortcomings prompting us to propose deep learning based approaches in this research. First we report the work we have done in this field focusing on the color estimation of the artwork virtual cleaning and then we describe our methods for the spectral reflectance estimation of artwork in virtual cleaning. In the color estimation part, a deep convolutional neural network (CNN) and a deep generative network (DGN) are suggested, which estimate the RGB image of the cleaned artwork from an RGB image of the uncleaned artwork. Applying the networks to the images of the well-known artworks (such as the Mona Lisa and The Virgin and Child with Saint Anne) and Macbeth ColorChecker and comparing the results to the only physics-based model (which is the first model that has approached the issue of virtual cleaning from the physics-point of view, hence our reference to compare our models with) shows that our methods outperform that model and have great potentials of being applied to the real situations in which there might not be much information available on the painting, and all we have is an RGB image of the uncleaned artwork. Nonetheless, the methods proposed in the first part, cannot provide us with the spectral reflectance information of the artwork, therefore, the second part of the dissertation is proposed. This part focuses on the spectral estimation of the artwork virtual cleaning. Two deep learning-based approaches are also proposed here; the first one is deep generative network. This method receives a cube of the hyperspectral image of the uncleaned artwork and tries to output another cube which is the virtually cleaned hyperspectral image of the artwork. The second approach is 1D Convolutional Autoencoder (1DCA), which is based on 1D convolutional neural network and tries to find the spectra of the virtually cleaned artwork using the spectra of the physically cleaned artworks and their corresponding uncleaned spectra. The approaches are applied to hyperspectral images of Macbeth ColorChecker (simulated in the forms of cleaned and uncleaned hyperspectral images) and the \u27Haymakers\u27 (real hyperspectral images of both cleaned and uncleaned states). The results, in terms of Euclidean distance and spectral angle between the virtually cleaned artwork and the physically cleaned one, show that the proposed approaches have outperformed the physics-based model, with DGN outperforming the 1DCA. Methods proposed herein do not rely on finding a specific type of paint and color on the painting first and take advantage of the high accuracy offered by deep learning-based approaches and they are also applicable to other paintings

El lienzo inmersivo. Diseño y desarrollo de una experiencia pictórica en una realidad virtual.

[ES] Uno de los objetivos finales de una obra artística es, sin lugar a dudas, despertar emociones en el espectador. Actualmente, la Realidad Virtual representa un medio con un enorme potencial a la hora de evocar emociones debido a su capacidad de representar mundos sintéticos y de hacer que sean creíbles, cognitivamente, al usuario. Mezclar diferentes técnicas pictóricas tradicionales, inspiradas en la corriente de la pintura matérica, con las más vanguardistas técnicas 3d como la fotogrametría, los sistemas de partículas, materiales PBR , (physically based rendering) con el proposito de sumergimos en un entorno virtual generado por un motor de juegos a modo de lienzo inmersivo.[EN] One of the ultimate goals of an artistic work is, without a doubt, awaken emotions in the viewer. Currently, Virtual Reality represents a medium with enormous potential when it comes to evoking emotions due to their ability to represent synthetic worlds and to make them credible, cognitively, to the user. Mix different traditional painting techniques, inspired by the current of material painting, with the most avant-garde 3d techniques such as photogrammetry, particle systems, PBR materials, (physically based rendering) with the purpose of immersing ourselves in a virtual environment generated by an immersive canvas game engine.Esteban Rozalén, FJ. (2021). El lienzo inmersivo. Diseño y desarrollo de una experiencia pictórica en una realidad virtual. Universitat Politècnica de València. http://hdl.handle.net/10251/170222TFG

Presenting in Virtual Worlds: Towards an Architecture for a 3D Presenter explaining 2D-Presented Information

Entertainment, education and training are changing because of multi-party interaction technology. In the past we have seen the introduction of embodied agents and robots that take the role of a museum guide, a news presenter, a teacher, a receptionist, or someone who is trying to sell you insurances, houses or tickets. In all these cases the embodied agent needs to explain and describe. In this paper we contribute the design of a 3D virtual presenter that uses different output channels to present and explain. Speech and animation (posture, pointing and involuntary movements) are among these channels. The behavior is scripted and synchronized with the display of a 2D presentation with associated text and regions that can be pointed at (sheets, drawings, and paintings). In this paper the emphasis is on the interaction between 3D presenter and the 2D presentation

Painting in the light of digital reproduction

"The proliferation of digital photographs on the Internet is incomprehensibly vast. These images owe much to the categories and styles of traditional photography, yet often it is their unmediated low quality, in terms of selection, composition, and compression, which is particularly elevated to prominence by the new medium. The Internet represents a near infinite expansion of the mail-order catalogue, amateur snapshot or analogue video; a way of collecting visual information where the aesthetics of simple functionality or mediocrity is observed, as there is virtually no material cost involved. Photographers, filmmakers and painters have already trawled the found-image archive extensively. Gerhard Richter's encyclopaedic Atlas project or the photographic collections of Fischli and Weiss are clear examples of the artistic imperative to gather, filter and categorise pictures. Trying to develop taxonomies of images is like assembling a Thesaurus, where it is possible to cross-reference through every definition. Now it would seem that found images are all we have thanks to the Internet's primary function as consumer and diffuser of information, a generator of simulacra. Paradoxically, this infinite source seems to have more veracity due

De/construction sites: Romans and the digital playground

The Roman world as attested to archaeologically and as interacted with today has its expression in a great many computational and other media. The place of visualisation within this has been paramount. This paper argues that the process of digitally constructing the Roman world and the exploration of the resultant models are useful methods for interpretation and influential factors in the creation of a popular Roman aesthetic. Furthermore, it suggests ways in which novel computational techniques enable the systematic deconstruction of such models, in turn re-purposing the many extant representations of Roman architecture and material culture

Hyperpresent avatars

This paper will discuss two student projects, which were developed during a hybrid course between art/design and computer sciences at Sabancı University; both of which involve the creation of two avatars whose visual attributes are determined by data feeds from ‘Real Life’ sources by following up from Biocca's concept of the Cyborg’s Dilemma, we will describe the creative and technological processes which went into the materialization of these two avatars

Compositional structures in mural design : towards a site-specific deconstructive mural methodology

A thesis submitted to the University of Bedfordshire in partial fulfilment of the requirements of the degree of Doctor of Philosophy (PhD)Murals have been the formal visual interpretation of the cultural, social and political life of all ages. Throughout they have been consistently combined with their architectural setting, for example, in ancient Egyptian tombs, in Renaissance churches and on the external walls of buildings in Mexico in the twentieth century. This is a central feature of mural painting. However many contemporary murals do not integrate with their architectural settings, in other words, do not fulfil the site-specificity of the architectural spaces for which they were made. This means that the most important aspect that distinguishes murals from other types of painting is absent. I studied and analysed a number of murals produced in the Italian Renaissance, Baroque and Rococo as this particular period is considered to be not only one of the most significant in the history of art but also a period in which painting and architecture were very closely allied as practices. In particular the radical developments in painting of pictorial space took place along side the developments in architecture. I argue that Renaissance murals could be described, using the terminology of contemporary art, as site-specific art. By identifying the relationship between pictorial space, architectural space and compositional structure I was able to test, through my own practice, the importance of these relationships in understanding the site-specificity of the compositional structure of murals. To address the issue of sitespecificity in murals, I investigated and developed a set of compositional structures through my mural practice that could be applied in the design, execution, and teaching of contemporary mural design. I have developed the notion of a deconstructive method of mural design in which the illusory space of the mural derives its compositional structure from the architectural space in which it sited. I have applied it, tested it and refined it through the execution of a number of hypothetical and live mural commissions. I believe that the approach to the study and practice of mural design I have developed from the perspective of a practice lead researcher contributes to the furtherance of mural design as both a profession and field of study. In particular the identification of compositional structures in mural design and the proposal of a deconstructive method contributes to our understanding of what a mural is as well as current notions of site-specificity in contemporary art