A Novel Framework for Highlight Reflectance Transformation Imaging

We propose a novel pipeline and related software tools for processing the multi-light image collections (MLICs) acquired in different application contexts to obtain shape and appearance information of captured surfaces, as well as to derive compact relightable representations of them. Our pipeline extends the popular Highlight Reflectance Transformation Imaging (H-RTI) framework, which is widely used in the Cultural Heritage domain. We support, in particular, perspective camera modeling, per-pixel interpolated light direction estimation, as well as light normalization correcting vignetting and uneven non-directional illumination. Furthermore, we propose two novel easy-to-use software tools to simplify all processing steps. The tools, in addition to support easy processing and encoding of pixel data, implement a variety of visualizations, as well as multiple reflectance-model-fitting options. Experimental tests on synthetic and real-world MLICs demonstrate the usefulness of the novel algorithmic framework and the potential benefits of the proposed tools for end-user applications.Terms: "European Union (EU)" & "Horizon 2020" / Action: H2020-EU.3.6.3. - Reflective societies - cultural heritage and European identity / Acronym: Scan4Reco / Grant number: 665091DSURF project (PRIN 2015) funded by the Italian Ministry of University and ResearchSardinian Regional Authorities under projects VIGEC and Vis&VideoLa

Multispectral RTI Analysis of Heterogeneous Artworks

We propose a novel multi-spectral reflectance transformation imaging (MS-RTI) framework for the acquisition and direct analysis of the reflectance behavior of heterogeneous artworks. Starting from free-form acquisitions, we compute per-pixel calibrated multi-spectral appearance profiles, which associate a reflectance value to each sampled light direction and frequency. Visualization, relighting, and feature extraction is performed directly on appearance profile data, applying scattered data interpolation based on Radial Basis Functions to estimate per-pixel reflectance from novel lighting directions. We demonstrate how the proposed solution can convey more insights on the object materials and geometric details compared to classical multi-light methods that rely on low-frequency analytical model fitting eventually mixed with a separate handling of high-frequency components, hence requiring constraining priors on material behavior. The flexibility of our approach is illustrated on two heterogeneous case studies, a painting and a dark shiny metallic sculpture, that showcase feature extraction, visualization, and analysis of high-frequency properties of artworks using multi-light, multi-spectral (Visible, UV and IR) acquisitions.Terms: "European Union (EU)" & "Horizon 2020" / Action: H2020-EU.3.6.3. - Reflective societies - cultural heritage and European identity / Acronym: Scan4Reco / Grant number: 665091the DSURF (PRIN 2015) project funded by the Italian Ministry of University and ResearchSardinian Regional Authorities under projects VIGEC and Vis&VideoLa

A Practical Reflectance Transformation Imaging Pipeline for Surface Characterization in Cultural Heritage

We present a practical acquisition and processing pipeline to characterize the surface structure of cultural heritage objects. Using a free-form Reflectance Transformation Imaging (RTI) approach, we acquire multiple digital photographs of the studied object shot from a stationary camera. In each photograph, a light is freely positioned around the object in order to cover a wide variety of illumination directions. Multiple reflective spheres and white Lambertian surfaces are added to the scene to automatically recover light positions and to compensate for non-uniform illumination. An estimation of geometry and reflectance parameters (e.g., albedo, normals, polynomial texture maps coefficients) is then performed to locally characterize surface properties. The resulting object description is stable and representative enough of surface features to reliably provide a characterization of measured surfaces. We validate our approach by comparing RTI-acquired data with data acquired with a high-resolution microprofilometer.Terms: "European Union (EU)" & "Horizon 2020" / Action: H2020-EU.3.6.3. - Reflective societies - cultural heritage and European identity / Acronym: Scan4Reco / Grant number: 66509

Crack Detection in Single- and Multi-Light Images of Painted Surfaces using Convolutional Neural Networks

Cracks represent an imminent danger for painted surfaces that needs to be alerted before degenerating into more severe aging effects, such as color loss. Automatic detection of cracks from painted surfaces' images would be therefore extremely useful for art conservators; however, classical image processing solutions are not effective to detect them, distinguish them from other lines or surface characteristics. A possible solution to improve the quality of crack detection exploits Multi-Light Image Collections (MLIC), that are often acquired in the Cultural Heritage domain thanks to the diffusion of the Reflectance Transformation Imaging (RTI) technique, allowing a low cost and rich digitization of artworks' surfaces. In this paper, we propose a pipeline for the detection of crack on egg-tempera paintings from multi-light image acquisitions and that can be used as well on single images. The method is based on single or multi-light edge detection and on a custom Convolutional Neural Network able to classify image patches around edge points as crack or non-crack, trained on RTI data. The pipeline is able to classify regions with cracks with good accuracy when applied on MLIC. Used on single images, it can give still reasonable results. The analysis of the performances for different lighting directions also reveals optimal lighting directions

A novel framework for highlight reflectance transformation imaging

We propose a novel pipeline and related software tools for processing the multi-light image collections (MLICs) acquired in different application contexts to obtain shape and appearance information of captured surfaces, as well as to derive compact relightable representations of them. Our pipeline extends the popular Highlight Reflectance Transformation Imaging (H-RTI) framework, which is widely used in the Cultural Heritage domain. We support, in particular, perspective camera modeling, per-pixel interpolated light direction estimation, as well as light normalization correcting vignetting and uneven non-directional illumination. Furthermore, we propose two novel easy-to-use software tools to simplify all processing steps. The tools, in addition to support easy processing and encoding of pixel data, implement a variety of visualizations, as well as multiple reflectance-model-fitting options. Experimental tests on synthetic and real-world MLICs demonstrate the usefulness of the novel algorithmic framework and the potential benefits of the proposed tools for end-user applications

Artworks in the spotlight: characterization with a multispectral LED dome

We describe the design and realization of a novel multispectral light dome system and the associated software control and calibration tools used to process the acquired data, in a specialized pipeline geared towards the analysis of shape and appearance properties of cultural heritage items. The current prototype dome, built using easily available electronic and lighting components, can illuminate a target of size 20cm x 20cm from 52 directions uniformly distributed in a hemisphere. From each illumination direction, 3 LED lights cover the visible range of the electromagnetic spectrum, as well as long ultraviolet and near infrared. A dedicated control system implemented on Arduino boards connected to a controlling PC fully manages all lighting and a camera to support automated acquisition. The controlling software also allows real-time adjustment of the LED settings, and provides a live-view of the to-be-captured scene. We approach per-pixel light calibration by placing dedicated targets in the focal plane: four black reflective spheres for back-tracing the position of the LED lamps and a planar full- frame white paper to correct for the non-uniformity of radiance. Once the light calibration is safeguarded, the multispectral acquisition of an artwork can be completed in a matter of minutes, resulting in a spot-wise appearance profile, that stores at pixel level the per-frequency intensity value together with the light direction vector. By performing calibrated acquisition of multispectral Reflectance Transformation Imaging (RTI), with our analysis system it is possible to recover surface normals, to characterize matte and specular behavior of materials, and to explore different surface layers thanks to UV-VIS-IR LED light separation. To demonstrate the system features we present the outcomes of the on-site capture of metallic artwork at the National Archaeological Museum of Cagliari, Sardini

Multi-scale Painter Classification

<p>The characterization of a painter’s style is useful for a series of applications, such as documenting art history, planning style-aware conservation and restoration, and discarding forgery attempts. In this work, we propose a method to assign paintings to the right artist with two strategies: traditional machine learning and deep learning. In particular, we quantify the visual characteristics of a painting at multiple scales, covering low-level as well as mid-level features (pyramid of histogram of oriented gradients, residual convolutional neural network features). We focus on coeval artists, representing Impressionism, Expressionism and Cubism art periods. Our results are consistent with state-of-the-art findings in art and computer vision literature.</p&gt

Influence of color on visual saliency in short videos

International audienceThe architecture of computational models of visual attention designed for videos is generally the result of a direct extension of techniques dedicated to static images. These models try to extract the salient areas of dynamic scenes (i.e. areas that may capture visual attention) by fusing static saliency maps computed frame per frame with saliency maps independently obtained from dynamic features. The problem is that there is no evidence for assuming that visual saliency of videos can be accurately identified from such a fusion process. In addition, there is no guarantee that visual saliency is the same for still and dynamic scenes. Then we propose to investigate this issue for short videos from the perspective of color information that has been clearly identified as an important salient property in static images

Aging Prediction of Cultural Heritage Samples Based on Surface Microgeometry

A critical and challenging aspect for the study of Cultural Heritage (CH) assets is related to the characterization of the materials that compose them and to the variation of these materials with time. In this paper, we exploit a realistic dataset of artificially aged metallic samples treated with different coatings commonly used for artworks' protection in order to evaluate different approaches to extract material features from high-resolution depth maps. In particular, we estimated, on microprofilometric surface acquisitions of the samples, performed at different aging steps, standard roughness descriptors used in materials science as well as classical and recent image texture descriptors. We analyzed the ability of the features to discriminate different aging steps and performed supervised classification tests showing the feasibility of a texture-based aging analysis and the effectiveness of coatings in reducing the surfaces' change with time

Colour-Balanced Edge-Guided Digital Inpainting: Applications on Artworks

The virtual inpainting of artworks provides a nondestructive mode of hypothesis visualization, and it is especially attractive when physical restoration raises too many methodological and ethical concerns. At the same time, in Cultural Heritage applications, the level of details in virtual reconstruction and their accuracy are crucial. We propose an inpainting algorithm that is based on generative adversarial network, with two generators: one for edges and another one for colors. The color generator rebalances chromatically the result by enforcing a loss in the discretized gamut space of the dataset. This way, our method follows the modus operandi of an artist: edges first, then color palette, and, at last, color tones. Moreover, we simulate the stochasticity of the lacunae in artworks with morphological variations of a random walk mask that recreate various degradations, including craquelure. We showcase the performance of our model on a dataset of digital images of wall paintings from the Dunhuang UNESCO heritage site. Our proposals of restored images are visually satisfactory and they are quantitatively comparable to state-of-the-art approaches