Computer-assisted analysis of painting brushstrokes: digital image processing for unsupervised extraction of visible features from van Gogh's works

The automatic extraction of objective features from paintings, like brushstroke distribution, orientation, and shape, could be particularly useful for different artwork analyses and management tasks. In fact, these features contribute to provide a unique signature of the artists' style and can be effectively used for artist identification and classification, artwork examination and retrieval, etc. In this paper, an automatic technique for unsupervised extraction of individual brushstrokes from digital reproductions of van Gogh's paintings is presented. Through the iterative application of segmentation, characterization, and validation steps, valid brushstrokes complying with specific area and shape constraints are identified. On the extracted brushstrokes, several representative features can then be calculated, like orientation, length, and width. The accuracy of the devised method is evaluated by comparing numerical results obtained on a dataset of digital reproductions of van Gogh's oil-on-canvas works with observations made by human subjects and with another recent approach for automatic brushstroke analysis. Experimental tests showed that the devised methodology produces results that are rather close to those obtained by human subjects and, for some of the metrics considered, can provide improved performances with respect to alternative techniques

Automatic painting with economized strokes

Journal ArticleWe present a method that takes a raster image as input and produces a painting-like image composed of strokes rather than pixels. Unlike previous automatic painting methods, we attempt to use very few brush-strokes. This is accomplished by first segmenting the image into features, finding the medial axes points of these features, converting the medial axes points into ordered lists of image tokens, and finally rendering these lists as brush strokes. Our process creates images reminiscent of modern realist painters who often want an abstract or sketchy quality in their work

Fast Accurate and Automatic Brushstroke Extraction

Brushstrokes are viewed as the artist’s “handwriting” in a painting. In many applications such as style learning and transfer, mimicking painting, and painting authentication, it is highly desired to quantitatively and accurately identify brushstroke characteristics from old masters’ pieces using computer programs. However, due to the nature of hundreds or thousands of intermingling brushstrokes in the painting, it still remains challenging. This article proposes an efficient algorithm for brush Stroke extraction based on a Deep neural network, i.e., DStroke. Compared to the state-of-the-art research, the main merit of the proposed DStroke is to automatically and rapidly extract brushstrokes from a painting without manual annotation, while accurately approximating the real brushstrokes with high reliability. Herein, recovering the faithful soft transitions between brushstrokes is often ignored by the other methods. In fact, the details of brushstrokes in a master piece of painting (e.g., shapes, colors, texture, overlaps) are highly desired by artists since they hold promise to enhance and extend the artists’ powers, just like microscopes extend biologists’ powers. To demonstrate the high efficiency of the proposed DStroke, we perform it on a set of real scans of paintings and a set of synthetic paintings, respectively. Experiments show that the proposed DStroke is noticeably faster and more accurate at identifying and extracting brushstrokes, outperforming the other methods

The brushstroke and materials of Amadeo de Souza-Cardoso combined in an authentication tool

Nowadays, authentication studies for paintings require a multidisciplinary approach, based on the contribution of visual features analysis but also on characterizations of materials and techniques. Moreover, it is important that the assessment of the authorship of a painting is supported by technical studies of a selected number of original artworks that cover the entire career of an artist. This dissertation is concerned about the work of modernist painter Amadeo de Souza-Cardoso. It is divided in three parts. In the first part, we propose a tool based on image processing that combines information obtained by brushstroke and materials analysis. The resulting tool provides qualitative and quantitative evaluation of the authorship of the paintings; the quantitative element is particularly relevant, as it could be crucial in solving authorship controversies, such as judicial disputes. The brushstroke analysis was performed by combining two algorithms for feature detection, namely Gabor filter and Scale Invariant Feature Transform. Thanks to this combination (and to the use of the Bag-of-Features model), the proposed method shows an accuracy higher than 90% in distinguishing between images of Amadeo’s paintings and images of artworks by other contemporary artists. For the molecular analysis, we implemented a semi-automatic system that uses hyperspectral imaging and elemental analysis. The system provides as output an image that depicts the mapping of the pigments present, together with the areas made using materials not coherent with Amadeo’s palette, if any. This visual output is a simple and effective way of assessing the results of the system. The tool proposed based on the combination of brushstroke and molecular information was tested in twelve paintings obtaining promising results. The second part of the thesis presents a systematic study of four selected paintings made by Amadeo in 1917. Although untitled, three of these paintings are commonly known as BRUT, Entrada and Coty; they are considered as his most successful and genuine works. The materials and techniques of these artworks have never been studied before. The paintings were studied with a multi-analytical approach using micro-Energy Dispersive X-ray Fluorescence spectroscopy, micro-Infrared and Raman Spectroscopy, micro-Spectrofluorimetry and Scanning Electron Microscopy. The characterization of Amadeo’s materials and techniques used on his last paintings, as well as the investigation of some of the conservation problems that affect these paintings, is essential to enrich the knowledge on this artist. Moreover, the study of the materials in the four paintings reveals commonalities between the paintings BRUT and Entrada. This observation is supported also by the analysis of the elements present in a photograph of a collage (conserved at the Art Library of the Calouste Gulbenkian Foundation), the only remaining evidence of a supposed maquete of these paintings. The final part of the thesis describes the application of the image processing tools developed in the first part of the thesis on a set of case studies; this experience demonstrates the potential of the tool to support painting analysis and authentication studies. The brushstroke analysis was used as additional analysis on the evaluation process of four paintings attributed to Amadeo, and the system based on hyperspectral analysis was applied on the painting dated 1917. The case studies therefore serve as a bridge between the first two parts of the dissertation

Image preprocessing for artistic robotic painting

Artistic robotic painting implies creating a picture on canvas according to a brushstroke map preliminarily computed from a source image. To make the painting look closer to the human artwork, the source image should be preprocessed to render the effects usually created by artists. In this paper, we consider three preprocessing effects: aerial perspective, gamut compression and brushstroke coherence. We propose an algorithm for aerial perspective amplification based on principles of light scattering using a depth map, an algorithm for gamut compression using nonlinear hue transformation and an algorithm for image gradient filtering for obtaining a well-coherent brushstroke map with a reduced number of brushstrokes, required for practical robotic painting. The described algorithms allow interactive image correction and make the final rendering look closer to a manually painted artwork. To illustrate our proposals, we render several test images on a computer and paint a monochromatic image on canvas with a painting robot

Authentication of Amadeo de Souza-Cardoso Paintings and Drawings With Deep Learning

Art forgery has a long-standing history that can be traced back to the Roman period and has become more rampant as the art market continues prospering. Reports disclosed that uncountable artworks circulating on the art market could be fake. Even some principal art museums and galleries could be exhibiting a good percentage of fake artworks. It is therefore substantially important to conserve cultural heritage, safeguard the interest of both the art market and the artists, as well as the integrity of artists’ legacies. As a result, art authentication has been one of the most researched and well-documented fields due to the ever-growing commercial art market in the past decades. Over the past years, the employment of computer science in the art world has flourished as it continues to stimulate interest in both the art world and the artificial intelligence arena. In particular, the implementation of Artificial Intelligence, namely Deep Learning algorithms and Neural Networks, has proved to be of significance for specialised image analysis. This research encompassed multidisciplinary studies on chemistry, physics, art and computer science. More specifically, the work presents a solution to the problem of authentication of heritage artwork by Amadeo de Souza-Cardoso, namely paintings, through the use of artificial intelligence algorithms. First, an authenticity estimation is obtained based on processing of images through a deep learning model that analyses the brushstroke features of a painting. Iterative, multi-scale analysis of the images is used to cover the entire painting and produce an overall indication of authenticity. Second, a mixed input, deep learning model is proposed to analyse pigments in a painting. This solves the image colour segmentation and pigment classification problem using hyperspectral imagery. The result is used to provide an indication of authenticity based on pigment classification and correlation with chemical data obtained via XRF analysis. Further algorithms developed include a deep learning model that tackles the pigment unmixing problem based on hyperspectral data. Another algorithm is a deep learning model that estimates hyperspectral images from sRGB images. Based on the established algorithms and results obtained, two applications were developed. First, an Augmented Reality mobile application specifically for the visualisation of pigments in the artworks by Amadeo. The mobile application targets the general public, i.e., art enthusiasts, museum visitors, art lovers or art experts. And second, a desktop application with multiple purposes, such as the visualisation of pigments and hyperspectral data. This application is designed for art specialists, i.e., conservators and restorers. Due to the special circumstances of the pandemic, trials on the usage of these applications were only performed within the Department of Conservation and Restoration at NOVA University Lisbon, where both applications received positive feedback.A falsificação de arte tem uma história de longa data que remonta ao período romano e tornou-se mais desenfreada à medida que o mercado de arte continua a prosperar. Relatórios revelaram que inúmeras obras de arte que circulam no mercado de arte podem ser falsas. Mesmo alguns dos principais museus e galerias de arte poderiam estar exibindo uma boa porcentagem de obras de arte falsas. Por conseguinte, é extremamente importante conservar o património cultural, salvaguardar os interesses do mercado da arte e dos artis- tas, bem como a integridade dos legados dos artistas. Como resultado, a autenticação de arte tem sido um dos campos mais pesquisados e bem documentados devido ao crescente mercado de arte comercial nas últimas décadas.Nos últimos anos, o emprego da ciência da computação no mundo da arte floresceu à medida que continua a estimular o interesse no mundo da arte e na arena da inteligência artificial. Em particular, a implementação da Inteligência Artificial, nomeadamente algoritmos de aprendizagem profunda (ou Deep Learning) e Redes Neuronais, tem-se revelado importante para a análise especializada de imagens.Esta investigação abrangeu estudos multidisciplinares em química, física, arte e informática. Mais especificamente, o trabalho apresenta uma solução para o problema da autenticação de obras de arte patrimoniais de Amadeo de Souza-Cardoso, nomeadamente pinturas, através da utilização de algoritmos de inteligência artificial. Primeiro, uma esti- mativa de autenticidade é obtida com base no processamento de imagens através de um modelo de aprendizagem profunda que analisa as características de pincelada de uma pintura. A análise iterativa e multiescala das imagens é usada para cobrir toda a pintura e produzir uma indicação geral de autenticidade. Em segundo lugar, um modelo misto de entrada e aprendizagem profunda é proposto para analisar pigmentos em uma pintura. Isso resolve o problema de segmentação de cores de imagem e classificação de pigmentos usando imagens hiperespectrais. O resultado é usado para fornecer uma indicação de autenticidade com base na classificação do pigmento e correlação com dados químicos obtidos através da análise XRF. Outros algoritmos desenvolvidos incluem um modelo de aprendizagem profunda que aborda o problema da desmistura de pigmentos com base em dados hiperespectrais. Outro algoritmo é um modelo de aprendizagem profunda estabelecidos e nos resultados obtidos, foram desenvolvidas duas aplicações. Primeiro, uma aplicação móvel de Realidade Aumentada especificamente para a visualização de pigmentos nas obras de Amadeo. A aplicação móvel destina-se ao público em geral, ou seja, entusiastas da arte, visitantes de museus, amantes da arte ou especialistas em arte. E, em segundo lugar, uma aplicação de ambiente de trabalho com múltiplas finalidades, como a visualização de pigmentos e dados hiperespectrais. Esta aplicação é projetada para especialistas em arte, ou seja, conservadores e restauradores. Devido às circunstâncias especiais da pandemia, os ensaios sobre a utilização destas aplicações só foram realizados no âmbito do Departamento de Conservação e Restauro da Universidade NOVA de Lisboa, onde ambas as candidaturas receberam feedback positivo