Deep image prior inpainting of ancient frescoes in the Mediterranean Alpine arc

The unprecedented success of image reconstruction approaches based on deep neural networks has revolutionised both the processing and the analysis paradigms in several applied disciplines. In the field of digital humanities, the task of digital reconstruction of ancient frescoes is particularly challenging due to the scarce amount of available training data caused by ageing, wear, tear and retouching over time. To overcome these difficulties, we consider the Deep Image Prior (DIP) inpainting approach which computes appropriate reconstructions by relying on the progressive updating of an untrained convolutional neural network so as to match the reliable piece of information in the image at hand while promoting regularisation elsewhere. In comparison with state-of-the-art approaches (based on variational/PDEs and patch-based methods), DIP-based inpainting reduces artefacts and better adapts to contextual/non-local information, thus providing a valuable and effective tool for art historians. As a case study, we apply such approach to reconstruct missing image contents in a dataset of highly damaged digital images of medieval paintings located into several chapels in the Mediterranean Alpine Arc and provide a detailed description on how visible and invisible (e.g., infrared) information can be integrated for identifying and reconstructing damaged image regions.Comment: 26 page

From Augmentation to Inpainting:Improving Visual SLAM with Signal Enhancement Techniques and GAN-based Image Inpainting

This paper undertakes a comprehensive investigation that surpasses the conventional examination of signal enhancement techniques and their effects on visual Simultaneous Localization and Mapping (vSLAM) performance across diverse scenarios. Going beyond the conventional scope, the study extends its focus towards the seamless integration of signal enhancement techniques, aiming to achieve a substantial enhancement in the overall vSLAM performance. The research not only delves into the assessment of existing methods but also actively contributes to the field by proposing innovative denoising techniques that can play a pivotal role in refining the accuracy and reliability of vSLAM systems. This multifaceted approach encompasses a thorough exploration of the intricate relationships between signal enhancement, denoising strategies, their cumulative impact on the performance of vSLAM in real-world applications and the innovative use of Generative Adversarial Networks (GANs) for image inpainting. The GANs effectively fill in missing spaces following object detection and removal, presenting a novel state-of-the-art approach that significantly enhances overall accuracy and execution speed of vSLAM. This paper aims to contribute to the advancement of vSLAM algorithms in real-world scenarios, demonstrating improved accuracy, robustness, and computational efficiency through the amalgamation of signal enhancement and advanced denoising techniques

Intelligent enhancement of ancient Chinese murals based on multi-scale parallel structure

Ancient mural artwork preserves the historical background and cultural customs of that time through intricate details and bright colors. However, after the natural environment and man-made damage, these works of art are damaged in color, texture and content and lose their quality. In order to identify and enhance murals with large areas of color damage, we propose a multi-scale parallel GAN and parallel Unet structure, which can extract features from multiple scales or images to adapt to the changing scale of the target and provide a more diverse set of features. This structure can reduce the risk of overfitting the training data by learning more general features. The verification results of indicators such as PSNR on the ancient mural data set show that the method has a certain performance improvement effect

Generative Adversarial Networks Based Reconstruction and Restoration of Cultural Heritage

Cultural heritage takes an important part in defining the identity and the history of a civilization or a nation. Valuing and preserving this heritage is thus a top priority for governments and heritage institutions. Through this paper, we present an image completion (inpainting) approach adapted for the curation and the completion of damaged artwork. Our approach uses a set of machine learning techniques such as Generative Adversarial Networks which are among the most powerful generative models that can be trained to generate realistic data samples. As we are focusing mostly on visual cultural heritage, the pipeline of our framework has many optimizations such as the use of clustering to optimize the training of the generative part to ensure a better performance across a variety of cultural data categories. The experimental results of our framework were validated on cultural dataset of paintings collected from Wiki-Art and the Rijksmuseum. We used the divide-and-conquer strategy by clustering the training data into different small clusters containing similarly looking images to train smaller Specialized DCGANs. The training has been made on five painting categories containing 2000 paintings each, which took an average of 6.1 training hours. Training the Specialized DCGAN on 1200 paintings from one of the clusters took 3.4 training hours. The inpainting results of the Specialized DCGANs are clearly better in quality than the results of a DCGAN trained on mixture of paintings or on painting category

On Using Perceptual Loss within the U-Net Architecture for the Semantic Inpainting of Textile Artefacts with Traditional Motifs

It is impressive when one gets to see a hundreds or thousands years old artefact exhibited in the museum, whose appearance seems to have been untouched by centuries. Its restoration had been in the hands of a multidisciplinary team of experts and it had undergone a series of complex procedures. To this end, computational approaches that can support in deciding the most visually appropriate inpainting for very degraded historical items would be helpful as a second objective opinion for the restorers. The present paper thus attempts to put forward a U-Net approach with a perceptual loss for the semantic inpainting of traditional Romanian vests. Images taken of pieces from the collection of the Oltenia Museum in Craiova, along with such images with garments from the Internet, have been given to the deep learning model. The resulting numerical error for inpainting the corrupted parts is adequately low, however the visual similarity still has to be improved by considering further possibilities for finer tuning.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech

Methods for 3D Geometry Processing in the Cultural Heritage Domain

This thesis presents methods for 3D geometry processing under the aspects of cultural heritage applications. After a short overview over the relevant basics in 3D geometry processing, the present thesis investigates the digital acquisition of 3D models. A particular challenge in this context are on the one hand difficult surface or material properties of the model to be captured. On the other hand, the fully automatic reconstruction of models even with suitable surface properties that can be captured with Laser range scanners is not yet completely solved. This thesis presents two approaches to tackle these challenges. One exploits a thorough capture of the object’s appearance and a coarse reconstruction for a concise and realistic object representation even for objects with problematic surface properties like reflectivity and transparency. The other method concentrates on digitisation via Laser-range scanners and exploits 2D colour images that are typically recorded with the range images for a fully automatic registration technique. After reconstruction, the captured models are often still incomplete, exhibit holes and/or regions of insufficient sampling. In addition to that, holes are often deliberately introduced into a registered model to remove some undesired or defective surface part. In order to produce a visually appealing model, for instance for visualisation purposes, for prototype or replica production, these holes have to be detected and filled. Although completion is a well-established research field in 2D image processing and many approaches do exist for image completion, surface completion in 3D is a fairly new field of research. This thesis presents a hierarchical completion approach that employs and extends successful exemplar-based 2D image processing approaches to 3D and fills in detail-equipped surface patches into missing surface regions. In order to identify and construct suitable surface patches, selfsimilarity and coherence properties of the surface context of the hole are exploited. In addition to the reconstruction and repair, the present thesis also investigates methods for a modification of captured models via interactive modelling. In this context, modelling is regarded as a creative process, for instance for animation purposes. On the other hand, it is also demonstrated how this creative process can be used to introduce human expertise into the otherwise automatic completion process. This way, reconstructions are feasible even of objects where already the data source, the object itself, is incomplete due to corrosion, demolition, or decay.Methoden zur 3D-Geometrieverarbeitung im Kulturerbesektor In dieser Arbeit werden Methoden zur Bearbeitung von digitaler 3D-Geometrie unter besonderer Berücksichtigung des Anwendungsbereichs im Kulturerbesektor vorgestellt. Nach einem kurzen Überblick über die relevanten Grundlagen der dreidimensionalen Geometriebehandlung wird zunächst die digitale Akquise von dreidimensionalen Objekten untersucht. Eine besondere Herausforderung stellen bei der Erfassung einerseits ungünstige Oberflächen- oder Materialeigenschaften der Objekte dar (wie z.B. Reflexivität oder Transparenz), andererseits ist auch die vollautomatische Rekonstruktion von solchen Modellen, die sich verhältnismäßig problemlos mit Laser-Range Scannern erfassen lassen, immer noch nicht vollständig gelöst. Daher bilden zwei neuartige Verfahren, die diesen Herausforderungen begegnen, den Anfang. Auch nach der Registrierung sind die erfassten Datensätze in vielen Fällen unvollständig, weisen Löcher oder nicht ausreichend abgetastete Regionen auf. Darüber hinaus werden in vielen Anwendungen auch, z.B. durch Entfernen unerwünschter Oberflächenregionen, Löcher gewollt hinzugefügt. Für eine optisch ansprechende Rekonstruktion, vor allem zu Visualisierungszwecken, im Bildungs- oder Unterhaltungssektor oder zur Prototyp- und Replik-Erzeugung müssen diese Löcher zunächst automatisch detektiert und anschließend geschlossen werden. Obwohl dies im zweidimensionalen Fall der Bildbearbeitung bereits ein gut untersuchtes Forschungsfeld darstellt und vielfältige Ansätze zur automatischen Bildvervollständigung existieren, ist die Lage im dreidimensionalen Fall anders, und die Übertragung von zweidimensionalen Ansätzen in den 3D stellt vielfach eine große Herausforderung dar, die bislang keine zufriedenstellenden Lösungen erlaubt hat. Nichtsdestoweniger wird in dieser Arbeit ein hierarchisches Verfahren vorgestellt, das beispielbasierte Konzepte aus dem 2D aufgreift und Löcher in Oberflächen im 3D unter Ausnutzung von Selbstähnlichkeiten und Kohärenzeigenschaften des Oberflächenkontextes schließt. Um plausible Oberflächen zu erzeugen werden die Löcher dabei nicht nur glatt gefüllt, sondern auch feinere Details aus dem Kontext rekonstruiert. Abschließend untersucht die vorliegende Arbeit noch die Modifikation der vervollständigten Objekte durch Freiformmodellierung. Dies wird dabei zum einen als kreativer Prozess z.B. zu Animationszwecken betrachtet. Zum anderen wird aber auch untersucht, wie dieser kreative Prozess benutzt werden kann, um etwaig vorhandenes Expertenwissen in die ansonsten automatische Vervollständigung mit einfließen zu lassen. Auf diese Weise werden auch Rekonstruktionen ermöglicht von Objekten, bei denen schon die Datenquelle, also das Objekt selbst z.B. durch Korrosion oder mutwillige Zerstörung unvollständig ist

THE ROLE OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING IN PRESERVING CULTURAL HERITAGE AND ART WORKS VIA VIRTUAL RESTORATION

Artifacts including paintings, drawings, prints, mosaics, sculptures, historical structures and monuments, as well as archaeological sites present a key part of our cultural heritage. It consists of Intangible culture (such as folklore, traditions, language, and knowledge), tangible culture (such as buildings, monuments, landscapes, archival materials, books, works of art, and artifacts), and natural heritage (such as biodiversity and culturally significant landscapes) .Now we will concentrate on tangible culture and its problems and how to handle them. One of its biggest problems is that over the years the nature of the materials used in the creation of the artwork make them prone to cracks, fractures, stains, and colors fading and blurring. The causes of their damage could be of natural or human- related reasons. The natural causes range from war, fires, earthquakes, natural disasters and the human-related causes range from accidental events like to pollution which results of climate changes, which like acid rain. It is a must to consider the environment in which you store your artwork. Our regular environment's light, heat, moisture, and pollution levels can lead to harmful chemical and physical reactions in artwork. There are several reasons why it is necessary to preserve ancient works of art. The fact that it enables us to comprehend the historical and cultural context of the era in which it was made is one of the key factors. Then preserving our artworks is a must and this can be done by manual techniques or using machine learning algorithms

Innovative Techniques for Digitizing and Restoring Deteriorated Historical Documents

Recent large-scale document digitization initiatives have created new modes of access to modern library collections with the development of new hardware and software technologies. Most commonly, these digitization projects focus on accurately scanning bound texts, some reaching an efficiency of more than one million volumes per year. While vast digital collections are changing the way users access texts, current scanning paradigms can not handle many non-standard materials. Documentation forms such as manuscripts, scrolls, codices, deteriorated film, epigraphy, and rock art all hold a wealth of human knowledge in physical forms not accessible by standard book scanning technologies. This great omission motivates the development of new technology, presented by this thesis, that is not-only effective with deteriorated bound works, damaged manuscripts, and disintegrating photonegatives but also easily utilized by non-technical staff. First, a novel point light source calibration technique is presented that can be performed by library staff. Then, a photometric correction technique which uses known illumination and surface properties to remove shading distortions in deteriorated document images can be automatically applied. To complete the restoration process, a geometric correction is applied. Also unique to this work is the development of an image-based uncalibrated document scanner that utilizes the transmissivity of document substrates. This scanner extracts intrinsic document color information from one or both sides of a document. Simultaneously, the document shape is estimated to obtain distortion information. Lastly, this thesis provides a restoration framework for damaged photographic negatives that corrects photometric and geometric distortions. Current restoration techniques for the discussed form of negatives require physical manipulation to the photograph. The novel acquisition and restoration system presented here provides the first known solution to digitize and restore deteriorated photographic negatives without damaging the original negative in any way. This thesis work develops new methods of document scanning and restoration suitable for wide-scale deployment. By creating easy to access technologies, library staff can implement their own scanning initiatives and large-scale scanning projects can expand their current document-sets