Cornelis, Bruno

Martens, Maximiliaan

Meeus, Laurens

Pizurica, Aleksandra

Sizyakin, Roman

Voronin, Viacheslav

Ghent University Academic Bibliography

A deep learning approach to crack detection in panel paintingsR. Sizyakin1, B. Cornelis2, L. Meeus3, M. Martens4, V. Voronin1 and A. Pižurica3.1Department of Radio-Electronics Systems, Don State Technical University, Russian Federation .2Department of Electronics and Informatics, Vrije Universiteit Brussel, Belgium.3Department of Telecommunications and Information Processing, TELIN-IPI-imec, Ghent University, Belgium.4Department of Art, Music and Theatre Sciences, Ghent University, Belgium.Abstract— The accurate detection of cracks in paintings, whichgenerally portray rich and varying content, is a challenging task.Furthermore, traditional crack detection methods are often notwell suited for recent acquisitions of paintings as they are notdesigned for high-resolution images and do not fully exploit theinformation from the different imaging modalities at hand. Inthis work, we propose a fast crack detection framework that al-leviates the aforementioned challenges. The method consists of amorphological ﬁltering operation followed by a classiﬁcation stepby means of a convolutional neural network architecture. Theproposed online method is capable of continuously learning fromnewly acquired visual data, thus further improving classiﬁcationresults as more data becomes available.1 IntroductionPaint cracking (or craquelure) is the most common type ofdeterioration encountered in old master paintings. Generallyspeaking, cracks appear in paint layers when pressure devel-ops within or on it through the inﬂuence of various factors andcause the material to break [1]. The automatic detection ofcrack patterns is desirable for many reason. Most importantly,crack patterns can offer insights on the structural condition andconservation history of a painting [1]. Crack detection is alsoused as a preprocessing step for the digital restoration of paint-ings [2].Many crack detection methods have been developed over therecent years, see e.g. [2–6] and the reference therein. Still,some important challenges remain especially in terms of fea-ture selection, which often has to be adapted for different paint-ings, parameter tuning, and complexity, which limit practicalapplicability.In this paper, we propose a new deep learning based ap-proach for crack detection in paintings. The method consistsof two processing stages: (i) a morphological ﬁltering stage,and (ii) a classiﬁcation stage. The morphological ﬁltering es-sentially ensures that the amount of pixels to be classiﬁed bythe CNN in the second stage is strongly reduced as only thosepixels that are similar in structure to cracks are selected. In theclassiﬁcation stage, we employ a convolutional neural network(CNN). CNNs demonstrated recent success in many applica-tion where they have outperformed, often by a substantial mar-gin, traditional machine learning algorithms [7–10]. We are notaware of any reported works that apply CNNs to crack detec-tion in paintings. Some recent works applied CNN to detectcracks in roads [11, 12]. Our problem is, however, much morechallenging not only because of the huge variability of cracks inpaintings but also due to complex background and the fact thatsome painted details can closely resemble cracks. Therefore,our approach needs to incorporate multimodal data, which to-gether with huge spatial resolution of digitized paintings posesadditional challenges for the classiﬁer.2 Proposed approachThe ﬁrst step in the proposed method is morphological ﬁlteringof the available modalities. This operation creates a preliminarycrack map in a similar way as in [4, 5]. Each ﬁltered result isfollowed by a thresholding step, producing binary images. Thethreshold is set based on the method of Otsu [13]. The binarymaps are then combined into a single one using the logical ORoperation. The morphological ﬁltering step improves greatlythe classiﬁcation speed. The classiﬁer is run only on pixelsmarked as crack in this ﬁrst stage.The input of our classiﬁcation network consists of tensors ofsize m ×m × N . These tensors are formed by concatenatingm×m sized patches extracted from the N considered modali-ties (the three color channels of the visual macro-photographs,the single-channel infrared macro-photographs and X-ray im-ages, along with their grayscale morphologically ﬁltered resultsadd up to N = 9 modalities.1). An input sample is representedby the tensor x(u1, u2, υ0) ∈ Rm×m×N , where u1, u2 are spa-tial coordinates and υ0 is the index that identiﬁes the chosenmodality. For our experiments, we ﬁx m = 8, resulting in ten-sors with dimensions of 8 × 8 × 9. The convolutions over υ0are calculated in the ﬁrst layer of the CNN as follows:x1(u1, u2, υ1) = ρ�x(u1, u2, υ0) ∗ wυ1(u1, u2, υ0)�, (1)where x1(u1, u2, υ1) is the feature map obtained by the convo-lution of x(u1, u2, υ0) with wυ1(u1, u2, υ0), indexed by υ1 (inour architecture 1 ≤ υ1 ≤ 12 for the ﬁrst, 1 ≤ υ2 ≤ 24 forsecond and 1 ≤ υ2 ≤ 48 for the third convolution layers), andwhere ρ is an activation function. We choose the well-knownrectiﬁed linear unit (ReLU) [7], deﬁned as ρ = max(0, x). Allkernels are initialised randomly in the beginning of the train-ing procedure. The core of our CNN architecture consists ofperforming a cascade of convolutions at each layer j ≥ 2 asfollows:xj(u1, u2, υj) = ρ�xj−1(·, υj−1) ∗ wυj (·)�, (2)where, as we navigate through the subsequent layers, the reso-lution of xj(u1, u2, υj) progressively reduces [14,15]. The lastlayer of the architecture consist of a softmax function, whichensures that all class probabilities sum up to 1.1A disk-shaped structuring element with a diameter of 5 pixels was used forall experiments.40Figure 1: Left: A fragment of the panel Virgin Annunciate. Right: Resultsof the proposed method in comparison with BCTF. Blue: cracks identiﬁed byboth methods; red: cracks detected by BCTF only; green: cracks detected bythe proposed method only.3 Experimental ResultsAll experiments are performed on a high-resolution multi-modal dataset of theGhent Altarpiece, publicly available on theCloser to Van Eyck website2. In particular, we focus on threepanels of the polyptych, named Virgin Annunciate, Singing An-gels and John Evangelist. For comparsion we use the BayesianConditional Tensor Factorisation (BCTF) method from [3],which was also evaluated and compared to other methods in [6].Figure 1(Left) shows crack detection results on a part of thepanel Virgin Annunciate which is particularly challenging be-cause some painted features resemble cracks. The results dis-played in Fig. 1(Right) show that BCTF falsely labels a signif-icant amount of pixels (such as the decorative elements aroundthe big letter “P”), while our method successfully differentiatestrue cracks from those painted features. It should be noted thatboth methods use the same image modalities.Similar conclusions follow from the results on the panelSinging Angels displayed in Fig. 2(Top). In general, the pro-posed method detects more cracks while reducing false detec-tions. Furthermore, the proposed method can be trained in anonline fashion, i.e. without re-training the whole network, con-tinuously improving detection results (Fig. 2(Bottom)). An im-portant asset is also rapid processing, especially once the net-work is trained. This makes our framework integrable in a fastand interactive tool that can be used by art professionals.Figure 3 depicts results on a small area from the panel ofJohn the Evangelist. For this example, only one modalitywas considered, namely the color photograph, for both meth-ods. The BCTF method was trained speciﬁcally for this image,while the proposed method was pre-trained on other panels andonly a small fraction of labels (approximately 3,000 patches oftwo types) from the present panel were used to re-train the net-work. This experiment indicates that the proposed method canbe deployed for a variety of paintings, with relative little effort.In general, the proposed method demonstrates potential toimprove upon the current state-of-the-art methods by detectingmore cracks while also reducing false detections. Furthermore,the proposed method can be trained in an online fashion, i.e.without re-training the whole network, continuously improvingdetection results. An important asset is also rapid processing,especially once the network is trained. This makes our frame-work integrable in a fast and interactive tool that can be usedby art professionals.2Link: http://closertovaneyck.kikirpa.be/Figure 2: Top: A fragment of the panel Singing Angels. Bottom: Resultsof the proposed method in comparison with BCTF. Blue: cracks identiﬁed byboth methods; red: cracks detected by BCTF only; green: cracks detected bythe proposed method only.4 ConclusionIn this paper, we propose a novel crack detection frameworkcapable of handling acquisitions from different modalities. Ina ﬁrst step, we apply morphological ﬁltering for a coarse initialidentiﬁcation of crack pixels. This step substantially reducesthe amount of data to be processed later on. We then train aCNN architecture with user labelled data to further reﬁne theresults obtained in the ﬁrst step. We show that our method im-proves upon the current state-of-the-art in this application. Anadditional advantage is the possibility of re-training the net-work using newly available data. This feature allows to im-prove the already obtained result without signiﬁcant time costs.Figure 3: Left: A fragment of the panel John Evangelist. Right: Results of theproposed method (pre-trained on other panels and re-trained with few labelsfrom the present panel) in comparison with BCTF (trained for this particularimage). Blue: cracks identiﬁed by both methods, red: cracks detected by BCTFonly, and green: cracks detected by the proposed method only.4142References[1] J. Mohen, M. Menu, and B. Mottin, “Mona lisa: Insidethe painting,” Harry N. Abrams, New York, NY, USA,,2006.[2] B. Cornelis, T. Ružic´, E. Gezels, A. Dooms, A. Pižurica,L. Platiša, J. Cornelis, M. Martens, M. De Mey, andI. Daubechies, “Crack detection and inpainting for virtualrestoration of paintings: The case of the ghent altarpiece,”Signal process, vol. 93, no. 3, pp. 605–619, 2013.[3] B. Cornelis, Y. Yang, Joshua T. Vogelstein, A. Dooms, I.Daubechies, and D. B. Dunson, “Bayesian crack detec-tion in ultra high resolution multimodal images of paint-ings,” 18th International Conference on Digital SignalProcessing, IEEE, 2013.[4] A. Gupta, V. Khandelwal, A. Gupta, and M. C. S. Tham-masat, “Image processing methods for the restoration ofdigitized paintings,” Thammasat Int. J. Sc. Tech, vol. 13,no. 3, 2008.[5] I. Giakoumis, N. Nikolaidis, and I Pitas, “Digital imageprocessing techniques for the detection and removal ofcracks in digitized paintings,” IEEE Trans. Image Pro-cessing, vol. 15, no. 1, pp. 178–188, 2006.[6] A. Pižurica, L. Platiša, T. Ruž|ic´, B. Cornelis, A. Dooms,M. Martens, H. Dubois, B. Devolder, M. De Mey, andI. Daubechies, “Digital image processing of the Ghent al-tarpiece: supporting the painting’s study and conservationtreatment,” Signal Processing Magazine, IEEE , vol. 32,2015.[7] A. Krizhevsky and I. Sutskever and G. Hinton, “Imagenetclassiﬁcation with deep convolutional neural networks,”Advances in Neural Information Processing Systems, pp.1097–1105, 2012.[8] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed,Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke,and Andrew Rabinovich, “Going deeper with convolu-tions,” CoRR, vol. abs/1409.4842, 2014.[9] K. Simonyan and A. Zisserman, “Very deep convolutionalnetworks for large-scale image recognition,” CoRR, vol.abs/1409.1556, 2014.[10] K. He, X. Zhang, S. Ren, and J. Sun, “Deepresidual learning for image recognition,” CoRR, vol.abs/1512.03385, 2015.[11] Z. Lei, Y. Fan, D.Z. Yimin, and J. Z. Ying, “Roadcrack detection using deep convolutional neural,” ImageProcessing (ICIP), 2016 IEEE International Conference,2016.[12] Y.-Jin Cha, W. Choi, and O. Büyüköztürk, “Deeplearning-based crack damage detection using convolu-tional neural networks,” Comput.-Aided Civ. Infrastruct.Eng., vol. 32, pp. 361–378, 2017.[13] N. Otsu, “A threshold selection method from gray-levelhistogram,” IEEE Transaction on Systems, Man, and Cy-bernatics, vol. SMC-9, no. 1, pp. 62–66, 1979.[14] S. Mallat, “Understanding deep convolutional networks,”Phil. Trans. R. Soc. A, vol. 374(2065):20150203, 2016.[15] J. H. Jacobsen, E. Oyallon, S. Mallat, and A. W. M.Smeulders, “Multiscale hierarchical convolutional net-works,” CoRR, vol. abs/1703.04140, 2017.

A deep learning approach to crack detection in panel paintings

https://biblio.ugent.be/publication/8577805/file/8577806.pdf

A deep learning approach to crack detection in panel paintings

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography