nfrared thermography is a valuable non-destructive tool for inspection of materials. It measures the surface temperature evolution, from which hidden defects may be detected. Yet, thermal cameras typically have a low native spatial resolution resulting in a blurry and low-quality thermal image sequence and videos. In this study, a novel adversarial deep learning framework, called Dual-IRT-GAN, is proposed for performing super-resolution tasks. The proposed Dual-IRT-GAN attempts to achieve the objective of improving local texture details, as well as highlighting defective regions. Technically speaking, the proposed model consists of two modules SEGnet and SRnet that carry out defect detection and super resolution tasks, respectively. By leveraging the defect information from SEGnet, SRnet is capable of generating plausible high-resolution thermal images with an enhanced focus on defect regions. The generated high-resolution images are then delivered to the discriminator for adversarial training using GAN's framework. The proposed Dual-IRT-GAN model, which is trained on an exclusive virtual dataset, is demonstrated on experimental thermographic data obtained from fiber reinforced polymers having a variety of defect types, sizes, and depths. The obtained results show its high performance in maintaining background color consistency and removing undesired noise, and in highlighting defect zones with finer detailed textures in high-resolution

Cheng, Liangliang

Kersemans, Mathias

English

Proceedings - University of Groningen

   University of GroningenDefect-aware Super-resolution Thermography by Adversarial LearningCheng, Liangliang; Kersemans, MathiasIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2023Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Cheng, L., & Kersemans, M. (2023). Defect-aware Super-resolution Thermography by AdversarialLearning.CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 01-11-2023      Defect-aware Super-resolution Thermography by Adversarial Learning  Liangliang Cheng1,2, and Mathias Kersemans1 1 Mechanics of Materials and Structures (MMS), Ghent University, Belgium 2 Faculty of Science and Engineering, University of Groningen, Netherlands E-mail: liangliang.cheng@rug.nl; mathias.kersemans@ugent.be  Abstract  Infrared thermography is a valuable non-destructive tool for inspection of materials. It measures the surface temperature evolution, from which hidden defects may be detected. Yet, thermal cameras typically have a low native spatial resolution resulting in a blurry and low-quality thermal image sequence and videos. In this study, a novel adversarial deep learning framework, called Dual-IRT-GAN, is proposed for performing super-resolution tasks. The proposed Dual-IRT-GAN attempts to achieve the objective of improving local texture details, as well as highlighting defective regions. The generated high-resolution images are then delivered to the discriminator for adversarial training using GAN's framework.  The proposed Dual-IRT-GAN model, which is trained on an exclusive virtual dataset, is demonstrated on experimental thermographic data obtained from fibre reinforced polymers having a variety of defect types, sizes, and depths. The obtained results show  its high performance in maintaining background colour consistency and removing undesired noise, and in highlighting defect zones with finer detailed textures in high-resolution.   KEYWORDS: Deep learning; Super-resolution; GAN; Infrared thermography; Composite; Defect detection; Non-destructive testing   1.  Introduction  Nowadays, carbon fibre reinforced polymer (CFRP) and glass fibre reinforced polymer (GFRP) composites are widely used in a range of applications, with a focus on the aerospace and automotive industries. Therefore, Non-Destructive Testing (NDT) has become a crucial tool for guaranteeing the structural integrity of composite components.  Active InfraRed Thermography (IRT) is an attractive non-destructive testing (NDT) technique for diagnostic purposes that utilizes an IR camera to monitor the heat response promptly and precisely in order to discover and quantify defects [1]. The information of high-frequency textures, however, tends to be lost in the received thermal images due to the nature of an infrared imaging system having a limited number of pixels and pixel size. As a result, the obtained thermal images become blurry and fuzzy. This forms a hot research topic termed single image super-resolution (SISR).  More info about this article:https://www.ndt.net/?id=28108© 2023 The Authors. Published by NDT.net under License CC-BY-4.0 https://creativecommons.org/licenses/by/4.0/ 2 In recent years, CFRP and GFRP have gained popularity in recent years due to their high strength and low weight. Numerous signal post-processing approaches [2-4] have been used to determine the presence of defects in CFRP and GFRP specimens. These approaches, however, are restricted in resolution and are susceptible to noise, regardless of the capability in defect identification.  In order to overcome the difficulties presented by low resolution (LR), we propose a defect-aware deep adversarial learning framework, as a continuation of previous work IRT-GAN [5], termed Dual-IRT-GAN, to enhance the SR image quality of real LR infrared images by leveraging the defect information.   2.  The Dual-IRT-GAN architecture   Figure 1. The overview of the proposed Dual-IRT-GAN framework  The overview of the proposed Dual-IRT-GAN model at the general architecture level is displayed in Figure 1. In general, it consists of two generators: SRnet and SEGnet and two discriminators DSR and DSEG, with SRnet attempting to generate plausible photo-realistic HR infrared images 𝐼𝐼𝑅𝑇_𝐻𝑅 with a scaling factor of 4 from LR infrared images 𝑰𝐼𝑅𝑇_𝐿𝑅, and SEGnet aiming to segment the defects 𝑰𝑆𝐸𝐺_𝐿𝑅 from ground-truth LR infrared images 𝑰𝐼𝑅𝑇_𝐿𝑅 in composites. To determine their authenticity, the generated HR infrared images from SRnet and segmentation maps from SEGnet are fed into two discriminators, DSR and DSEG, respectively. To compensate for the domain gap between the infrared images and realistic real-world images, a domain adaptation process is incorporated into the Dual-IRT-GAN training process to bias the parameters in SRnet and ensure SRnet learning to reconstruct with realistic textures and maintain color consistency. The SRnet is additionally trained on LR/HR ( 𝑰𝐼𝑅𝑇_𝐿𝑅 / 𝑰𝐼𝑅𝑇_𝐻𝑅_𝐺𝑇  in Figure 1) image pairs from the DIV2K dataset [6], containing 1000 real-world 2k resolution images.  A detailed description of the generator and discriminator network architectures is described in the Figure 2 and Figure 3. The proposed Generator, inspired by ESRGAN [7], unifies two subnets SRnet and SEGnet to jointly learn both tasks of SR image generation and defect segmentation,  3 respectively, in an end-to-end training framework via the trade-off between reconstruction loss and segmentation loss.   Figure 2. The network architecture of the generators (SRnet and SEGnet)   Figure 3. The proposed defect cross-attention module (DefectCAM) for the Dual-IRT-GAN To customize our SR task, modifications are designated in an effort to generate more photorealistic HR images with finer textures, as shown in Figure 2. As an auxiliary network, SEGnet attempts to investigate all potential defects in the IRT dataset of composites, thereby guiding and assisting SRnet in concentrating on defect areas via a proposed DefectCAM block, see Figure 3, inspired by Convolutional Block Attention Module(CBAM) [8].  Inspired by real-ESRGAN [9], the discriminators DSR and DSEG in this study employ the identical Unet architecture with spectral normalization to determine the authenticity or falsity of generated HR images (𝑰𝐼𝑅𝑇_𝐻𝑅 and 𝑰𝐷𝐼𝑉2𝐾_𝐻𝑅) and segmentation maps (𝑰𝑆𝐸𝐺_𝐻𝑅), respectively. Rather than focusing exclusively on local textures originating from PatchGAN [10], a modification to the discriminator in real-ESRGAN by paying extra attention to a global structure is formulated to the original PatchGAN output. The framework of the discriminators DSR and DSEG is displayed in Figure 4.    4  Figure 4. The architecture of DSR and DSEG  3.  Training  3.1 Loss function  In this work, three types of loss functions are used concurrently to achieve the desired result:  Content loss 𝑳𝟏, Perceptual loss 𝑳𝑽𝑮𝑮 and Adversarial loss 𝑳𝑨𝑫𝑽. The total loss 𝑳𝒕𝒐𝒕, therefore, can be therefore concluded as 𝐿𝑡𝑜𝑡 = 𝛼 𝐿𝑉𝐺𝐺 + 𝛽 𝐿1  + 𝛾 𝐿𝐴𝐷𝑉        (1) Where 𝜶, 𝜷 and 𝜸 are the loss weights. For more details, the readers are referred to [11].  3.2 Implementation details  The proposed Dual-IRT-GAN model is trained entirely in the absence of experimental data using a virtual dataset implemented in the Fortran90 environment, see [11] for more details. Prior to training the Dual-IRT-GAN, the pre-processing of virtual numerical IRT dataset in source-domain training and DIV2K dataset in domain-adaptation training is conducted to obtain the original training HR image set 𝑰𝐼𝑅𝑇_𝐻𝑅_𝐺𝑇  and 𝑰𝐷𝐼𝑉2𝐾_𝐻𝑅_𝐺𝑇by randomly cropping with a size of 256×256 and the corresponding training LR image set 𝑰𝐼𝑅𝑇_𝐿𝑅and 𝑰𝐷𝐼𝑉2𝐾_𝐿𝑅 with a size of 64×64 are obtained following a second-degradation process [8].  The Dual-IRT-GAN is implemented using the PyTorch framework in Python (Python 3.7.10 and CUDA v11.0.221) using an NVIDIA Quadro RTX 6000 GPU card with 24GB of RAM. The IRT-GAN model employs the Adam optimizer with an initial learning rate lr = 2×10-4, and momentum parameters 𝛽1=0.9, 𝛽2=0.999. The batch size for training is 8. The training epoch is set to be 1500, empirically shown to yield a good performance.  4. Validation on experimental IRT dataset  A schematic diagram of the thermographic setup is displayed in Figure 5(a). The optical energy input is supplied by a Hensel linear flash lamp with 6 kJ energy and 5 ms flash duration, and a FLIR A6750sc infrared camera is used to capture the sample’s surface temperature.   5   Figure 5: (a) Schematic of the experimental setup for flash thermography, (b) photograph and defect parameters of CFRPFBH  The testing sample, referred to as CFRPFBH, is a 5.5 mm thick compression-moulded CFRP plate with a quasi-isotropic stacking sequence of [(+45/0/-45/90)3]s. From the backside of the sample, 12 circular FBHs with diameters ranging from 7 to 25 mm and a remaining thickness of up to 4 mm were milled.   Table.1 The inference outcomes for CFRPFBH dataset utilizing the Bicubic interpolation, Nearest interpolation, SRGAN, ESRGAN, and Dual-IRT-GAN models Sample Up-sampling (×4)    From Table.1, we observe that the proposed model is capable of producing relatively cleaner and sharper edges and accurately preserving finer textures, resulting in HR images that are perceptually plausible to humans. In order to evaluate the quality of HR images that were generated, a widely-used metric known as peak signal-to-noise ratio (PSNR) [12] was employed. However, due to the absence of a ground-truth reference, a cubic image was used as a reference instead. This poses a challenge because the resulting PSNR values, expressed as blue values in Table.1, may not accurately reflect the true quality of the generated images. For instance, from Table.1, the highest PSNR value can be observed for Nearest up-sampling image.  5. Conclusions  Dual-IRT-GAN, a novel deep learning model, is presented. It consists of a generator with two subnets, SRnet and SEGnet, to jointly perform super-resolution and defect segmentation tasks for infrared images, and two discriminators, DSR and DSEG, to determine the veracity (or falsity) of the generated high-resolution images and segmentation maps, respectively. The Dual-IRT-GAN model can be directed toward learning plausible realistic high-resolution solutions, improving the visibility of defected  6 regions and in-depth textures. This is made possible by the implementation of the DefectCAM Module, which extracts and fuses the attention maps into the feature maps. On the basis of thermographic data obtained from a CFRP sample with FBH defects, the performance of the Dual-IRT-GAN is assessed.   Acknowledgements The authors express their gratitude towards Honda R&D Co. Japan and Sabca Limburg for supplying material for this research. This research has received funding from UGent Bijzonder OnderzoeksFonds BOF 01N01719.  References [1] Ibarra-Castanedo C, Genest M, Piau JM, Guibert S, Bendada A, Maldague XP. Active infrared thermography techniques for the nondestructive testing of materials. InUltrasonic and advanced methods for nondestructive testing and material characterization 2007 (pp. 325-348). [2] Poelman G, Hedayatrasa S, Segers J, Van Paepegem W, Kersemans M. Adaptive spectral band integration in flash thermography: Enhanced defect detectability and quantification in composites. Composites Part B: Engineering. 2020 Dec 1;202:108305. [3] Rajic N. Principal component thermography for flaw contrast enhancement and flaw depth characterisation in composite structures. Composite structures. 2002 Dec 1;58(4):521-8. [4] Maldague X, Marinetti S. Pulse phase infrared thermography. Journal of applied physics. 1996 Mar 1;79(5):2694-8. [5] Cheng L, Tong Z, Xie S, Kersemans M. IRT-GAN: A generative adversarial network with a multi-headed fusion strategy for automated defect detection in composites using infrared thermography. Composite Structures. 2022 Jun 15;290:115543. [6] Agustsson E, Timofte R. Ntire 2017 challenge on single image super-resolution: Dataset and study. InProceedings of the IEEE conference on computer vision and pattern recognition workshops 2017 (pp. 126-135). [7] Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Qiao Y, Change Loy C. Esrgan: Enhanced super-resolution generative adversarial networks. InProceedings of the European conference on computer vision (ECCV) workshops 2018 (pp. 0-0). [8] Woo S, Park J, Lee JY, Kweon IS. Cbam: Convolutional block attention module. InProceedings of the European conference on computer vision (ECCV) 2018 (pp. 3-19). [9] Wang X, Xie L, Dong C, Shan Y. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. InProceedings of the IEEE/CVF International Conference on Computer Vision 2021 (pp. 1905-1914). [10] Isola P, Zhu JY, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. InProceedings of the IEEE conference on computer vision and pattern recognition 2017 (pp. 1125-1134). [11] Cheng L, Kersemans M. Dual-IRT-GAN: A defect-aware deep adversarial network to perform super-resolution tasks in infrared thermographic inspection[J]. Composites Part B: Engineering, 2022, 247: 110309. [12] Poobathy D, Chezian RM. Edge detection operators: Peak signal to noise ratio based comparison. IJ Image, Graphics and Signal Processing. 2014 Sep 1;10:55-61. 

Defect-aware Super-resolution Thermography by Adversarial Learning

https://pure.rug.nl/ws/files/701081324/ECNDT2023_PAPER_243_3_.pdf

Defect-aware Super-resolution Thermography by Adversarial Learning

Abstract

Similar works

Full text

Available Versions

Proceedings - University of Groningen