Artificial intelligence for advanced manufacturing quality

100 p.This Thesis addresses the challenge of AI-based image quality control systems applied to manufacturing industry, aiming to improve this field through the use of advanced techniques for data acquisition and processing, in order to obtain robust, reliable and optimal systems. This Thesis presents contributions onthe use of complex data acquisition techniques, the application and design of specialised neural networks for the defect detection, and the integration and validation of these systems in production processes. It has been developed in the context of several applied research projects that provided a practical feedback of the usefulness of the proposed computational advances as well as real life data for experimental validation

Photometric Stereo-Based Defect Detection System for Steel Components Manufacturing Using a Deep Segmentation Network

This paper presents an automatic system for the quality control of metallic components using a photometric stereo-based sensor and a customized semantic segmentation network. This system is designed based on interoperable modules, and allows capturing the knowledge of the operators to apply it later in automatic defect detection. A salient contribution is the compact representation of the surface information achieved by combining photometric stereo images into a RGB image that is fed to a convolutional segmentation network trained for surface defect detection. We demonstrate the advantage of this compact surface imaging representation over the use of each photometric imaging source of information in isolation. An empirical analysis of the performance of the segmentation network on imaging samples of materials with diverse surface reflectance properties is carried out, achieving Dice performance index values above 0.83 in all cases. The results support the potential of photometric stereo in conjunction with our semantic segmentation network

Generative Adversarial Networks to Improve the Robustness of Visual Defect Segmentation by Semantic Networks in Manufacturing Components

This paper describes the application of Semantic Networks for the detection of defects in images of metallic manufactured components in a situation where the number of available samples of defects is small, which is rather common in real practical environments. In order to overcome this shortage of data, the common approach is to use conventional data augmentation techniques. We resort to Generative Adversarial Networks (GANs) that have shown the capability to generate highly convincing samples of a specific class as a result of a game between a discriminator and a generator module. Here, we apply the GANs to generate samples of images of metallic manufactured components with specific defects, in order to improve training of Semantic Networks (specifically DeepLabV3+ and Pyramid Attention Network (PAN) networks) carrying out the defect detection and segmentation. Our process carries out the generation of defect images using the StyleGAN2 with the DiffAugment method, followed by a conventional data augmentation over the entire enriched dataset, achieving a large balanced dataset that allows robust training of the Semantic Network. We demonstrate the approach on a private dataset generated for an industrial client, where images are captured by an ad-hoc photometric-stereo image acquisition system, and a public dataset, the Northeastern University surface defect database (NEU). The proposed approach achieves an improvement of 7% and 6% in an intersection over union (IoU) measure of detection performance on each dataset over the conventional data augmentation

Automated classification of civil structures defects based on Convolutional Neural Network

Today, the most used method for civil infrastructure inspection is based on visual assessment performed by certified inspectors following prescribed protocols. However, the increase in aggressive environmental and load conditions, coupled with the achievement for many structures of the end life-cycle, highlighted the need to automate damage identification to satisfy the number of structures that need to be inspected. To overcome this challenge, the current paper presents a method to automate the concrete damage classification using a deep Convolutional Neural Network (CNN). The CNN is designed after an experimental investigation among a wide number of pretrained networks, all applying the transfer learning technique. Training and Validation are performed using a built database with 1352 images balanced between “undamaged”, “cracked”, and “delaminated” concrete surface. To increase the network robustness compared to images with real-world situations, different configurations of images has been collected from Internet and on-field bridge inspections. The GoogLeNet model is selected as the most suitable network for the concrete damage classification, having the highest validation accuracy of about 94%. The results confirm that the proposed model can correctly classify images from real concrete surface of bridges, tunnel and pavement, resulting an effective alternative to the current visual inspection

Surface Defect Detection Using YOLO Network

Detecting defects on surfaces such as steel can be a challenging task because defects have complex and unique features. These defects happen in many production lines and differ between each one of these production lines. In order to detect these defects, the You Only Look Once (YOLO) detector which uses a Convolutional Neural Network (CNN), is used and received only minor modifications. YOLO is trained and tested on a dataset containing six kinds of defects to achieve accurate detection and classification. The network can also obtain the coordinates of the detected bounding boxes, giving the size and location of the detected defects. Since manual defect detection is expensive, labor-intensive and inefficient, this paper contributes to the sophistication and improvement of manufacturing processes. This system can be installed on chipsets and deployed to a factory line to greatly improve quality control and be part of smart internet of things (IoT) based factories in the future. YOLO achieves a respectable 70.66% mean average precision (mAP) despite the small dataset and minor modifications to the network

A Systematic Review of Convolutional Neural Network-Based Structural Condition Assessment Techniques

With recent advances in non-contact sensing technology such as cameras, unmanned aerial and ground vehicles, the structural health monitoring (SHM) community has witnessed a prominent growth in deep learning-based condition assessment techniques of structural systems. These deep learning methods rely primarily on convolutional neural networks (CNNs). The CNN networks are trained using a large number of datasets for various types of damage and anomaly detection and post-disaster reconnaissance. The trained networks are then utilized to analyze newer data to detect the type and severity of the damage, enhancing the capabilities of non-contact sensors in developing autonomous SHM systems. In recent years, a broad range of CNN architectures has been developed by researchers to accommodate the extent of lighting and weather conditions, the quality of images, the amount of background and foreground noise, and multiclass damage in the structures. This paper presents a detailed literature review of existing CNN-based techniques in the context of infrastructure monitoring and maintenance. The review is categorized into multiple classes depending on the specific application and development of CNNs applied to data obtained from a wide range of structures. The challenges and limitations of the existing literature are discussed in detail at the end, followed by a brief conclusion on potential future research directions of CNN in structural condition assessment

Crack Detection in Single- and Multi-Light Images of Painted Surfaces using Convolutional Neural Networks

Cracks represent an imminent danger for painted surfaces that needs to be alerted before degenerating into more severe aging effects, such as color loss. Automatic detection of cracks from painted surfaces' images would be therefore extremely useful for art conservators; however, classical image processing solutions are not effective to detect them, distinguish them from other lines or surface characteristics. A possible solution to improve the quality of crack detection exploits Multi-Light Image Collections (MLIC), that are often acquired in the Cultural Heritage domain thanks to the diffusion of the Reflectance Transformation Imaging (RTI) technique, allowing a low cost and rich digitization of artworks' surfaces. In this paper, we propose a pipeline for the detection of crack on egg-tempera paintings from multi-light image acquisitions and that can be used as well on single images. The method is based on single or multi-light edge detection and on a custom Convolutional Neural Network able to classify image patches around edge points as crack or non-crack, trained on RTI data. The pipeline is able to classify regions with cracks with good accuracy when applied on MLIC. Used on single images, it can give still reasonable results. The analysis of the performances for different lighting directions also reveals optimal lighting directions

3D inspection methods for specular or partially specular surfaces

Deflectometric techniques are a powerful tool for the automated quality control of specular or shiny surfaces. These techniques are based on using a camera to observe a reference pattern reflected on the surface under inspection, exploiting the dependence of specular reflections on surface normals to recover shape information from the acquired images. Although deflectometry is already used in industrial environments such as the quality control of lenses or car bodies, there are still some open problems. On the one hand, using quantitative deflectometry, the normal vector field and the 3D shape of a surface can be obtained, but these techniques do not yet take full advantage of their local sensitivity because the achieved global accuracies are affected by calibration errors. On the other hand, qualitative deflectometry is used to detect surface imperfections without absolute measurements, exploiting the local sensitivity of deflectometric recordings with reduced calibration requirements. However, this qualitative approach requires further processing that can involve a considerable engineering effort, particularly for aesthetic defects which are inherently subjective. The first part of this thesis aims to contribute to a better understanding of how deflectometric setups and their calibration errors affect quantitative measurements. Different error sources are considered including the camera calibration uncertainty and several non-ideal characteristics of LCD screens used to generate the light patterns. Experiments performed using real measurements and simulations show that the non-planarity of the LCD screen and the camera calibration are the dominant sources of error. The second part of the thesis investigates the use of deep learning to identify geometrical imperfections and texture defects based on deflectometric data. Two different approaches are explored to extract and combine photometric and geometric information using convolutional neural network architectures: one for automated classification of defective samples, and another one for automated segmentation of defective regions in a sample. The experimental results in a real industrial case study indicate that both architectures are able to learn relevant features from deflectometric data, enabling the classification and segmentation of defects based on a dataset of user-provided examples.Teknika deflektometrikoak tresna baliotsuak dira gainazal espekular edo distiratsuen kalitate kontrol automatikoa gauzatzeko. Teknika hauetan, kamera bat erabiltzen da ikuskatu beharreko gainazalean islatutako erreferentziazko patroi bat behatzeko, eta isladapen espekularrek gainazalen bektore normalengan duten menpekotasuna ustiatzen dute irudietatik informazio geometrikoa berreskuratzeko. Zenbait industria-aplikaziotan deflektometria jada erabiltzen bada ere –adibidez, betaurrekoen edo autoen karrozerien kalitate kontrolean-, oraindik badaude hobetu beharreko hainbat esparru. Batetik, deflektometria kuantitatiboak aukera ematen du gainazal baten bektore-eremu normala eta 3D forma lortzeko, baina gaur egun teknika hauek ez dute beren sentsibilitate lokal guztia aprobetxatzen kalibrazio-akatsek zehaztasun globalean duten eraginagatik. Bestetik, deflektometria kualitatiboa neurketa absoluturik egin gabe gainazal akatsak antzemateko erabili daiteke, kalibrazio-eskakizun murriztuekin sentsibilitate lokala ustiatuz. Hala ere, teknika horiek algoritmoen garapenean esfortzu handia ekar dezakeen prozesamendu bat eskatzen dute, bereziki bere baitan subjektiboak diren akats estetikoetarako. Hala ere, teknika horiek algoritmoen garapenean esfortzu handia ekar dezakeen prozesamendu bat eskatzen dute, bereziki bere baitan subjektiboak diren akats estetikoetarako. Tesi honen lehen zatiaren helburua adkizizio sistema osatzen duten gailuek eta horien kalibrazioek neurketa kuantitatiboei nola eragiten dieten hobeto ulertzen laguntzea da. Hainbat errore-iturri hartzen dira kontuan, besteak beste kameraren kalibrazioaren ziurgabetasuna, eta argi-patroiak sortzeko erabilitako LCD pantailen zenbait ezaugarri ez-ideal. Neurketa errealetan eta simulazioetan egindako esperimentuek erakusten dute LCD pantailaren deformazioak eta kameraren kalibrazioak eragindako erroreak direla neurketen akats eta ziurgabetasun iturri nagusiak. Tesiaren bigarren zatian, datu deflektometrikoetatik abiatuz, inperfekzio geometrikoak eta testura-akatsak identifikatzeko ikaskuntza sakoneko metodoen erabilera ikertzen da. Helburu honekin, irudietatik informazio fotometrikoa eta geometrikoa atera eta konbinatzen duten sare neuronal konboluzionaletan oinarritutako bi arkitektura proposatzen dira: bata, lagin akastunak automatikoki sailkatzeko; eta, bestea, laginetako eremu akastunak automatikoki segmentatzeko. Automobilgintza industriako kasu praktiko baten lortutako emaitzek erakusten dute erabilitako arkitekturek datu deflektometrikoetatik ezaugarri esanguratsuak ikas ditzaketela, erabiltzaileak emandako adibide multzo batean oinarrituta gainazal akatsak sailkatu eta segmentatzea ahalbidetuz.Las técnicas deflectométricas son una herramienta valiosa para automatizar el control de calidad de superficies especulares o reflectantes. Estas técnicas se basan en el uso de una cámara para observar un patrón de referencia reflejado en la superficie bajo inspección, explotando la dependencia de los reflejos especulares en la normal de la superficie para recuperar información geométrica a partir de las imágenes adquiridas. Aunque la deflectometría ya se usa en algunas aplicaciones industriales, tales como el control de calidad de lentes o carrocerías de coches, todavía hay algunos problemas abiertos. Por un lado, la deflectometría cuantitativa permite obtener el campo vectorial normal y la forma 3D de una superficie, pero a día de hoy no es capaz de aprovechar al máximo su sensibilidad local ya que la precisión global se ve afectada por errores de calibración. Por otro lado, la deflectometría cualitativa se utiliza para detectar imperfecciones de la superficie sin mediciones absolutas, explotando la sensibilidad local de la deflectometría con requisitos de calibración reducidos. Sin embargo, estos métodos requieren un procesamiento adicional que puede implicar un esfuerzo considerable en el desarrollo de algoritmos, particularmente para defectos estéticos que son inherentemente subjetivos. La primera parte de esta tesis tiene como objetivo contribuir a una mejor comprensión de cómo el sistema de adquisición y su calibración afectan a las mediciones cuantitativas. Se consideran dife-rentes fuentes de error, incluida la incertidumbre de calibración de la cámara y varias características no ideales de las pantallas LCD utilizadas para generar los patrones de luz. Los experimentos realizados con mediciones reales y simulaciones indican que los errores inducidos por la deformación de la pantalla LCD y la calibración de la cámara son las principales fuentes de error e incertidumbre. La segunda parte de la tesis investiga el uso del aprendizaje profundo para identificar imperfecciones geométricas y defectos de textura a partir de datos deflectométricos. Se adoptan dos enfoques diferentes para extraer y combinar información fotométrica y geométrica utilizando sendas arquitecturas basadas en redes neuronales convolucionales: una para la clasificación automatizada de muestras defectuosas y otra para la segmentación automatizada de regiones defectuosas en una muestra. Los resultados experimentales en un caso de estudio industrial real indican que ambas arquitecturas pueden aprender características relevantes de los datos deflectométricos, permitiendo la clasificación y segmentación de defectos en base a un conjunto de datos de ejemplos proporcionados por el usuario