END-TO-END PREDICTION OF WELD PENETRATION IN REAL TIME BASED ON DEEP LEARNING

Welding is an important joining technique that has been automated/robotized. In automated/robotic welding applications, however, the parameters are preset and are not adaptively adjusted to overcome unpredicted disturbances, which cause these applications to not be able to meet the standards from welding/manufacturing industry in terms of quality, efficiency, and individuality. Combining information sensing and processing with traditional welding techniques is a significant step toward revolutionizing the welding industry. In practical welding, the weld penetration as measured by the back-side bead width is a critical factor when determining the integrity of the weld produced. However, the back-side bead width is difficult to be directly monitored during manufacturing because it occurs underneath the surface of the welded workpiece. Therefore, predicting back-side bead width based on conveniently sensible information from the welding process is a fundamental issue in intelligent welding. Traditional research methods involve an indirect process that includes defining and extracting key characteristic information from the sensed data and building a model to predict the target information from the characteristic information. Due to a lack of feature information, the cumulative error of the extracted information and the complex sensing process directly affect prediction accuracy and real-time performance. An end-to-end, data-driven prediction system is proposed to predict the weld penetration status from top-side images during welding. In this method, a passive-vision sensing system with two cameras to simultaneously monitor the top-side and back-bead information is developed. Then the weld joints are classified into three classes (i.e., under penetration, desirable penetration, and excessive penetration) according to the back-bead width. Taking the weld pool-arc images as inputs and corresponding penetration statuses as labels, an end-to-end convolutional neural network (CNN) is designed and trained so the features are automatically defined and extracted. In order to increase accuracy and training speed, a transfer learning approach based on a residual neural network (ResNet) is developed. This ResNet-based model is pre-trained on an ImageNet dataset to process a better feature-extracting ability, and its fully connected layers are modified based on our own dataset. Our experiments show that this transfer learning approach can decrease training time and improve performance. Furthermore, this study proposes that the present weld pool-arc image is fused with two previous images that were acquired 1/6s and 2/6s earlier. The fused single image thus reflects the dynamic welding phenomena, and prediction accuracy is significantly improved with the image-sequence data by fusing temporal information to the input layer of the CNN (early fusion). Due to the critical role of weld penetration and the negligible impact on system implementation, this method represents major progress in the field of weld-penetration monitoring and is expected to provide more significant improvements during welding using pulsed current where the process becomes highly dynamic

Machine-human Cooperative Control of Welding Process

An innovative auxiliary control system is developed to cooperate with an unskilled welder in a manual GTAW in order to obtain a consistent welding performance. In the proposed system, a novel mobile sensing system is developed to non-intrusively monitor a manual GTAW by measuring three-dimensional (3D) weld pool surface. Specifically, a miniature structured-light laser amounted on torch projects a dot matrix pattern on weld pool surface during the process; Reflected by the weld pool surface, the laser pattern is intercepted by and imaged on the helmet glass, and recorded by a compact camera on it. Deformed reflection pattern contains the geometry information of weld pool, thus is utilized to reconstruct its $3$D surface. An innovative image processing algorithm and a reconstruction scheme have been developed for (3D) reconstruction. The real-time spatial relations of the torch and the helmet is formulated during welding. Two miniature wireless inertial measurement units (WIMU) are mounted on the torch and the helmet, respectively, to detect their rotation rates and accelerations. A quaternion based unscented Kalman filter (UKF) has been designed to estimate the helmet/torch orientations based on the data from the WIMUs. The distance between the torch and the helmet is measured using an extra structure-light low power laser pattern. Furthermore, human welder\u27s behavior in welding performance has been studied, e.g., a welder`s adjustments on welding current were modeled as response to characteristic parameters of the three-dimensional weld pool surface. This response model as a controller is implemented both automatic and manual gas tungsten arc welding process to maintain a consistent full penetration

Monitoração e análises da penetração do cordão de solda atraves da observação da oscilação da poça de fusão no processo GMAW-S

Tese (doutorado)—Universidade de Brasília, Faculdade de Tecnologia, Departamento de Engenharia Mecânica, 2018.A busca por métodos de produção com melhor controle da qualidade e maior produtividade tem impulsionado o uso de sistemas automatizados em processos industriais como a soldagem. Porém, tornar a soldagem eficiente e econômica, é necessário para reduzir o desperdício de material e tempo gasto na produção e ensaios de verificação de qualidade. Isso pode ser conseguido por meio de sistemas automatizados que substituam os soldadores especialistas e sejam capazes de prever a geometria do cordão de solda a partir dos parâmetros de soldagem – permitindo que um processo realizado com os parâmetros determinados forneça uma junta com as propriedades mecânicas desejadas. Durante anos, muito se tem feito no sentido de prever os problemas na soldagem com o intuito de torná-la um processo estável, capaz de efetuar uniões de peças com o mínimo de interferência humana. Dos vários sensores utilizados em processos de soldagem, ainda não há uma opção eficaz capaz de identificar, diretamente, as características do cordão obtido durante o processo. Esse é um fator limitante no controle do processo, pois somente é possível determinar as características do cordão após a realização da solda através de ensaios (destrutivos ou não), quando nenhuma ação de controle pode ser tomada. Este trabalho propõe o desenvolvimento de um sistema de monitoramento da poça de fusão em tempo real usado para obter imagens do comportamento da oscilação da poça durante a solda. Uma nova abordagem para este tipo de imagens é a utilização de um sistema de iluminação por laser do processo, de modo que uma imagem de alta qualidade natural da poça de fusão, eletrodo e cordão de solda possa ser obtida, dando detalhes da poça e arredores. Essa estratégia, independente de modelos pré-definidos do processo, permite controlar a penetração dos cordões de solda no processo GMAW no modo de transferência metálica por curto-circuito (GMAW-S). Para o modelo e controlador definiu-se a utilização de sistemas inteligentes focados diretamente nas medições da oscilação da poça de fusão e a estimação da penetração dos cordões de solda a partir dos parâmetros de processo. Finalmente, um modelo para relacionar a profundidade da penetração, a frequência de oscilação da poça com a formação e o padrão das escamas na superfície do cordão de solda é proposto.Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPQ).The search for methods of production with better quality control and greater productivity has promoted the use of automated systems in industrial processes such as welding. However, make the welding efficient and economical; it is necessary to reduce the waste and time spent on the production and quality tests. This can be achieved by means of automated systems to replace those skilled welders and be able to predict the geometry of the weld bead as welding parameters - allowing a process performed with the determined parameters provide a joint with the desired mechanical properties. For years, much has been done to predict problems in welding in order to make it a stable process capable of making unions parts with minimal human interference. The various sensors used in welding processes, there is still no effective option able to identify, directly, the weld bead characteristics obtained during the process. This is a limiting factor in the process control, because only can be determined the weld bead characteristics after the completion of welding through testing (destructive or not) when no control action can be taken. This work proposes the development of a real-time weld pool monitoring system to obtain the images of the weld pool oscillation behavior during welding. A novel approach to this type of images is the use of a laser lighting system for illumination of the process, so that a high quality natural image of the weld pool, electrode and weld bead can be obtained, giving details of the weld pool and surrounding area. This strategy, regardless of predefined models, can control the weld bead penetration in the GMAW-S process. For the proposed model and controller is defined the use of intelligent systems focused on the measurements of the weld pool oscillations and the estimation of the weld bead penetration from the process parameters. Finally, a model to relate the weld penetration depth, the weld pool oscillation frequency with the formation and the pattern of the ripples on the weld bead surface is proposed

Virtualized Welding Based Learning of Human Welder Behaviors for Intelligent Robotic Welding

Combining human welder (with intelligence and sensing versatility) and automated welding robots (with precision and consistency) can lead to next generation intelligent welding systems. In this dissertation intelligent welding robots are developed by process modeling / control method and learning the human welder behavior. Weld penetration and 3D weld pool surface are first accurately controlled for an automated Gas Tungsten Arc Welding (GTAW) machine. Closed-form model predictive control (MPC) algorithm is derived for real-time welding applications. Skilled welder response to 3D weld pool surface by adjusting the welding current is then modeled using Adaptive Neuro-Fuzzy Inference System (ANFIS), and compared to the novice welder. Automated welding experiments confirm the effectiveness of the proposed human response model. A virtualized welding system is then developed that enables transferring the human knowledge into a welding robot. The learning of human welder movement (i.e., welding speed) is first realized with Virtual Reality (VR) enhancement using iterative K-means based local ANFIS modeling. As a separate effort, the learning is performed without VR enhancement utilizing a fuzzy classifier to rank the data and only preserve the high ranking “correct” response. The trained supervised ANFIS model is transferred to the welding robot and the performance of the controller is examined. A fuzzy weighting based data fusion approach to combine multiple machine and human intelligent models is proposed. The data fusion model can outperform individual machine-based control algorithm and welder intelligence-based models (with and without VR enhancement). Finally a data-driven approach is proposed to model human welder adjustments in 3D (including welding speed, arc length, and torch orientations). Teleoperated training experiments are conducted in which a human welder tries to adjust the torch movements in 3D based on his observation on the real-time weld pool image feedback. The data is off-line rated by the welder and a welder rating system is synthesized. ANFIS model is then proposed to correlate the 3D weld pool characteristic parameters and welder’s torch movements. A foundation is thus established to rapidly extract human intelligence and transfer such intelligence into welding robots