Design and Analysis of the Virtual Reality Welding Training

A thesis presented to the faculty of the College of Business and Technology at Morehead State University in partial fulfillment of the requirements for the Degree Master of Science by Ritesh Chakradhar on November 19, 2021

TOWARD INTELLIGENT WELDING BY BUILDING ITS DIGITAL TWIN

To meet the increasing requirements for production on individualization, efficiency and quality, traditional manufacturing processes are evolving to smart manufacturing with the support from the information technology advancements including cyber-physical systems (CPS), Internet of Things (IoT), big industrial data, and artificial intelligence (AI). The pre-requirement for integrating with these advanced information technologies is to digitalize manufacturing processes such that they can be analyzed, controlled, and interacted with other digitalized components. Digital twin is developed as a general framework to do that by building the digital replicas for the physical entities. This work takes welding manufacturing as the case study to accelerate its transition to intelligent welding by building its digital twin and contributes to digital twin in the following two aspects (1) increasing the information analysis and reasoning ability by integrating deep learning; (2) enhancing the human user operative ability to physical welding manufacturing via digital twins by integrating human-robot interaction (HRI). Firstly, a digital twin of pulsed gas tungsten arc welding (GTAW-P) is developed by integrating deep learning to offer the strong feature extraction and analysis ability. In such a system, the direct information including weld pool images, arc images, welding current and arc voltage is collected by cameras and arc sensors. The undirect information determining the welding quality, i.e., weld joint top-side bead width (TSBW) and back-side bead width (BSBW), is computed by a traditional image processing method and a deep convolutional neural network (CNN) respectively. Based on that, the weld joint geometrical size is controlled to meet the quality requirement in various welding conditions. In the meantime, this developed digital twin is visualized to offer a graphical user interface (GUI) to human users for their effective and intuitive perception to physical welding processes. Secondly, in order to enhance the human operative ability to the physical welding processes via digital twins, HRI is integrated taking virtual reality (VR) as the interface which could transmit the information bidirectionally i.e., transmitting the human commends to welding robots and visualizing the digital twin to human users. Six welders, skilled and unskilled, tested this system by completing the same welding job but demonstrate different patterns and resulted welding qualities. To differentiate their skill levels (skilled or unskilled) from their demonstrated operations, a data-driven approach, FFT-PCA-SVM as a combination of fast Fourier transform (FFT), principal component analysis (PCA), and support vector machine (SVM) is developed and demonstrates the 94.44% classification accuracy. The robots can also work as an assistant to help the human welders to complete the welding tasks by recognizing and executing the intended welding operations. This is done by a developed human intention recognition algorithm based on hidden Markov model (HMM) and the welding experiments show that developed robot-assisted welding can help to improve welding quality. To further take the advantages of the robots i.e., movement accuracy and stability, the role of the robot upgrades to be a collaborator from an assistant to complete a subtask independently i.e., torch weaving and automatic seam tracking in weaving GTAW. The other subtask i.e., welding torch moving along the weld seam is completed by the human users who can adjust the travel speed to control the heat input and ensure the good welding quality. By doing that, the advantages of humans (intelligence) and robots (accuracy and stability) are combined together under this human-robot collaboration framework. The developed digital twin for welding manufacturing helps to promote the next-generation intelligent welding and can be applied in other similar manufacturing processes easily after small modifications including painting, spraying and additive manufacturing

Virtual Reality Applied to Welder Training

Welding is a challenging, risky, and time-consuming profession. Recently, there has been a documented shortage of trained welders, and as a result, the market is pushing for an increase in the rate at which new professionals are trained. To address this growing demand, training institutions are exploring alternative methods to train future professionals with the goals of improving learner retention of information, shortening training periods, and lowering associated expenses. The emergence of virtual reality technologies has led to initiatives to explore their potential for welding training. Multiple studies have suggested that virtual reality training delivers comparable, or even superior, results when compared to more conventional approaches, with shorter training times and reduced costs in consumables. Additionally, virtual reality allows trainees to try out different approaches to their work. The primary goal of this dissertation is to develop a virtual reality welding simulator. To achieve this objective effectively, the creation of a classification system capable of identifying the simulator’s key characteristics becomes imperative. Therefore, the secondary objective of this thesis is to develop a classification system for the accurate evaluation and comparison of virtual reality welding simulators. Regarding the virtual reality welding simulation, the HTC VIVE Pro 2 virtual reality equipment was employed, to transfer the user’s action from the physical to the virtual world. Within this virtual environment, it was introduced a suite of welding tools and integrated a Smoothed Particle Hydrodynamics simulator to mimic the weld creation. After conducting comprehensive testing that revealed certain limitations in welding quality and in the simulator performance, the project opted to incorporate a Computational Fluid Dynamics (CFD) simulator. The development of the CFD simulator proved to be a formidable challenge, and regrettably, its complete implementation was unattainable. Nevertheless, the project delved into three distinct grid architectures, from these, the dynamic grid was ultimately implemented. It also proficiently integrated two crucial solvers for the Navier-Stokes equations. These functions were implemented in the Graphics Processing Unit (GPU), to improve their efficiency. Upon comparing GPU and Central Processing Unit (CPU) performance, the project highlighted the substantial computational advantages of GPUs and the advantages it brings to fluid simulations.A soldadura é uma profissão exigente, perigosa e que requer um grande investimento de tempo para alcançar resultados satisfatórios. Recentemente, tem sido registada uma falta de profissionais qualificados na área da soldadura. Como resultado, o mer cado está a pressionar para um aumento do ritmo a que os novos trabalhadores são formados. Para responder a esta crescente procura, as instituições de formação estão a explorar métodos alternativos para formar futuros profissionais, com o objetivo de melhorar a retenção de informação, encurtar os períodos de treino e reduzir as despe sas associadas. Com o desenvolvimento de tecnologias nas áreas de realidade virtual e realidade aumentada, têm surgido iniciativas para explorar o potencial destas na formação de soldadura. Vários estudos sugeriram que a formação em realidade virtual proporciona resultados comparáveis, ou mesmo superiores, aos de abordagens mais convencionais, com tempos de formação mais curtos e reduções nos custos de consumíveis. Além disso, a realidade virtual permite aos formandos experimentar diferentes abordagens ao seu trabalho. O objetivo principal desta dissertação é o desenvolvimento de um simulador de soldadura em realidade virtual. Para atingir este objetivo de forma eficaz, torna-se imperativa a criação de um sistema de classificação capaz de identificar as características chave do simulador. Assim, o objetivo secundário desta dissertação é desenvolver um sistema de classificação para a avaliação e comparação precisas de simuladores de soldadura em realidade virtual. Relativamente ao simulador de soldadura em realidade virtual, foi utilizado o kit de realidade virtual HTC VIVE Pro 2, para transferir as ações do utilizador no mundo físico para o mundo virtual. No ambiente virtual, foi introduzido um con junto de ferramentas de soldadura e integrado um simulador de Hidrodinâmica de Partículas Suavizadas para simular a criação da solda. Após a realização de testes exaustivos que revelaram algumas limitações na qualidade da solda e no desempenho do simulador, o projeto optou por incorporar um simulador de Dinâmica de Fluidos Computacional (CFD). O desenvolvimento do simulador CFD revelou-se um desa fio formidável e, infelizmente, não foi possível completar a sua implementação. No entanto, o projeto aprofundou três arquiteturas de grelha distintas, das quais foi implementada a grelha dinâmica. O projeto também implementou duas funções cru ciais para resolver as equações de Navier-Stokes. As funções relativas ao simulador de fluidos foram implementadas na Unidade de Processamento Gráfico (GPU), a fim de melhorar a sua eficiência. Ao comparar o desempenho da GPU com o da Unidade Central de Processamento (CPU), o projeto evidenciou os beneficios computacionais das GPUs e as vantagens que trazem para as simulações de fluidos

Varying Feedback Strategy and Scheduling in Simulator Training: Effects on Learner Perceptions, Initial Learning, and Transfer

This experimental study investigated the effects of visual feedback on initial learning, perceived self-efficacy, workload, near transfer, far transfer, and perceived realism during a simulator-based training task. Prior studies indicate that providing feedback is critical for schema development (Salmoni, Schmidt, & Walter 1984; Sterman, 1994). However, its influence has been shown to dissipate and is not directly proportionate to the frequency at which it is given (Wulf, Shea, & Matschiner, 1998). A total of 54 participants completed the study forming six treatment groups. The independent treatment, visual feedback, was manipulated as scheduling (absolute—every practice trial or relative—every third trial) and strategies (gradual decrease of visual cues within the interface, gradual increase of visual cues within the interface, or a single consistent cue for each trial). Participants completed twelve practice trials of welding under one of six feedback manipulations; then, participants completed twelve practice trials of welding without it. Lastly, participants performed the weld task on actual equipment in a shop area. No treatment showed significant difference among groups with regard to initial learning, retention, near transfer, and far transfer measures. However, a statistical significance was found during initial learning and retention within each treatment group. Findings support empirical evidence that a variability of practice paradigm promotes learning (Lee & Carnahan, 1990; Shea & Morgan, 1979). Learner perceptions of realism suggest that novice learners perceive simulator fidelity as high, however, these perceptions may dissipate as the learner practices. Those groups that involved the greatest number of cues at the onset of practice or having cues available at every other trial reported the greatest amount of workload. All groups reported increases in perceptions of self-efficacy during practice on the simulator, but those perceptions decreased when participants performed the weld task on actual equipment. Findings suggest that contextual-interference of increasing, decreasing, or changing feedback counteracts the guidance effect of feedback as found in previous studies

Virtualized Welding Based Learning of Human Welder Behaviors for Intelligent Robotic Welding

Combining human welder (with intelligence and sensing versatility) and automated welding robots (with precision and consistency) can lead to next generation intelligent welding systems. In this dissertation intelligent welding robots are developed by process modeling / control method and learning the human welder behavior. Weld penetration and 3D weld pool surface are first accurately controlled for an automated Gas Tungsten Arc Welding (GTAW) machine. Closed-form model predictive control (MPC) algorithm is derived for real-time welding applications. Skilled welder response to 3D weld pool surface by adjusting the welding current is then modeled using Adaptive Neuro-Fuzzy Inference System (ANFIS), and compared to the novice welder. Automated welding experiments confirm the effectiveness of the proposed human response model. A virtualized welding system is then developed that enables transferring the human knowledge into a welding robot. The learning of human welder movement (i.e., welding speed) is first realized with Virtual Reality (VR) enhancement using iterative K-means based local ANFIS modeling. As a separate effort, the learning is performed without VR enhancement utilizing a fuzzy classifier to rank the data and only preserve the high ranking “correct” response. The trained supervised ANFIS model is transferred to the welding robot and the performance of the controller is examined. A fuzzy weighting based data fusion approach to combine multiple machine and human intelligent models is proposed. The data fusion model can outperform individual machine-based control algorithm and welder intelligence-based models (with and without VR enhancement). Finally a data-driven approach is proposed to model human welder adjustments in 3D (including welding speed, arc length, and torch orientations). Teleoperated training experiments are conducted in which a human welder tries to adjust the torch movements in 3D based on his observation on the real-time weld pool image feedback. The data is off-line rated by the welder and a welder rating system is synthesized. ANFIS model is then proposed to correlate the 3D weld pool characteristic parameters and welder’s torch movements. A foundation is thus established to rapidly extract human intelligence and transfer such intelligence into welding robots

Training for Open-Ended Drilling through a Virtual Reality Simulation

Virtual Reality (VR) can support effective and scalable training of psychomotor skills in manufacturing. However, many industry training modules offer experiences that are close-ended and do not allow for human error. We aim to address this gap in VR training tools for psychomotor skills training by exploring an open-ended approach to the system design. We designed a VR training simulation prototype to perform open-ended practice of drilling using a 3-axis milling machine. The simulation employs near "end-to-end" instruction through a safety module, a setup and drilling tutorial, open-ended practice complete with warnings of mistakes and failures, and a function to assess the geometries and locations of drilled holes against an engineering drawing. We developed and conducted a user study within an undergraduate-level introductory fabrication course to investigate the impact of open-ended VR practice on learning outcomes. Study results reveal positive trends, with the VR group successfully completing the machining task of drilling at a higher rate (75% vs 64%), with fewer mistakes (1.75 vs 2.14 score), and in less time (17.67 mins vs 21.57 mins) compared to the control group. We discuss our findings and limitations and implications for the design of open-ended VR training systems for learning psychomotor skills.Comment: 10 pages, 4 figures, 9 table

VR welding kit: welding training simulation in mobile virtual reality using multiple marker tracking method

Welding simulation design using virtual reality (VR) is a challenge, as numerous developments and research in the mechanical engineering fields are involved. One of the key challenges is the improvement of realism by considering a mixed system of real and virtual equipment. A conceptual design and research management framework is currently lacking which leveraging the combination of VR and marker tracking techniques. This study seeks to examine and evaluating the use of mobile VR in welding training and how multiple markers tracking methods can be incorporated to overcome the current problems in VR for welding training simulation. In this study, the VR Welding Kit application is created by utilizing the Vuforia tracking engine to provide an alternative interaction for mobile devices. The results of the experiment revealed a benchmark comparison with Oculus Quest, the high-end VR system, to investigate the efficiency of the proposed multiple marker interaction technique. Performance for both devices was recorded. The System Usability Scales (SUS) have also been used to obtain users' acceptance rates using these devices. The Simulator Sickness Questionnaire (SSQ) was used to assess the cybersickness of participants. The performance results show that mobile VR have a moderate gap completion time in seconds if compared to Oculus Quest. The SUS scored a satisfactory result which is 73.33. Besides, SSQ surveys result shows that most of the participant felt the simulation sickness was minimal