Visual Sensing and Defect Detection of Gas Tungsten Arc Welding

Weld imperfections or defects such as incomplete penetration and lack of fusion are critical issues that affect the integration of welding components. The molten weld pool geometry is the major source of information related to the formation of these defects. In this dissertation, a new visual sensing system has been designed and set up to obtain weld pool images during GTAW. The weld pool dynamical behavior can be monitored using both active and passive vision method with the interference of arc light in the image significantly reduced through the narrow band pass filter and laser based auxiliary light source.Computer vision algorithms based on passive vision images were developed to measure the 3D weld pool surface geometry in real time. Specifically, a new method based on the reversed electrode image (REI) was developed to calculate weld pool surface height in real time. Meanwhile, the 2D weld pool boundary was extracted with landmarks detection algorithms. The method was verified with bead-on-plate and butt-joint welding experiments.Supervised machine learning was used to develop the capability to predict, in real-time, the incomplete penetration on thin SS304 plate with the key features extracted from weld pool images. An integrated self-adaptive close loop control system consisting the non-contact visual sensor, machine learning based defect predictor, and welding power source was developed for real-time welding penetration control for bead on plate welding. Moreover, the data driven methods were first applied to detect incomplete penetration and LOF in multi-pass U groove welding. New features extracted from reversed electrode image played the most important role to predict these defects. Finally, real time welding experiments were conducted to verify the feasibility of the developed models

Applications of a Biomechanical Patient Model for Adaptive Radiation Therapy

Biomechanical patient modeling incorporates physical knowledge of the human anatomy into the image processing that is required for tracking anatomical deformations during adaptive radiation therapy, especially particle therapy. In contrast to standard image registration, this enforces bio-fidelic image transformation. In this thesis, the potential of a kinematic skeleton model and soft tissue motion propagation are investigated for crucial image analysis steps in adaptive radiation therapy. The first application is the integration of the kinematic model in a deformable image registration process (KinematicDIR). For monomodal CT scan pairs, the median target registration error based on skeleton landmarks, is smaller than (1.6 ± 0.2) mm. In addition, the successful transferability of this concept to otherwise challenging multimodal registration between CT and CBCT as well as CT and MRI scan pairs is shown to result in median target registration error in the order of 2 mm. This meets the accuracy requirement for adaptive radiation therapy and is especially interesting for MR-guided approaches. Another aspect, emerging in radiotherapy, is the utilization of deep-learning-based organ segmentation. As radiotherapy-specific labeled data is scarce, the training of such methods relies heavily on augmentation techniques. In this work, the generation of synthetically but realistically deformed scans used as Bionic Augmentation in the training phase improved the predicted segmentations by up to 15% in the Dice similarity coefficient, depending on the training strategy. Finally, it is shown that the biomechanical model can be built-up from automatic segmentations without deterioration of the KinematicDIR application. This is essential for use in a clinical workflow

Elastic Inflatable Actuators for Soft Robotic Applications

The 20th century’s robotic systems have been made out of stiff materials and much of the developments in the field have pursued ever more accurate and dynamic robots which thrive in industrial automation settings and will probably continue to do so for many decades to come. However, the 21st century’s robotic legacy may very well become that of soft robots. This emerging domain is characterized by continuous soft structures that simultaneously fulfil the role of robotic link and robotic actuator, where prime focus is on design and fabrication of the robotic hardware instead of software control to achieve a desired operation. These robots are anticipated to take a prominent role in delicate tasks where classic robots fail, such as in minimally invasive surgery, active prosthetics and automation tasks involving delicate irregular objects. Central to the development of these robots is the fabrication of soft actuators to generate movement. This paper reviews a particularly attractive type of soft actuators that are driven by pressurized fluids. These actuators have recently gained substantial traction on the one hand due to the technology push from better simulation tools and new manufacturing technologies including soft-lithography and additive manufacturing, and on the other hand by a market pull from the applications listed above. This paper provides an overview of the different advanced soft actuator configurations, their design, fabrication and applications.This research is supported by the Fund for Scientific Research-Flanders (FWO), and the European Research Council (ERC starting grant HIENA)

Elastic Inflatable Actuators for Soft Robotic Applications

The 20th century’s robotic systems have been made out of stiff materials and much of the developments in the field have pursued ever more accurate and dynamic robots which thrive in industrial automation settings and will probably continue to do so for many decades to come. However, the 21st century’s robotic legacy may very well become that of soft robots. This emerging domain is characterized by continuous soft structures that simultaneously fulfil the role of robotic link and robotic actuator, where prime focus is on design and fabrication of the robotic hardware instead of software control to achieve a desired operation. These robots are anticipated to take a prominent role in delicate tasks where classic robots fail, such as in minimally invasive surgery, active prosthetics and automation tasks involving delicate irregular objects. Central to the development of these robots is the fabrication of soft actuators to generate movement. This paper reviews a particularly attractive type of soft actuators that are driven by pressurized fluids. These actuators have recently gained substantial traction on the one hand due to the technology push from better simulation tools and new manufacturing technologies including soft-lithography and additive manufacturing, and on the other hand by a market pull from the applications listed above. This paper provides an overview of the different advanced soft actuator configurations, their design, fabrication and applications.This research is supported by the Fund for Scientific Research-Flanders (FWO), and the European Research Council (ERC starting grant HIENA)

Neural radiance fields in the industrial and robotics domain: applications, research opportunities and use cases

The proliferation of technologies, such as extended reality (XR), has increased the demand for high-quality three-dimensional (3D) graphical representations. Industrial 3D applications encompass computer-aided design (CAD), finite element analysis (FEA), scanning, and robotics. However, current methods employed for industrial 3D representations suffer from high implementation costs and reliance on manual human input for accurate 3D modeling. To address these challenges, neural radiance fields (NeRFs) have emerged as a promising approach for learning 3D scene representations based on provided training 2D images. Despite a growing interest in NeRFs, their potential applications in various industrial subdomains are still unexplored. In this paper, we deliver a comprehensive examination of NeRF industrial applications while also providing direction for future research endeavors. We also present a series of proof-of-concept experiments that demonstrate the potential of NeRFs in the industrial domain. These experiments include NeRF-based video compression techniques and using NeRFs for 3D motion estimation in the context of collision avoidance. In the video compression experiment, our results show compression savings up to 48\% and 74\% for resolutions of 1920x1080 and 300x168, respectively. The motion estimation experiment used a 3D animation of a robotic arm to train Dynamic-NeRF (D-NeRF) and achieved an average peak signal-to-noise ratio (PSNR) of disparity map with the value of 23 dB and an structural similarity index measure (SSIM) 0.97

Usage of Digital Image Correlation in Single and Multiple Stereo Systems for Strain and Shape Measurement

This dissertation details the application of digital image correlation (DIC) in novel methods for mechanical testing and shape characterization. The first usage manifested in the development of a novel composite test specimen that provided the ability to characterize the matrix compression energy release rate at initiation and subsequent crack propagation. Experimental efforts with finite element derived test specimens, augmented with a stereo DIC system, verified the correct mechanical loading and permitted material property characterization. DIC was critical in enabling matrix crack evolution to be accurately measured. The second novel work resulted in a low-cost metrology system methodically designed for the shape characterization of additively manufactured powdered metal objects using a circular camera array and the application of digital image correlation. The multi-stereo camera system provides data rich, micron scale, object-shape point clouds for both green and sintered components using only their native surface texture. Excellent measurement performance was verified against precision test objects. The last body of work merged variable focus deformable lenses with digital image correlation to overcome limitations in the available focal distance. This novel combination of technologies provided enhanced capabilities for both mechanical testing and shape characterization. Objects of various sizes and at multiple locations can be effectively imaged allowing their shape to be recovered and/or strain fields measured. The appropriate lens calibration for two commercially available variable focus lenses was determined and their performance was demonstrated using multiple examples. This ground-breaking work provides a paradigm-change in the utilization of stereo DIC and can straightforwardly be combined in multi-DIC systems, profoundly expanding their functional utility

Robust 3D Object Pose Estimation and Tracking from Monocular Images in Industrial Environments

Recent advances in Computer Vision are changing our way of living and enabling new applications for both leisure and professional use. Regrettably, in many industrial domains the spread of state-of-the-art technologies is made challenging by the abundance of nuisances that corrupt existing techniques beyond the required dependability. This is especially true for object localization and tracking, that is, the problem of detecting the presence of objects on images and videos and estimating their pose. This is a critical task for applications such as Augmented Reality (AR), robotic autonomous navigation, robotic object grasping, or production quality control; unfortunately, the reliability of existing techniques is harmed by visual features such as the abundance of specular and poorly textured objects, cluttered scenes, or artificial and in-homogeneous lighting. In this thesis, we propose two methods for robustly estimating the pose of a rigid object under the challenging conditions typical of industrial environments. Both methods rely on monocular images to handle metallic environments, on which depth cameras would fail; both are conceived with a limited computational and memory footprint, so that they are suitable for real-time applications such as AR. We test our methods on datasets issued from real user case scenarios, exhibiting challenging conditions. The first method is based on a global image alignment framework and a robust dense descriptor. Its global approach makes it robust in presence of local artifacts such as specularities appearing on metallic objects, ambiguous patterns like screws or wires, and poorly textured objects. Employing a global approach avoids the need of reliably detecting and matching local features across images, that become ill-conditioned tasks in the considered environments; on the other hand, current methods based on dense image alignment usually rely on luminous intensities for comparing the pixels, which is not robust in presence of challenging illumination artifacts. We show how the use of a dense descriptor computed as a non-linear function of luminous intensities, that we refer to as ``Descriptor Fields'', greatly enhances performances at a minimal computational overhead. Their low computational complexity and their ease of implementation make Descriptor Fields suitable for replacing intensities in a wide number of state-of-the-art techniques based on dense image alignment. Relying on a global approach is appropriate for overcoming local artifacts, but it can be un-effective when the target object undergoes extreme occlusions in cluttered environments. For this reason, we propose a second approach based on the detection of discriminative object parts. At the core of our approach is a novel representation for the 3D pose of the parts, that allows us to predict the 3D pose of the object even when only a single part is visible; when several parts are visible, we can easily combine them to compute a better pose of the object. The 3D pose we obtain is usually very accurate, even when only few parts are visible. We show how to use this representation in a robust 3D tracking framework. In addition to extensive comparisons with the state-of-the-art, we demonstrate our method on a practical Augmented Reality application for maintenance assistance in the ATLAS particle detector at CERN

Using the Fringe Field of MRI Scanner for the Navigation of Microguidewires in the Vascular System

Le traitement du cancer, la prévention des accidents vasculaires cérébraux et le diagnostic ou le traitement des maladies vasculaires périphériques sont tous des cas d'application d'interventions à base de cathéter par le biais d'un traitement invasif minimal. Cependant, la pratique du cathétérisme est généralement pratiquée manuellement et dépend fortement de l'expérience et des compétences de l'interventionniste. La robotisation du cathétérisme a été étudiée pour faciliter la procédure en augmentant les niveaux d’autonomie par rapport à cette pratique clinique. En ce qui concerne ce problème, un des problèmes concerne le placement super sélectif du cathéter dans les artères plus étroites nécessitant une miniaturisation de l'instrument cathéter / fil de guidage attaché. Un microguide qui fonctionne dans des vaisseaux sanguins étroits et tortueux subit différentes forces mécaniques telles que le frottement avec la paroi du vaisseau. Ces forces peuvent empêcher la progression de la pointe du fil de guidage dans les vaisseaux. Une méthode proposée consiste à appliquer une force de traction à la pointe du microguide pour diriger et insérer le dispositif tout en poussant l’instrument attaché à partir de l’autre extrémité n’est plus pratique, et à exploiter le gradient du champ de franges IRM surnommé Fringe Field Navigation (FFN ) est proposée comme solution pour assurer cet actionnement. Le concept de FFN repose sur le positionnement d'un patient sur six DOF dans le champ périphérique du scanner IRM afin de permettre un actionnement directionnel pour la navigation du fil-guide. Ce travail rend compte des développements requis pour la mise en oeuvre de la FFN et l’étude du potentiel et des possibilités qu’elle offre au cathétérisme, en veillant au renforcement de l’autonomie. La cartographie du champ de franges d'un scanner IRM 3T est effectuée et la structure du champ de franges en ce qui concerne son uniformité locale est examinée. Une méthode pour la navigation d'un fil de guidage le long d'un chemin vasculaire souhaité basée sur le positionnement robotique du patient à six DOF est développée. Des expériences de FFN guidées par rayons X in vitro et in vivo sur un modèle porcin sont effectuées pour naviguer dans un fil de guidage dans la multibifurcation et les vaisseaux étroits. Une caractéristique unique de FFN est le haut gradient du champ magnétique. Il est démontré in vitro et in vivo que cette force surmonte le problème de l'insertion d'un fil microguide dans des vaisseaux tortueux et étroits pour permettre de faire avancer le fil-guide avec une distale douce au-delà de la limite d'insertion manuelle. La robustesse de FFN contre les erreurs de positionnement du patient est étudiée en relation avec l'uniformité locale dans le champ périphérique. La force élevée du champ magnétique disponible dans le champ de franges IRM peut amener les matériaux magnétiques doux à son état de saturation. Ici, le concept d'utilisation d'un ressort est présenté comme une alternative vi déformable aux aimants permanents solides pour la pointe du fil-guide. La navigation d'un microguide avec une pointe de ressort en structure vasculaire complexe est également réalisée in vitro. L'autonomie de FFN en ce qui concerne la planification d'une procédure avec autonomie de tâche obtenue dans ce travail augmente le potentiel de FFN en automatisant certaines étapes d'une procédure. En conclusion, FFN pour naviguer dans les microguides dans la structure vasculaire complexe avec autonomie pour effectuer le positionnement du patient et contrôler l'insertion du fil de guidage - avec démonstration in vivo dans un modèle porcin - peut être considéré comme un nouvel outil robotique facilitant le cathétérisme vasculaire. tout en aidant à cibler les vaisseaux lointains dans le système vasculaire.----------ABSTRACT Treatment of cancer, prevention of stroke, and diagnosis or treatment of peripheral vascular diseases are all the cases of application of catheter-based interventions through a minimal-invasive treatment. However, performing catheterization is generally practiced manually, and it highly depends on the experience and the skills of the interventionist. Robotization of catheterization has been investigated to facilitate the procedure by increasing the levels of autonomy to this clinical practice. Regarding it, one issue is the super selective placement of the catheter in the narrower arteries that require miniaturization of the tethered catheter/guidewire instrument. A microguidewire that operates in narrow and tortuous blood vessels experiences different mechanical forces like friction with the vessel wall. These forces can prevent the advancement of the tip of the guidewire in the vessels. A proposed method is applying a pulling force at the tip of the microguidewire to steer and insert the device while pushing the tethered instrument from the other end is no longer practical, and exploiting the gradient of the MRI fringe field dubbed as Fringe Field Navigation (FFN) is proposed as a solution to provide this actuation. The concept of FFN is based on six DOF positioning of a patient in the fringe field of the MRI scanner to enable directional actuation for the navigation of the guidewire. This work reports on the required developments for implementing FFN and investigating the potential and the possibilities that FFN introduces to the catheterization, with attention to enhancing the autonomy. Mapping the fringe field of a 3T MRI scanner is performed, and the structure of the fringe field regarding its local uniformity is investigated. A method for the navigation of a guidewire along a desired vascular path based on six DOF robotic patient positioning is developed. In vitro and in vivo x-ray Guided FFN experiments on a swine model of are performed to navigate a guidewire in the multibifurcation and narrow vessels. A unique feature of FFN is the high gradient of the magnetic field. It is demonstrated in vitro and in vivo that this force overcomes the issue of insertion of a microguidewire in tortuous and narrow vessels to enable advancing the guidewire with a soft distal beyond the limit of manual insertion. Robustness of FFN against the error in the positioning of the patient is investigated in relation to the local uniformity in the fringe field. The high strength of the magnetic field available in MRI fringe field can bring soft magnetic materials to its saturation state. Here, the concept of using a spring is introduced as a deformable alternative to solid permanent magnets for the tip of the guidewire. Navigation of a microguidewire with a viii spring tip in complex vascular structure is also performed in vitro. The autonomy of FFN regarding planning a procedure with Task Autonomy achieved in this work enhances the potential of FFN by automatization of certain steps of a procedure. As a conclusion, FFN to navigate microguidewires in the complex vascular structure with autonomy in performing tasks of patient positioning and controlling the insertion of the guidewire – with in vivo demonstration in swine model – can be considered as a novel robotic tool for facilitating the vascular catheterization while helping to target remote vessels in the vascular system

Adaptive geometry transformation and repair methodology for hybrid manufacturing

With hybrid manufacturing maturing into a commercial scale, industries are pushing to integrate and fully utilize this new technology in their production facilities. Using the capability to interleave additive and subtractive manufacturing, these systems provide an opportunity to perform component repair through additive material deposition and resurfacing via machining. This is particularly attractive to industries which utilize complex, often freeform, components which require a large capital investment, such as the aerospace and mold and die industries. However, in service these components may experience unique distortions or wear, and therefore each require a unique repair strategy. This work seeks to create an adaptive transformation method for part geometry, which can adapt the process to match the needs of an individual component within the context of a commercial hybrid manufacturing system using currently available on machine inspection technology; greatly improving the efficiency of repair processes. To accomplish this, a new methodology for the adaptation of a nominal CAD geometry to a component is presented which combines data registration and reverse engineering strategies for aero engine components. The accuracy of this deformation method is first examined, then simulations are completed to explore the potential efficiency gains in both the additive and subtractive phases of a hybrid repair process.M.S