Vision-based Manipulation of Deformable and Rigid Objects Using Subspace Projections of 2D Contours

This paper proposes a unified vision-based manipulation framework using image contours of deformable/rigid objects. Instead of using human-defined cues, the robot automatically learns the features from processed vision data. Our method simultaneously generates---from the same data---both, visual features and the interaction matrix that relates them to the robot control inputs. Extraction of the feature vector and control commands is done online and adaptively, with little data for initialization. The method allows the robot to manipulate an object without knowing whether it is rigid or deformable. To validate our approach, we conduct numerical simulations and experiments with both deformable and rigid objects

Shared control for natural motion and safety in hands-on robotic surgery

Hands-on robotic surgery is where the surgeon controls the tool's motion by applying forces and torques to the robot holding the tool, allowing the robot-environment interaction to be felt though the tool itself. To further improve results, shared control strategies are used to combine the strengths of the surgeon with those of the robot. One such strategy is active constraints, which prevent motion into regions deemed unsafe or unnecessary. While research in active constraints on rigid anatomy has been well-established, limited work on dynamic active constraints (DACs) for deformable soft tissue has been performed, particularly on strategies which handle multiple sensing modalities. In addition, attaching the tool to the robot imposes the end effector dynamics onto the surgeon, reducing dexterity and increasing fatigue. Current control policies on these systems only compensate for gravity, ignoring other dynamic effects. This thesis presents several research contributions to shared control in hands-on robotic surgery, which create a more natural motion for the surgeon and expand the usage of DACs to point clouds. A novel null-space based optimization technique has been developed which minimizes the end effector friction, mass, and inertia of redundant robots, creating a more natural motion, one which is closer to the feeling of the tool unattached to the robot. By operating in the null-space, the surgeon is left in full control of the procedure. A novel DACs approach has also been developed, which operates on point clouds. This allows its application to various sensing technologies, such as 3D cameras or CT scans and, therefore, various surgeries. Experimental validation in point-to-point motion trials and a virtual reality ultrasound scenario demonstrate a reduction in work when maneuvering the tool and improvements in accuracy and speed when performing virtual ultrasound scans. Overall, the results suggest that these techniques could increase the ease of use for the surgeon and improve patient safety.Open Acces

Efficient techniques for soft tissue modeling and simulation

Performing realistic deformation simulations in real time is a challenging problem in computer graphics. Among numerous proposed methods including Finite Element Modeling and ChainMail, we have implemented a mass spring system because of its acceptable accuracy and speed. Mass spring systems have, however, some drawbacks such as, the determination of simulation coefficients with their iterative nature. Given the correct parameters, mass spring systems can accurately simulate tissue deformations but choosing parameters that capture nonlinear deformation behavior is extremely difficult. Since most of the applications require a large number of elements i. e. points and springs in the modeling process it is extremely difficult to reach realtime performance with an iterative method. We have developed a new parameter identification method based on neural networks. The structure of the mass spring system is modified and neural networks are integrated into this structure. The input space consists of changes in spring lengths and velocities while a "teacher" signal is chosen as the total spring force, which is expressed in terms of positional changes and applied external forces. Neural networks are trained to learn nonlinear tissue characteristics represented by spring stiffness and damping in the mass spring algorithm. The learning algorithm is further enhanced by an adaptive learning rate, developed particularly for mass spring systems. In order to avoid the iterative approach in deformation simulations we have developed a new deformation algorithm. This algorithm defines the relationships between points and springs and specifies a set of rules on spring movements and deformations. These rules result in a deformation surface, which is called the search space. The deformation algorithm then finds the deformed points and springs in the search space with the help of the defined rules. The algorithm also sets rules on each element i. e. triangle or tetrahedron so that they do not pass through each other. The new algorithm is considerably faster than the original mass spring systems algorithm and provides an opportunity for various deformation applications. We have used mass spring systems and the developed method in the simulation of craniofacial surgery. For this purpose, a patient-specific head model was generated from MRI medical data by applying medical image processing tools such as, filtering, the segmentation and polygonal representation of such model is obtained using a surface generation algorithm. Prism volume elements are generated between the skin and bone surfaces so that different tissue layers are included to the head model. Both methods produce plausible results verified by surgeons

Closed-Loop Planning and Control of Steerable Medical Needles

Steerable needles have the potential to increase the effectiveness of needle-based clinical procedures such as biopsy, drug delivery, and radioactive seed implantation for cancer treatment. These needles can trace curved paths when inserted into tissue, thereby increasing maneuverability and targeting accuracy while reaching previously inaccessible targets that are behind sensitive or impenetrable anatomical regions. Guiding these flexible needles along an intended path requires continuously inserting and twisting the needle at its base, which is not intuitive for a human operator. In addition, the needle often deviates from its intended trajectory due to factors such as tissue deformation, needle-tissue interaction, noisy actuation and sensing, modeling errors, and involuntary patient motions. These challenges can be addressed with the assistance of robotic systems that automatically compensate for these perturbations by performing motion planning and feedback control of the needle in a closed-loop fashion under sensory feedback. We present two approaches for efficient closed-loop guidance of steerable needles to targets within clinically acceptable accuracy while safely avoiding sensitive or impenetrable anatomical structures. The first approach uses a fast motion planning algorithm that unifies planning and control by continuously replanning, enabling correction for perturbations as they occur. We evaluate our method using a needle steering system in phantom and ex vivo animal tissues. The second approach integrates motion planning and feedback control of steerable needles in highly deformable environments. We demonstrate that this approach significantly improves the probability of success compared to prior approaches that either consider uncertainty or deformations but not both simultaneously. We also propose a data-driven method to estimate parameters of stochastic models of steerable needle motion. These models can be used to create realistic medical simulators for clinicians wanting to train for steerable needle procedures and to improve the effectiveness of existing planning and control methods. This dissertation advances the state of the art in planning and control of steerable needles and is an important step towards realizing needle steering in clinical practice. The methods developed in this dissertation also generalize to important applications beyond medical needle steering, such as manipulating deformable objects and control of mobile robots.Doctor of Philosoph

Multi-rotor Aerial Vehicles in Physical Interactions: A Survey

Research on Multi-rotor Aerial Vehicles (MAVs) has experienced remarkable advancements over the past two decades, propelling the field forward at an accelerated pace. Through the implementation of motion control and the integration of specialized mechanisms, researchers have unlocked the potential of MAVs to perform a wide range of tasks in diverse scenarios. Notably, the literature has highlighted the distinctive attributes of MAVs that endow them with a competitive edge in physical interaction when compared to other robotic systems. In this survey, we present a categorization of the various types of physical interactions in which MAVs are involved, supported by comprehensive case studies. We examine the approaches employed by researchers to address different challenges using MAVs and their applications, including the development of different types of controllers to handle uncertainties inherent in these interactions. By conducting a thorough analysis of the strengths and limitations associated with different methodologies, as well as engaging in discussions about potential enhancements, this survey aims to illuminate the path for future research focusing on MAVs with high actuation capabilities

Bin-picking de precisão usando um sensor 3D e um sensor laser 1D

The technique that is being used by a robot to grab objects that are randomly placed inside a box or on a pallet is called bin-picking. This process is of great interest in an industrial environment as it provides enhanced automation, increased production and cost reduction. Bin-picking has evolved greatly over the years due to tremendous strides empowered by advanced vision technology, software, and gripping solutions which are in constant development. However, the creation of a versatile system, capable of collecting any type of object without deforming it, regardless of the disordered environment around it, remains a challenge. To this goal, the use of 3D perception is unavoidable. Still, the information acquired by some lower cost 3D sensors is not very precise; therefore, the combination of this information with the one of other devices is an approach already in study. The main goal of this work is to develop a solution for the execution of a precise bin-picking process capable of grasping small and fragile objects without breaking or deforming them. This may be done by combining the information provided by two sensors: one 3D sensor (Kinect) used to analyse the workspace and identify the object, and a 1D laser sensor to determine the exact distance to the object when approaching it. Additionally, the developed system may be placed at the end of a manipulator in order to become an active perception unit. Once the global system of sensors, their controllers and the robotic manipulator are integrated into a ROS (Robot Operating System) infrastructure, the data provided by the sensors can be analysed and combined to provide a bin-picking solution. Finally, the testing phase demonstrated the viability and the reliability of the developed bin-picking process.À tecnologia usada por um robô para agarrar objetos que estão dispostos de forma aleatória dentro de uma caixa ou sobre uma palete chama-se binpicking. Este processo é de grande interesse para a industria uma vez que oferece maior autonomia, aumento de produção e redução de custos. O binpicking tem evoluido de forma significativa ao longo dos anos graças aos avanços possibilitados pelo desenvolvimento tecnológico na área da visão, software e soluções de diferentes garras que estão em constante evolução. Contudo, a criação de um sistema versátil, capaz de agarrar qualquer tipo de objeto sem o deformar, independentemente do ambiente desordenado à sua volta, continua a ser o principal objetivo. Para esse fim, o recurso à perceção 3D é imprescindível. Ainda assim, a informação adquirida por sensores 3D não é muito precisa e, por isso, a combinação deste com a de outros dispositivos é uma abordagem ainda em estudo. O objetivo principal deste trabalho é então desenvolver uma solução para a execução de um processo de bin-picking capaz de agarrar objetos pequenos e frágeis sem os partir ou deformar. Isto poderá ser feito através da combinação entre a informação proveniente de dois sensores: um sensor 3D (Kinect) usado para analisar o espaço de trabalho e identificar o objeto, e um sensor laser 1D usado para determinar a sua distância exata e assim se poder aproximar. Adicionalmente, o sistema desenvolvido pode ser acoplado a um manipulador de forma a criar uma unidade de perceção ativa. Uma vez tendo um sistema global de sensores, os seus controladores e o manipulador robótico integrados numa infraestrutura ROS (Robot Operating System), os dados fornecidos pelos sensores podem ser analisados e combinados, e uma solução de bin-picking pode ser desenvolvida. Por último, a fase de testes demonstrou, depois de alguns ajustes nas medidas do sensor laser, a viabilidade e fiabilidade do processo de bin-picking desenvolvido.Mestrado em Engenharia Mecânic

Efficient techniques for soft tissue modeling and simulation

Performing realistic deformation simulations in real time is a challenging problem in computer graphics. Among numerous proposed methods including Finite Element Modeling and ChainMail, we have implemented a mass spring system because of its acceptable accuracy and speed. Mass spring systems have, however, some drawbacks such as, the determination of simulation coefficients with their iterative nature. Given the correct parameters, mass spring systems can accurately simulate tissue deformations but choosing parameters that capture nonlinear deformation behavior is extremely difficult. Since most of the applications require a large number of elements i. e. points and springs in the modeling process it is extremely difficult to reach realtime performance with an iterative method. We have developed a new parameter identification method based on neural networks. The structure of the mass spring system is modified and neural networks are integrated into this structure. The input space consists of changes in spring lengths and velocities while a "teacher" signal is chosen as the total spring force, which is expressed in terms of positional changes and applied external forces. Neural networks are trained to learn nonlinear tissue characteristics represented by spring stiffness and damping in the mass spring algorithm. The learning algorithm is further enhanced by an adaptive learning rate, developed particularly for mass spring systems. In order to avoid the iterative approach in deformation simulations we have developed a new deformation algorithm. This algorithm defines the relationships between points and springs and specifies a set of rules on spring movements and deformations. These rules result in a deformation surface, which is called the search space. The deformation algorithm then finds the deformed points and springs in the search space with the help of the defined rules. The algorithm also sets rules on each element i. e. triangle or tetrahedron so that they do not pass through each other. The new algorithm is considerably faster than the original mass spring systems algorithm and provides an opportunity for various deformation applications. We have used mass spring systems and the developed method in the simulation of craniofacial surgery. For this purpose, a patient-specific head model was generated from MRI medical data by applying medical image processing tools such as, filtering, the segmentation and polygonal representation of such model is obtained using a surface generation algorithm. Prism volume elements are generated between the skin and bone surfaces so that different tissue layers are included to the head model. Both methods produce plausible results verified by surgeons.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

A survey of robot manipulation in contact

In this survey, we present the current status on robots performing manipulation tasks that require varying contact with the environment, such that the robot must either implicitly or explicitly control the contact force with the environment to complete the task. Robots can perform more and more manipulation tasks that are still done by humans, and there is a growing number of publications on the topics of (1) performing tasks that always require contact and (2) mitigating uncertainty by leveraging the environment in tasks that, under perfect information, could be performed without contact. The recent trends have seen robots perform tasks earlier left for humans, such as massage, and in the classical tasks, such as peg-in-hole, there is a more efficient generalization to other similar tasks, better error tolerance, and faster planning or learning of the tasks. Thus, in this survey we cover the current stage of robots performing such tasks, starting from surveying all the different in-contact tasks robots can perform, observing how these tasks are controlled and represented, and finally presenting the learning and planning of the skills required to complete these tasks