A Continuum Robot and Control Interface for Surgical Assist in Fetoscopic Interventions

Twin-twin transfusion syndrome requires interventional treatment using a fetoscopically introduced laser to sever the shared blood supply between the fetuses. This is a delicate procedure relying on small instrumentation with limited articulation to guide the laser tip and a narrow field of view to visualize all relevant vascular connections. In this letter, we report on a mechatronic design for a comanipulated instrument that combines concentric tube actuation to a larger manipulator constrained by a remote centre of motion. A stereoscopic camera is mounted at the distal tip and used for imaging. Our mechanism provides enhanced dexterity and stability of the imaging device. We demonstrate that the imaging system can be used for computing geometry and enhancing the view at the operating site. Results using electromagnetic sensors for verification and comparison to visual odometry from the distal sensor show that our system is promising and can be developed further for multiple clinical needs in fetoscopic procedures

Design and Modeling of Multi-Arm Continuum Robots

Continuum robots are snake-like systems able to deliver optimal therapies to pathologies deep inside the human cavity by following 3D complex paths. They show promise when anatomical pathways need to be traversed thanks to their enhanced flexibility and dexterity and show advantages when deployed in the field of single-port surgery. This PhD thesis concerns the development and modelling of multi-arm and hybrid continuum robots for medical interventions. The flexibility and steerability of the robot’s end-effector are achieved through concentric tube technology and push/pull technology. Medical robotic prototypes have been designed as proof of concepts and testbeds of the proposed theoretical works.System design considers the limitations and constraints that occur in the surgical procedures for which the systems were proposed for. Specifically, two surgical applications are considered. Our first prototype was designed to deliver multiple tools to the eye cavity for deep orbital interventions focusing on a currently invasive intervention named Optic Nerve Sheath Fenestration (ONSF). This thesis presents the end-to-end design, engineering and modelling of the prototype. The developed prototype is the first suggested system to tackle the challenges (limited workspace, need for enhanced flexibility and dexterity, danger for harming tissue with rigid instruments, extensive manipulation of the eye) arising in ONSF. It was designed taking into account the clinical requirements and constraints while theoretical works employing the Cosserat rod theory predict the shape of the continuum end-effector. Experimental runs including ex vivo experimental evaluations, mock-up surgical scenarios and tests with and without loading conditions prove the concept of accessing the eye cavity. Moreover, a continuum robot for thoracic interventions employing push/pull technology was designed and manufactured. The developed system can reach deep seated pathologies in the lungs and access regions in the bronchial tree that are inaccessible with rigid and straight instruments either robotically or manually actuated. A geometrically exact model of the robot that considers both the geometry of the robot and mechanical properties of the backbones is presented. It can predict the shape of the bronchoscope without the constant curvature assumption. The proposed model can also predict the robot shape and micro-scale movements accurately in contrast to the classic geometric model which provides an accurate description of the robot’s differential kinematics for large scale movements

Robotic manipulators for in situ inspections of jet engines

Jet engines need to be inspected periodically and, in some instances, repaired. Currently, some of these maintenance operations require the engine to be removed from the wing and dismantled, which has a significant associated cost. The capability of performing some of these inspections and repairs while the engine is on-wing could lead to important cost savings. However, existing technology for on-wing operations is limited, and does not suffice to satisfy some of the needs. In this work, the problem of performing on-wing operations such as inspection and repair is analysed, and after an extensive literature review, a novel robotic system for the on-wing insertion and deployment of probes or other tools is proposed. The system consists of a fine-positioner, which is a miniature and dexterous robotic manipulator; a gross-positioner, which is a device to insert the fine-positioner to the engine region of interest; an end-effector, such as a probe; a deployment mechanism, which is a passive device to ensure correct contact between probe and component; and a feedback system that provides information about the robot state for control. The research and development work conducted to address the main challenges to create this robotic system is presented in this thesis. The work is focussed on the fine-positioner, as it is the most relevant and complex part of the system. After a literature review of relevant work, and as part of the exploration of potential robot concepts for the system, the kinematic capabilities of concentric tube robots (CTRs) are first investigated. The complete set of stable trajectories that can be traced in follow-the-leader motion is discovered. A case study involving simulations and an experiment is then presented to showcase and verify the work. The research findings indicate that CTRs are not suitable for the fine-positioner. However, they show that CTRs with non-annular cross section can be used for the gross-positioner. In addition, the new trajectories discovered show promise in minimally invasive surgery (MIS). Soft robotic manipulators with fluidic actuation are then selected as the most suitable concept for the fine-positioner. The design of soft robotic manipulators with fluidic actuation is investigated from a general perspective. A general framework for the design of these devices is proposed, and a set of design principles are derived. These principles are first applied in a MIS case study to illustrate and verify the work. Finite element (FE) simulations are then reported to perform design optimisation, and thus complete the case study. The design study is then applied to determine the most suitable design for the fine-positioner. An additional analytical derivation is developed, followed by FE simulations, which extend those of the case study. Eventually, this work yields a final design of the fine-positioner. The final design found is different from existing ones, and is shown to provide an important performance improvement with respect to existing soft robots in terms of wrenches it can support. The control of soft and continuum robots relevant to the fine-positioner is also studied. The full kinematics of continuum robots with constant curvature bending and extending capabilities are first investigated, which correspond to a preliminary design concept conceived for the fine-positioner. Closed-form solutions are derived, closing an open problem. These kinematics, however, do not exactly match the final fine-positioner design selected. Thus, an alternative control approach based on closed-loop control laws is then adopted. For this, a mechanical model is first developed. Closed-loop control laws are then derived based on this mechanical model for planar operation of a segment of the fine-positioner. The control laws obtained represent the foundation for the subsequent development of control laws for a full fine-positioner operating in 3D. Furthermore, work on path planning for nonholonomic systems is also reported, and a new algorithm is presented, which can be applied for the insertion of the overall robotic system. Solutions to the other parts of the robotic system for on-wing operations are also reported. A gross-positioner consisting of a non-annular CTR is proposed. Solutions for a deployment mechanism are also presented. Potential feedback systems are outlined. In addition, methods for the fabrication of the systems are reported, and the electronics and systems required for the assembly of the different parts are described. Finally, the use of the robotic system to perform on-wing inspections in a representative case study is studied to determine the viability. Inspection strategies are shortlisted, and simulations and experiments are used to study them. The results, however, indicate that inspection is not viable since the signal to noise ratio is excessively low. Nonetheless, the robotic system proposed, and the research conducted, are still expected to be useful to perform a range of on-wing operations that require the insertion and deployment of a probe or other end-effector. In addition, the trajectories discovered for CTRs, the design found for the fine-positioner, and the advances on control, also have significant potential in MIS, where there is an important need for miniature robotic manipulators and similar devices.Open Acces

A continuum robotic platform for endoscopic non-contact laser surgery: design, control, and preclinical evaluation

The application of laser technologies in surgical interventions has been accepted in the clinical domain due to their atraumatic properties. In addition to manual application of fibre-guided lasers with tissue contact, non-contact transoral laser microsurgery (TLM) of laryngeal tumours has been prevailed in ENT surgery. However, TLM requires many years of surgical training for tumour resection in order to preserve the function of adjacent organs and thus preserve the patient’s quality of life. The positioning of the microscopic laser applicator outside the patient can also impede a direct line-of-sight to the target area due to anatomical variability and limit the working space. Further clinical challenges include positioning the laser focus on the tissue surface, imaging, planning and performing laser ablation, and motion of the target area during surgery. This dissertation aims to address the limitations of TLM through robotic approaches and intraoperative assistance. Although a trend towards minimally invasive surgery is apparent, no highly integrated platform for endoscopic delivery of focused laser radiation is available to date. Likewise, there are no known devices that incorporate scene information from endoscopic imaging into ablation planning and execution. For focusing of the laser beam close to the target tissue, this work first presents miniaturised focusing optics that can be integrated into endoscopic systems. Experimental trials characterise the optical properties and the ablation performance. A robotic platform is realised for manipulation of the focusing optics. This is based on a variable-length continuum manipulator. The latter enables movements of the endoscopic end effector in five degrees of freedom with a mechatronic actuation unit. The kinematic modelling and control of the robot are integrated into a modular framework that is evaluated experimentally. The manipulation of focused laser radiation also requires precise adjustment of the focal position on the tissue. For this purpose, visual, haptic and visual-haptic assistance functions are presented. These support the operator during teleoperation to set an optimal working distance. Advantages of visual-haptic assistance are demonstrated in a user study. The system performance and usability of the overall robotic system are assessed in an additional user study. Analogous to a clinical scenario, the subjects follow predefined target patterns with a laser spot. The mean positioning accuracy of the spot is 0.5 mm. Finally, methods of image-guided robot control are introduced to automate laser ablation. Experiments confirm a positive effect of proposed automation concepts on non-contact laser surgery.Die Anwendung von Lasertechnologien in chirurgischen Interventionen hat sich aufgrund der atraumatischen Eigenschaften in der Klinik etabliert. Neben manueller Applikation von fasergeführten Lasern mit Gewebekontakt hat sich die kontaktfreie transorale Lasermikrochirurgie (TLM) von Tumoren des Larynx in der HNO-Chirurgie durchgesetzt. Die TLM erfordert zur Tumorresektion jedoch ein langjähriges chirurgisches Training, um die Funktion der angrenzenden Organe zu sichern und damit die Lebensqualität der Patienten zu erhalten. Die Positionierung des mikroskopis chen Laserapplikators außerhalb des Patienten kann zudem die direkte Sicht auf das Zielgebiet durch anatomische Variabilität erschweren und den Arbeitsraum einschränken. Weitere klinische Herausforderungen betreffen die Positionierung des Laserfokus auf der Gewebeoberfläche, die Bildgebung, die Planung und Ausführung der Laserablation sowie intraoperative Bewegungen des Zielgebietes. Die vorliegende Dissertation zielt darauf ab, die Limitierungen der TLM durch robotische Ansätze und intraoperative Assistenz zu adressieren. Obwohl ein Trend zur minimal invasiven Chirurgie besteht, sind bislang keine hochintegrierten Plattformen für die endoskopische Applikation fokussierter Laserstrahlung verfügbar. Ebenfalls sind keine Systeme bekannt, die Szeneninformationen aus der endoskopischen Bildgebung in die Ablationsplanung und -ausführung einbeziehen. Für eine situsnahe Fokussierung des Laserstrahls wird in dieser Arbeit zunächst eine miniaturisierte Fokussieroptik zur Integration in endoskopische Systeme vorgestellt. Experimentelle Versuche charakterisieren die optischen Eigenschaften und das Ablationsverhalten. Zur Manipulation der Fokussieroptik wird eine robotische Plattform realisiert. Diese basiert auf einem längenveränderlichen Kontinuumsmanipulator. Letzterer ermöglicht in Kombination mit einer mechatronischen Aktuierungseinheit Bewegungen des Endoskopkopfes in fünf Freiheitsgraden. Die kinematische Modellierung und Regelung des Systems werden in ein modulares Framework eingebunden und evaluiert. Die Manipulation fokussierter Laserstrahlung erfordert zudem eine präzise Anpassung der Fokuslage auf das Gewebe. Dafür werden visuelle, haptische und visuell haptische Assistenzfunktionen eingeführt. Diese unterstützen den Anwender bei Teleoperation zur Einstellung eines optimalen Arbeitsabstandes. In einer Anwenderstudie werden Vorteile der visuell-haptischen Assistenz nachgewiesen. Die Systemperformanz und Gebrauchstauglichkeit des robotischen Gesamtsystems werden in einer weiteren Anwenderstudie untersucht. Analog zu einem klinischen Einsatz verfolgen die Probanden mit einem Laserspot vorgegebene Sollpfade. Die mittlere Positioniergenauigkeit des Spots beträgt dabei 0,5 mm. Zur Automatisierung der Ablation werden abschließend Methoden der bildgestützten Regelung vorgestellt. Experimente bestätigen einen positiven Effekt der Automationskonzepte für die kontaktfreie Laserchirurgie

Surgical Subtask Automation for Intraluminal Procedures using Deep Reinforcement Learning

Intraluminal procedures have opened up a new sub-field of minimally invasive surgery that use flexible instruments to navigate through complex luminal structures of the body, resulting in reduced invasiveness and improved patient benefits. One of the major challenges in this field is the accurate and precise control of the instrument inside the human body. Robotics has emerged as a promising solution to this problem. However, to achieve successful robotic intraluminal interventions, the control of the instrument needs to be automated to a large extent. The thesis first examines the state-of-the-art in intraluminal surgical robotics and identifies the key challenges in this field, which include the need for safe and effective tool manipulation, and the ability to adapt to unexpected changes in the luminal environment. To address these challenges, the thesis proposes several levels of autonomy that enable the robotic system to perform individual subtasks autonomously, while still allowing the surgeon to retain overall control of the procedure. The approach facilitates the development of specialized algorithms such as Deep Reinforcement Learning (DRL) for subtasks like navigation and tissue manipulation to produce robust surgical gestures. Additionally, the thesis proposes a safety framework that provides formal guarantees to prevent risky actions. The presented approaches are evaluated through a series of experiments using simulation and robotic platforms. The experiments demonstrate that subtask automation can improve the accuracy and efficiency of tool positioning and tissue manipulation, while also reducing the cognitive load on the surgeon. The results of this research have the potential to improve the reliability and safety of intraluminal surgical interventions, ultimately leading to better outcomes for patients and surgeons

Map-Based Localization for Unmanned Aerial Vehicle Navigation

Unmanned Aerial Vehicles (UAVs) require precise pose estimation when navigating in indoor and GNSS-denied / GNSS-degraded outdoor environments. The possibility of crashing in these environments is high, as spaces are confined, with many moving obstacles. There are many solutions for localization in GNSS-denied environments, and many different technologies are used. Common solutions involve setting up or using existing infrastructure, such as beacons, Wi-Fi, or surveyed targets. These solutions were avoided because the cost should be proportional to the number of users, not the coverage area. Heavy and expensive sensors, for example a high-end IMU, were also avoided. Given these requirements, a camera-based localization solution was selected for the sensor pose estimation. Several camera-based localization approaches were investigated. Map-based localization methods were shown to be the most efficient because they close loops using a pre-existing map, thus the amount of data and the amount of time spent collecting data are reduced as there is no need to re-observe the same areas multiple times. This dissertation proposes a solution to address the task of fully localizing a monocular camera onboard a UAV with respect to a known environment (i.e., it is assumed that a 3D model of the environment is available) for the purpose of navigation for UAVs in structured environments. Incremental map-based localization involves tracking a map through an image sequence. When the map is a 3D model, this task is referred to as model-based tracking. A by-product of the tracker is the relative 3D pose (position and orientation) between the camera and the object being tracked. State-of-the-art solutions advocate that tracking geometry is more robust than tracking image texture because edges are more invariant to changes in object appearance and lighting. However, model-based trackers have been limited to tracking small simple objects in small environments. An assessment was performed in tracking larger, more complex building models, in larger environments. A state-of-the art model-based tracker called ViSP (Visual Servoing Platform) was applied in tracking outdoor and indoor buildings using a UAVs low-cost camera. The assessment revealed weaknesses at large scales. Specifically, ViSP failed when tracking was lost, and needed to be manually re-initialized. Failure occurred when there was a lack of model features in the cameras field of view, and because of rapid camera motion. Experiments revealed that ViSP achieved positional accuracies similar to single point positioning solutions obtained from single-frequency (L1) GPS observations standard deviations around 10 metres. These errors were considered to be large, considering the geometric accuracy of the 3D model used in the experiments was 10 to 40 cm. The first contribution of this dissertation proposes to increase the performance of the localization system by combining ViSP with map-building incremental localization, also referred to as simultaneous localization and mapping (SLAM). Experimental results in both indoor and outdoor environments show sub-metre positional accuracies were achieved, while reducing the number of tracking losses throughout the image sequence. It is shown that by integrating model-based tracking with SLAM, not only does SLAM improve model tracking performance, but the model-based tracker alleviates the computational expense of SLAMs loop closing procedure to improve runtime performance. Experiments also revealed that ViSP was unable to handle occlusions when a complete 3D building model was used, resulting in large errors in its pose estimates. The second contribution of this dissertation is a novel map-based incremental localization algorithm that improves tracking performance, and increases pose estimation accuracies from ViSP. The novelty of this algorithm is the implementation of an efficient matching process that identifies corresponding linear features from the UAVs RGB image data and a large, complex, and untextured 3D model. The proposed model-based tracker improved positional accuracies from 10 m (obtained with ViSP) to 46 cm in outdoor environments, and improved from an unattainable result using VISP to 2 cm positional accuracies in large indoor environments. The main disadvantage of any incremental algorithm is that it requires the camera pose of the first frame. Initialization is often a manual process. The third contribution of this dissertation is a map-based absolute localization algorithm that automatically estimates the camera pose when no prior pose information is available. The method benefits from vertical line matching to accomplish a registration procedure of the reference model views with a set of initial input images via geometric hashing. Results demonstrate that sub-metre positional accuracies were achieved and a proposed enhancement of conventional geometric hashing produced more correct matches - 75% of the correct matches were identified, compared to 11%. Further the number of incorrect matches was reduced by 80%