Medical SLAM in an autonomous robotic system

One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-operative morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilities by observing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted instruments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This thesis addresses the ambitious goal of achieving surgical autonomy, through the study of the anatomical environment by Initially studying the technology present and what is needed to analyze the scene: vision sensors. A novel endoscope for autonomous surgical task execution is presented in the first part of this thesis. Which combines a standard stereo camera with a depth sensor. This solution introduces several key advantages, such as the possibility of reconstructing the 3D at a greater distance than traditional endoscopes. Then the problem of hand-eye calibration is tackled, which unites the vision system and the robot in a single reference system. Increasing the accuracy in the surgical work plan. In the second part of the thesis the problem of the 3D reconstruction and the algorithms currently in use were addressed. In MIS, simultaneous localization and mapping (SLAM) can be used to localize the pose of the endoscopic camera and build ta 3D model of the tissue surface. Another key element for MIS is to have real-time knowledge of the pose of surgical tools with respect to the surgical camera and underlying anatomy. Starting from the ORB-SLAM algorithm we have modified the architecture to make it usable in an anatomical environment by adding the registration of the pre-operative information of the intervention to the map obtained from the SLAM. Once it has been proven that the slam algorithm is usable in an anatomical environment, it has been improved by adding semantic segmentation to be able to distinguish dynamic features from static ones. All the results in this thesis are validated on training setups, which mimics some of the challenges of real surgery and on setups that simulate the human body within Autonomous Robotic Surgery (ARS) and Smart Autonomous Robotic Assistant Surgeon (SARAS) projects

Medical SLAM in an autonomous robotic system

One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-operative morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilities by observing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted instruments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This thesis addresses the ambitious goal of achieving surgical autonomy, through the study of the anatomical environment by Initially studying the technology present and what is needed to analyze the scene: vision sensors. A novel endoscope for autonomous surgical task execution is presented in the first part of this thesis. Which combines a standard stereo camera with a depth sensor. This solution introduces several key advantages, such as the possibility of reconstructing the 3D at a greater distance than traditional endoscopes. Then the problem of hand-eye calibration is tackled, which unites the vision system and the robot in a single reference system. Increasing the accuracy in the surgical work plan. In the second part of the thesis the problem of the 3D reconstruction and the algorithms currently in use were addressed. In MIS, simultaneous localization and mapping (SLAM) can be used to localize the pose of the endoscopic camera and build ta 3D model of the tissue surface. Another key element for MIS is to have real-time knowledge of the pose of surgical tools with respect to the surgical camera and underlying anatomy. Starting from the ORB-SLAM algorithm we have modified the architecture to make it usable in an anatomical environment by adding the registration of the pre-operative information of the intervention to the map obtained from the SLAM. Once it has been proven that the slam algorithm is usable in an anatomical environment, it has been improved by adding semantic segmentation to be able to distinguish dynamic features from static ones. All the results in this thesis are validated on training setups, which mimics some of the challenges of real surgery and on setups that simulate the human body within Autonomous Robotic Surgery (ARS) and Smart Autonomous Robotic Assistant Surgeon (SARAS) projects

Characterisation and State Estimation of Magnetic Soft Continuum Robots

Minimally invasive surgery has become more popular as it leads to less bleeding, scarring, pain, and shorter recovery time. However, this has come with counter-intuitive devices and steep surgeon learning curves. Magnetically actuated Soft Continuum Robots (SCR) have the potential to replace these devices, providing high dexterity together with the ability to conform to complex environments and safe human interactions without the cognitive burden for the clinician. Despite considerable progress in the past decade in their development, several challenges still plague SCR hindering their full realisation. This thesis aims at improving magnetically actuated SCR by addressing some of these challenges, such as material characterisation and modelling, and sensing feedback and localisation. Material characterisation for SCR is essential for understanding their behaviour and designing effective modelling and simulation strategies. In this work, the material properties of commonly employed materials in magnetically actuated SCR, such as elastic modulus, hyper-elastic model parameters, and magnetic moment were determined. Additionally, the effect these parameters have on modelling and simulating these devices was investigated. Due to the nature of magnetic actuation, localisation is of utmost importance to ensure accurate control and delivery of functionality. As such, two localisation strategies for magnetically actuated SCR were developed, one capable of estimating the full 6 degrees of freedom (DOFs) pose without any prior pose information, and another capable of accurately tracking the full 6-DOFs in real-time with positional errors lower than 4~mm. These will contribute to the development of autonomous navigation and closed-loop control of magnetically actuated SCR

Development of a fiber-based shape sensor for navigating flexible medical tools

Robot-assisted minimally invasive surgical procedure (RAMIS) is a subfield of minimally invasive surgeries with enhanced manual dexterity, manipulability, and intraoperative image guidance. In typical robotic surgeries, it is common to use rigid instruments with functional articulating tips. However, in some operations where no adequate and direct access to target anatomies is available, continuum robots can be more practical, as they provide curvilinear and flexible access. However, their inherent deformable design makes it difficult to accurately estimate their 3D shape during the operation in real-time. Despite extensive model-based research that relies on kinematics and mechanics, accurate shape sensing of continuum robots remains challenging. The state-of-the-art tracking technologies, including optical trackers, EM tracking systems, and intraoperative imaging modalities, are also unsuitable for this task, as they all have shortcomings. Optical fiber shape sensing solutions offer various advantages compared to other tracking modalities and can provide high-resolution shape measurements in real-time. However, commercially available fiber shape sensors are expensive and have limited accuracy. In this thesis, we propose two cost-effective fiber shape sensing solutions based on multiple single-mode fibers with FBG (fiber Bragg grating) arrays and eccentric FBGs. First, we present the fabrication and calibration process of two shape sensing prototypes based on multiple single-mode fibers with semi-rigid and super-elastic substrates. Then, we investigate the sensing mechanism of edge-FBGs, which are eccentric Bragg gratings inscribed off-axis in the fiber's core. Finally, we present a deep learning algorithm to model edge-FBG sensors that can directly predict the sensor's shape from its signal and does not require any calibration or shape reconstruction steps. In general, depending on the target application, each of the presented fiber shape sensing solutions can be used as a suitable tracking device. The developed fiber sensor with the semi-rigid substrate has a working channel in the middle and can accurately measure small deflections with an average tip error of 2.7 mm. The super-elastic sensor is suitable for measuring medium to large deflections, where a centimeter range tip error is still acceptable. The tip error in such super-elastic sensors is higher compared to semi-rigid sensors (9.9-16.2 mm in medium and large deflections, respectively), as there is a trade-off between accuracy and flexibility in substrate-based fiber sensors. Edge-FBG sensor, as the best performing sensing mechanism among the investigated fiber shape sensors, can achieve a tip accuracy of around 2 mm in complex shapes, where the fiber is heavily deflected. The developed edge-FBG shape sensing solution can compete with the state-of-the-art distributed fiber shape sensors that cost 30 times more

Optical Multicore Fiber Shape Sensors. A numerical and experimental performance assessment

[EN] Structural Health Monitoring (SHM) is a discipline that quantitatively assesses the integrity and performance of infrastructures, relying on sensors, and support the development of efficient Maintenance and Rehabilitation (M&R) plans. Optical Multicore Fiber (MCF) Shape Sensors offer an innovative alternative to traditional methods and enable the reconstruction of the deformed shape of structures directly and in real-time, with no need of computation models or visual contact and exploiting all the advantages of Optical Fiber Sensors (OFS) technology. Despite the intense research efforts centered on this topic by research groups worldwide, a comprehensive investigation on the parameters that influence the performance of these sensors has not been conducted yet. The first part of the thesis presents a numerical study that examines the effects of strain measurement accuracy and core position errors on the performance of optical multicore fiber shape sensors in sensing three-dimensional curvature, which is at the basis of shape reconstruction. The analysis reproduces the strain measurement process using Monte Carlo Method (MCM) and identifies several parameters which play a key role in the phenomenon, including core spacing (distance between outer cores and sensor axis), number of cores and curvature measured. Finally, a set of predictive models were calibrated, by fitting the results of the simulations, to predict the sensors performance. Afterward, an experimental study is proposed to evaluate the performance of optical multicore fiber in sensing shape, with particular focus on the influence of strain sensors length. Two shape sensors were fabricated, by inscribing long (8.0 mm) and short (1.5 mm) Fiber Bragg Gratings (FBG) into the cores of a multicore seven-core fiber. Thus, the performance of the two sensors was assessed and compared, at all the necessary phases for shape reconstruction: strain sensing, curvature calculation and shape reconstruction. To conclude, an innovative approach, based on the Saint-Venant's Torsion Theory, is presented to determine the twisting of multicore fiber and to compensate the errors due to twisting during shape reconstruction. The efficiency of the theoretical approach was then corroborated performing a series of twisting tests on a shape sensor, fabricated by inscribing FBGs sensors into an optical spun multicore seven-core fiber. The investigation of the mechanical behavior of multicore optical shape sensors has synergically involved diverse disciplines: Solid Mechanics, Photonics, Statistics and Data Analysis. Such multidisciplinary research has arisen from the prolific cooperation between the Institutes of the Institute of Science and Technology of Concrete (ICITECH) and the Institute of Telecommunications and Multimedia Applications (iTEAM) - Photonics Research Labs (PRL) - of Universitat Politècnica de València (UPV), in addition to valuable collaboration with other members of the European ITN-FINESSE project, to which this work belongs. This research work aims to enhance the performance optical multicore fiber shape sensors and support the development of new sensor geometries, with great potential for structural health monitoring applications.[ES] La Monitorización de la Salud Estructural (MSE) evalúa cuantitativamente la integridad y el comportamiento de las infraestructuras y permite desarrollar planes eficaces de Mantenimiento y Rehabilitación (M&R), utilizando los datos de los sensores. Sensores de forma basados en fibra óptica multinúcleo ofrecen una alternativa a los métodos tradicionales y permiten la reconstrucción de la deformada de estructuras de forma directa y en tiempo real, sin necesidad de modelos de cálculo o contacto visual y con todas las ventajas de la tecnología de los Sensores de Fibra Óptica (SFO). A pesar de los grandes esfuerzos en la investigación centrada en este tema por parte de los grupos de investigación de todo el mundo, todavía no se ha realizado una investigación exhaustiva que estudie los parámetros que influyen en el comportamiento de estos sensores. En la primera parte de la tesis se presenta un estudio numérico en el que se examinan los efectos de la precisión de la medición de la tensión y los errores de posición del núcleo en el comportamiento de los sensores de forma basados en fibra óptica multinúcleo para definir la curvatura tridimensional, que es la base de la reconstrucción de la forma. El análisis reproduce el proceso de medición de la tensión utilizando el método de Monte Carlo (MC) e identifica una serie de parámetros que desempeñan un papel en el proceso, entre ellos la separación del núcleo (distancia entre los núcleos exteriores y el eje del sensor), el número de núcleos y la curvatura medida. Por último, se calibró un conjunto de modelos de predicción ajustando los resultados de las simulaciones para predecir el comportamiento de los sensores. A continuación, se propone un estudio experimental para evaluar el comportamiento de los sensores de forma basado en fibra óptica multinúcleo, con especial atención en la influencia de la longitud de los sensores de deformación. Se fabricaron dos sensores de forma, inscribiendo Fiber Bragg Gratings (FBG) con longitudes de 8,0 mm y 1,5 mm en los núcleos de una fibra multinúcleo de siete núcleos. Así, se evaluó y comparó el comportamiento de los dos sensores en todas las fases necesarias para la reconstrucción de la forma, incluyendo la medición de la tensión, el cálculo de la curvatura y la reconstrucción de la forma. Para concluir, se presenta un enfoque innovador, basado en la Teoría de la Torsión de Saint-Venant, para determinar la torsión de la fibra multinúcleo y compensar los errores debidos a la torsión durante la reconstrucción de la forma. La eficiencia del enfoque teórico fue verificada realizando una serie de pruebas de torsión en un sensor de forma, fabricado inscribiendo los sensores de FBGs en una fibra óptica multinúcleo torcida y siete núcleos. La investigación del comportamiento mecánico de los sensores ópticos de forma multinúcleo ha involucrado sinérgicamente diversas disciplinas: Mecánica del sólido, Fotónica, Estadística y Análisis de datos. Esta investigación multidisciplinaria ha surgido de la prolífica cooperación entre el Instituto de Ciencia y Tecnología del Hormigón (ICITECH) y el Instituto de Telecomunicaciones y Aplicaciones Multimedia (iTEAM) - Laboratorio de Investigación Fotónica (LIF) - de la Universidad Politécnica de Valencia (UPV), además de la valiosa colaboración con otros miembros del proyecto europeo ITN-FINESSE, al que pertenece este trabajo. Este trabajo de investigación puede permitir mejorar el comportamiento de los sensores de forma basados en fibra óptica multinúcleo y apoyar el desarrollo de nuevas geometrías de sensores, con un gran potencial para aplicaciones de control de la salud estructural.[CA] Structural Health Monitoring (SHM) avalua quantitativament la integritat i el comportament de les infraestructures i permet desenrotllar plans eficaços de Maintenance and Rehabilitation (M&R), utilitzant les dades dels sensors. Optical Multicore Fiber (MCF) Shape Sensors oferixen una alternativa als mètodes tradicionals i permeten la reconstrucció de la forma de la deformació de les estructures de forma directa i en temps real, sense necessitat de models de càlcul o contacte visual i amb tots els avantatges de l'Optical Fiber Sensors (OFS) Technology. A pesar dels grans esforços en la investigació centrada en aquest tema per part dels grups d'investigació de tot el món, encara no s'ha realitzat una investigació exhaustiva que estudie els paràmetres que influïxen en el comportament d'aquestos sensors. En la primera part de la tesi es presenta un estudi numèric en què s'examinen els efectes de la precisió del mesurament de la tensió i els errors de posició del nucli en el comportament dels sensors de forma basats en fibra òptica multinucli per a definir la curvatura tridimensional, que és la base de la reconstrucció de la forma. L'anàlisi reproduïx el procés de mesurament de la tensió utilitzant el mètode de Monte Carlo (MC) i identifica una sèrie de paràmetres que exercixen un paper en el procés, entre ells la separació del nucli (distància entre els nuclis exteriors i l'eix del sensor), el nombre de nuclis i la mesura de la curvatura. Finalment, es va calibrar un conjunt de models de predicció ajustant els resultats de les simulacions per a predir el comportament dels sensors. A continuació, es proposa un estudi experimental per a avaluar el comportament dels sensors de forma basat en fibra òptica multinucli, amb especial atenció en la influència de la longitud dels sensors de deformació. Es van fabricar dos sensors de forma, inscrivint Fiber Bragg Gratings (FBG) amb longituds de 8,0 mm i 1,5 mm en els nuclis d'una fibra multinucli de set nuclis. Així, es va avaluar i es va comparar el comportament dels dos sensors en totes les fases necessàries per a la reconstrucció de la forma, incloent el mesurament de la tensió, el càlcul de la curvatura i la reconstrucció de la forma. Per a concloure, es presenta un enfocament innovador, basat en la Teoria de la Torsió de Saint-Venant, per a determinar la torsió de la fibra multinucli i compensar els errors deguts a la torsió durant la reconstrucció de la forma. L'eficiència de l'enfocament teòric va ser verificada realitzant una sèrie de proves de torsió en un sensor de forma, fabricat inscrivint els sensors de FBGs en una fibra òptica de set nuclis de filat múltiple. La investigació del comportament mecànic dels sensors òptics de forma multinucli ha involucrat sinèrgicament diverses disciplines: Mecànica del sòlid, Fotónica, Estadística i Anàlisi de dades. Aquesta investigació multidisciplinària ha sorgit de la prolífica cooperació entre l'Institut de Ciència i Tecnologia del Formigó (ICITECH) i l'Institut de Telecomunicacions i Aplicacions Multimèdia (iTEAM) - Laboratori de investigación fotònica (LIF) - de la Universitat Politècnica de València (UPV), a més de la valuosa col·laboració amb altres membres del projecte europeu ITN- FINESSE, al qual pertany aquest treball. Aquest treball d'investigació pot permetre millorar el comportament dels sensors de forma basats en fibra òptica multinucli i ajudar al desenrotllament de noves geometries de sensors, amb un gran potencial per a aplicacions de control de la salut estructural.Floris, I. (2020). Optical Multicore Fiber Shape Sensors. A numerical and experimental performance assessment [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/148715TESI

Can the shape of a planar pathway be estimated using proximal forces of inserting a flexible shaft?

The shape information of flexible endoscopes or other continuum structures, e.g., intro-vascular catheters, is needed for accurate navigation, motion compensation, and haptic feedback in robotic surgical systems. Existing methods rely on optical fiber sensors, electromagnetic sensors, or expensive medical imaging modalities such as X-ray fluoroscopy, magnetic resonance imaging, and ultrasound to obtain the shape information of these flexible medical devices. Here, we propose to estimate the shape/curvature of a continuum structure by measuring the force required to insert a flexible shaft into the internal channel/pathway of the continuum. We found that there is a consistent correlation between the measured insertion force and curvature of the planar continuum pathway. A testbed was built to insert a flexible shaft into a planar continuum pathway with adjustable shapes. The insertion forces, insertion displacement, and the shapes of the pathway were recorded. A neural network model was developed to model this correlation based on the training data collected on the testbed. The trained model, tested on the testing data, can accurately estimate the curvature magnitudes and the accumulated bending angles of the pathway simply based on the measured insertion force at the proximal end of the shaft. The approach may be used to estimate the curvature magnitudes and accumulated bending angles of flexible endoscopic surgical robots or catheters for accurate motion compensation, haptic force feedback, localization, or navigation. The advantage of this approach is that the employed proximal force can be easily obtained outside the pathway or continuum structure without any embedded sensor in the continuum structure. Future work is needed to further investigate the correlation between insertion forces and the pathway and enhance the capability of the model in estimating more complex shapes, e.g., spatial shapes with multiple bends

DYNAMIC MEASUREMENT OF THREE-DIMENSIONAL MOTION FROM SINGLE-PERSPECTIVE TWO-DIMENSIONAL RADIOGRAPHIC PROJECTIONS

The digital evolution of the x-ray imaging modality has spurred the development of numerous clinical and research tools. This work focuses on the design, development, and validation of dynamic radiographic imaging and registration techniques to address two distinct medical applications: tracking during image-guided interventions, and the measurement of musculoskeletal joint kinematics. Fluoroscopy is widely employed to provide intra-procedural image-guidance. However, its planar images provide limited information about the location of surgical tools and targets in three-dimensional space. To address this limitation, registration techniques, which extract three-dimensional tracking and image-guidance information from planar images, were developed and validated in vitro. The ability to accurately measure joint kinematics in vivo is an important tool in studying both normal joint function and pathologies associated with injury and disease, however it still remains a clinical challenge. A technique to measure joint kinematics from single-perspective x-ray projections was developed and validated in vitro, using clinically available radiography equipmen