Vision-based real-time position control of a semi-automated system for robot-assisted joint fracture surgery

Purpose: Joint fracture surgery quality can be improved by robotic system with high-accuracy and high-repeatability fracture fragment manipulation. A new real-time vision-based system for fragment manipulation during robot-assisted fracture surgery was developed and tested. Methods: The control strategy was accomplished by merging fast open-loop control with vision-based control. This two-phase process is designed to eliminate the open-loop positioning errors by closing the control loop using visual feedback provided by an optical tracking system. Evaluation of the control system accuracy was performed using robot positioning trials, and fracture reduction accuracy was tested in trials on ex vivo porcine model.Results: The system resulted in high fracture reduction reliability with a reduction accuracy of 0.09mm (translations) and of (Formula presented.) (rotations), maximum observed errors in the order of 0.12mm (translations) and of (Formula presented.) (rotations), and a reduction repeatability of 0.02mm and (Formula presented.). Conclusions: The proposed vision-based system was shown to be effective and suitable for real joint fracture surgical procedures, contributing a potential improvement of their quality

FPGA-based High-Performance Collision Detection: An Enabling Technique for Image-Guided Robotic Surgery

Collision detection, which refers to the computational problem of finding the relative placement or con-figuration of two or more objects, is an essential component of many applications in computer graphics and robotics. In image-guided robotic surgery, real-time collision detection is critical for preserving healthy anatomical structures during the surgical procedure. However, the computational complexity of the problem usually results in algorithms that operate at low speed. In this paper, we present a fast and accurate algorithm for collision detection between Oriented-Bounding-Boxes (OBBs) that is suitable for real-time implementation. Our proposed Sweep and Prune algorithm can perform a preliminary filtering to reduce the number of objects that need to be tested by the classical Separating Axis Test algorithm, while the OBB pairs of interest are preserved. These OBB pairs are re-checked by the Separating Axis Test algorithm to obtain accurate overlapping status between them. To accelerate the execution, our Sweep and Prune algorithm is tailor-made for the proposed method. Meanwhile, a high performance scalable hardware architecture is proposed by analyzing the intrinsic parallelism of our algorithm, and is implemented on FPGA platform. Results show that our hardware design on the FPGA platform can achieve around 8X higher running speed than the software design on a CPU platform. As a result, the proposed algorithm can achieve a collision frame rate of 1 KHz, and fulfill the requirement for the medical surgery scenario of Robot Assisted Laparoscopy.published_or_final_versio

Towards Highly-Integrated Stereovideoscopy for \u3ci\u3ein vivo\u3c/i\u3e Surgical Robots

When compared to traditional surgery, laparoscopic procedures result in better patient outcomes: shorter recovery, reduced post-operative pain, and less trauma to incisioned tissue. Unfortunately, laparoscopic procedures require specialized training for surgeons, as these minimally-invasive procedures provide an operating environment that has limited dexterity and limited vision. Advanced surgical robotics platforms can make minimally-invasive techniques safer and easier for the surgeon to complete successfully. The most common type of surgical robotics platforms -- the laparoscopic robots -- accomplish this with multi-degree-of-freedom manipulators that are capable of a diversified set of movements when compared to traditional laparoscopic instruments. Also, these laparoscopic robots allow for advanced kinematic translation techniques that allow the surgeon to focus on the surgical site, while the robot calculates the best possible joint positions to complete any surgical motion. An important component of these systems is the endoscopic system used to transmit a live view of the surgical environment to the surgeon. Coupled with 3D high-definition endoscopic cameras, the entirety of the platform, in effect, eliminates the peculiarities associated with laparoscopic procedures, which allows less-skilled surgeons to complete minimally-invasive surgical procedures quickly and accurately. A much newer approach to performing minimally-invasive surgery is the idea of using in-vivo surgical robots -- small robots that are inserted directly into the patient through a single, small incision; once inside, an in-vivo robot can perform surgery at arbitrary positions, with a much wider range of motion. While laparoscopic robots can harness traditional endoscopic video solutions, these in-vivo robots require a fundamentally different video solution that is as flexible as possible and free of bulky cables or fiber optics. This requires a miniaturized videoscopy system that incorporates an image sensor with a transceiver; because of severe size constraints, this system should be deeply embedded into the robotics platform. Here, early results are presented from the integration of a miniature stereoscopic camera into an in-vivo surgical robotics platform. A 26mm X 24mm stereo camera was designed and manufactured. The proposed device features USB connectivity and 1280 X 720 resolution at 30 fps. Resolution testing indicates the device performs much better than similarly-priced analog cameras. Suitability of the platform for 3D computer vision tasks -- including stereo reconstruction -- is examined. The platform was also tested in a living porcine model at the University of Nebraska Medical Center. Results from this experiment suggest that while the platform performs well in controlled, static environments, further work is required to obtain usable results in true surgeries. Concluding, several ideas for improvement are presented, along with a discussion of core challenges associated with the platform. Adviser: Lance C. Pérez [Document = 28 Mb

Using High-Level Processing of Low-Level Signals to Actively Assist Surgeons with Intelligent Surgical Robots

Robotic surgical systems are increasingly used for minimally-invasive surgeries. As such, there is opportunity for these systems to fundamentally change the way surgeries are performed by becoming intelligent assistants rather than simply acting as the extension of surgeons' arms. As a step towards intelligent assistance, this thesis looks at ways to represent different aspects of robot-assisted surgery (RAS). We identify three main components: the robot, the surgeon actions, and the patient scene dynamics. Traditional learning algorithms in these domains are predominantly supervised methods. This has several drawbacks. First many of these domains are non-categorical, like how soft-tissue deforms. This makes labeling difficult. Second, surgeries vary greatly. Estimation of the robot state may be affected by how the robot is docked and cable tensions in the instruments. Estimation of the patient anatomy and its dynamics are often inaccurate, and in any case, may change throughout a surgery. To obtain the most accurate information, these aspects must be learned during the procedure. This limits the amount of labeling that could be done. On the surgeon side, different surgeons may perform the same procedure differently and the algorithm should provide personalized estimations for surgeons. All of these considerations motivated the use of self-supervised learning throughout this thesis. We first build a representation of the robot system. In particular, we looked at learning the dynamics model of the robot. We evaluate the model by using it to estimate forces. Once we can estimate forces in free space, we extend the algorithm to take into account patient-specific interactions, namely with the trocar and the cannula seal. Accounting for surgery-specific interactions is possible because our method does not require additional sensors and can be trained in less than five minutes, including time for data collection. Next, we use cross-modal training to understand surgeon actions by looking at the bottleneck layer when mapping video to kinematics. This should contain information about the latent space of surgeon-actions, while discarding some medium-specific information about either the video or the kinematics. Lastly, to understand the patient scene, we start with modeling interactions between a robot instrument and a soft-tissue phantom. Models are often inaccurate due to imprecise material parameters and boundary conditions, particularly in clinical scenarios. Therefore, we add a depth camera to observe deformations to correct the results of simulations. We also introduce a network that learns to simulate soft-tissue deformation from physics simulators in order to speed up the estimation. We demonstrate that self-supervised learning can be used for understanding each part of RAS. The representations it learns contain information about signals that are not directly measurable. The self-supervised nature of the methods presented in this thesis lends itself well to learning throughout a surgery. With such frameworks, we can overcome some of the main barriers to adopting learning methods in the operating room: the variety in surgery and the difficulty in labeling enough training data for each case

형태적응형 이력현상 모형을 이용한 유연구동 메커니즘의 모델링

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 기계항공공학부, 2020. 8. 김종원.Flexible surgical robots and instruments are slowly paving its way into the modern surgical arena. Compared to conventional laparoscopic surgical systems, flexible systems have some distinct advantages in that it can approach surgical targets that were unreachable before, leaves less scar and therefore reducing recovery time for patients. In order to drive the articulated surgical instruments joints, flexible instruments require a tendon-sheath mechanism (TSM). Utilization of TSM brings about a different attribute in a position control standpoint, compared to the rather simple cable-pulley system found in conventional robotic surgical instruments. In this research, a tendon-sheath mechanism was configured, taking into account the actual size constraint of a robotic surgical instrument and the material characteristics of the components. An experiment hardware was designed to measure the input signal and the corresponding output response while varying the shape configuration parameters of TSM. Twenty four distinct experiments with different shape configuration parameters were carried out to identify how the shape affects the performance and the hysteresis curve of the TSM. For modeling the hysteretic behavior of the TSM, a composite model consisting of elementary hysteresis operators is proposed. Such a composite models parameters are empirically identified with least-squares optimization, for every shape configurations defined. The model processes the input to produce an estimated output for a certain shape, and this was verified with various types of input signals. Lastly, for compensating TSMs hysteretic behavior, a recursive algorithm producing inverse control signal from the empirical model is proposed, with a guaranteed real-time performance. The inverse algorithms position control effectiveness was verified under various shape configurations and input signal types.본 연구에서는 유연한 로봇 수술도구를 구현하기 위해 사용되는 Tendon-Sheath Mechanism (TSM)이 형상에 따른 이력현상의 변화가 발생하는 것을 실험적으로 확인하였으며, 이러한 이력현상을 표현하기 위한 모형을 제안하고 이를 이용하여 이력현상을 보상할 수 있는 알고리즘을 제안하였다. 첫 단계로 TSM을 구성하는 부품인 Tendon과 Sheath를 선정하는데 있어, 이력현상에 일조 하는 비선형적 특성을 최소화하는 재료와 공정 및 후처리 방법을 고려하여 적용하였다. 다음으로 TSM의 형상 변수를 정의하고 이를 다양한 형상하에서 이력현상의 변화를 관찰하는 실험장치를 설계하여 실험 데이터를 수집하였다. 이를 토대로 입력에 대한 출력의 관계를 Preisach type 연산자를 이용하여 표현하였고 실험 데이터에 기반한 연산자의 변수들을 최소자승 최적화를 통해 탐색하였으며, 모델의 적합성을 다양한 형상하에서, 각기 다른 종류의 입력 신호에 대한 출력을 모델을 통해 생성되는 출력 추정치와의 오차 분석으로 검증하였다. 이러한 모델로 이력현상을 보상하기 위해서 Set-Point 출력에 대한 Inverse Control 신호를 생성하는 재귀적 알고리즘을 제안하였으며, 이러한 알고리즘이 다양한 Set-point 출력의 형태에 대해서 실시간성이 보장되는 빠른 연산이 가능하다는 점을 보였다. 이력현상이 보상된 실험데이터와 기존의 보상전 실험데이터의 비교를 통해 보상전략이 효과적이라는 것을 보였으며, 다양한 형태에서도 적용이 가능함을 검증하였다.Table of Contents Chapter 1. Introduction 1 1.1 Background 1 1.1.1 Evolution of surgical robots 1 1.1.2 Flexible robotic systems 3 1.2 Tendon-sheath mechanism 6 1.2.1 Application of TSM in flexible surgical instruments 6 1.2.2 Effects on motion transfer characteristics 8 1.3 Previous studies 10 1.4 Research objectives 12 Chapter 2. Configuration and fabrication of TSM 14 2.1 Sheath 17 2.2 Tendon 19 2.2.1 Cable 19 2.2.2 Fitting 23 Chapter 3. Hysteretic behavior of TSM 25 3.1 Experiment setup 26 3.1.1 Experiment design 26 3.1.2 Hardware design 28 3.2 Experiment results 34 3.2.1 Effect of curve angle variation 34 3.2.2 Effect of radius of curvature variation 39 3.2.3 Summary of results of hysteretic behavior 46 Chapter 4. Modeling Hysteresis of TSM 49 4.1 Preisach model and Hysterons 50 4.2 Mechanical play operator 53 4.3 Complex hysteresis operator: 56 4.4 Parameter identification for complex hysteresis operator 59 4.5 Result of experimental verification of complex hysteresis operator 60 4.5.1 Result of reference input profile sinusoidal excitation 63 4.5.2 Result of validation input profile triangular excitation 65 4.5.3 Result of reference input profile trapezoidal excitation 67 4.5.4 Obtained weights for all shape configurations and summary 69 4.6 Inverse operator formulation 60 4.7 Experimental verification of hysteresis compensation with the inverse operator: 77 4.7.1 Experiment setup 77 4.7.2 Result of hysteresis compensation for shape =90,r=30mm 79 4.7.3 Result of hysteresis compensation for shape =60,r=60mm 82 4.7.4 Error statistic and result analysis 85 Chapter 6. Conclusion 87 Bibliography 88 Abstract in Korean 92Docto

GPU-based proximity query processing on unstructured triangular mesh model

This paper presents a novel proximity query (PQ) approach capable to detect the collision and calculate the minimal Euclidean distance between two non-convex objects in 3D, namely the robot and the environment. Such approaches are often considered as computationally demanding problems, but are of importance to many applications such as online simulation of haptic feedback and robot collision-free trajectory. Our approach enables to preserve the representation of unstructured environment in the form of triangular meshes. The proposed PQ algorithm is computationally parallel so that it can be effectively implemented on graphics processing units (GPUs). A GPU-based computation scheme is also developed and customized, which shows >200 times faster than an optimized CPU with single core. Comprehensive validation is also conducted on two simulated scenarios in order to demonstrate the practical values of its potential application in image-guided surgical robotics and humanoid robotic control.published_or_final_versio

Platforms for prototyping minimally invasive instruments

The introduction of new technologies in medicine is often an issue because there are many stages to go through, from the idea to the approval by ethical committees and mass production. This work covers the first steps of the development of a medical device, dealing with the tools that can help to reduce the time for producing the laboratory prototype. These tools can involve electronics and software for the creation of a “universal”' hardware platform that can be used for many robotic applications, adapting only few components for the specific scenario. The platform is created by setting up a traditional computer with operating system and acquisition channels aimed at opening the system toward the real environment. On this platform algorithms can be implemented rapidly, allowing to assess the feasibility of an idea. This approach lets the designer concentrate on the application rather than on the selection of the appropriate hardware electronics every time that a new project starts. In the first part an overview of the existing instruments for minimally invasive interventions that can be found as commercial or research products is given. An introduction related to hardware electronics is presented with the requirements and the specific characteristics needed for a robotic application. The second part focuses on specific projects in MIS. The first project concerns the study and the development of a lightweight hand-held robotic instrument for laparoscopy. Motivations are related to the lack of dexterous hand-held laparoscopic instruments. The second project concerns the study and the presentation of a prototype of a robotic endoscope with enhanced resolution. The third project concerns the development of a system able to detect the inspiration and the expiration phases. The aim is to evaluate the weariness of the surgeon, since breathing can be related to fatigue

Implicit active constraints for concentric tube robots based on analysis of the safe and dexterous workspace

The use of concentric tube robots has recognized advantages for accessing target lesions while conforming to certain anatomical constraints. However, their complex kinematics makes their safe telemanipulation in convoluted anatomy a challenging task. Collaborative control schemes, which guide the operator through haptic and visual feedback, can simplify this task and reduce the cognitive burden of the operator. Guaranteeing stable, collision-free robot configurations during manipulation, however, is computationally demanding and, until now, either required long periods of pre-computation time or distributed computing clusters. Furthermore, the operator is often presented with guidance paths which have to be followed approximately. This paper presents a heterogeneous (CPU/GPU) computing approach to enable rapid workspace analysis on a single computer. The method is used in a new navigation scheme that guides the robot operator towards locations of high dexterity or manipulability of the robot. Under this guidance scheme, the user can make informed decisions and maintain full control of the path planning and manipulation processes, with intuitive visual feedback on when the robot's limitations are being reached