Automatic alignment of surgical videos using kinematic data

Over the past one hundred years, the classic teaching methodology of "see one, do one, teach one" has governed the surgical education systems worldwide. With the advent of Operation Room 2.0, recording video, kinematic and many other types of data during the surgery became an easy task, thus allowing artificial intelligence systems to be deployed and used in surgical and medical practice. Recently, surgical videos has been shown to provide a structure for peer coaching enabling novice trainees to learn from experienced surgeons by replaying those videos. However, the high inter-operator variability in surgical gesture duration and execution renders learning from comparing novice to expert surgical videos a very difficult task. In this paper, we propose a novel technique to align multiple videos based on the alignment of their corresponding kinematic multivariate time series data. By leveraging the Dynamic Time Warping measure, our algorithm synchronizes a set of videos in order to show the same gesture being performed at different speed. We believe that the proposed approach is a valuable addition to the existing learning tools for surgery.Comment: Accepted at AIME 201

Distance‐based time series classification approach for task recognition with application in surgical robot autonomy

BackgroundRobotic‐assisted surgery allows surgeons to perform many types of complex operations with greater precision than is possible with conventional surgery. Despite these advantages, in current systems, a surgeon should communicate with the device directly and manually. To allow the robot to adjust parameters such as camera position, the system needs to know automatically what task the surgeon is performing.MethodsA distance‐based time series classification framework has been developed which measures dynamic time warping distance between temporal trajectory data of robot arms and classifies surgical tasks and gestures using a k‐nearest neighbor algorithm.ResultsResults on real robotic surgery data show that the proposed framework outperformed state‐of‐the‐art methods by up to 9% across three tasks and by 8% across gestures.ConclusionThe proposed framework is robust and accurate. Therefore, it can be used to develop adaptive control systems that will be more responsive to surgeons’ needs by identifying next movements of the surgeon. Copyright © 2016 John Wiley & Sons, Ltd.Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/138333/1/rcs1766.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/138333/2/rcs1766_am.pd

Advances in automated surgery skills evaluation

Training a surgeon to be skilled and competent to perform a given surgical procedure, is an important step in providing a high quality of care and reducing the risk of complications. Traditional surgical training is carried out by expert surgeons who observe and assess the trainees directly during a given procedure. However, these traditional training methods are time-consuming, subjective, costly, and do not offer an overall surgical expertise evaluation criterion. The solution for these subjective evaluation methods is a sensor-based methodology able to objectively assess the surgeon's skill level. The development and advances in sensor technologies enable capturing and studying the information obtained from complex surgery procedures. If the surgical activities that occur during a procedure are captured using a set of sensors, then the skill evaluation methodology can be defined as a motion and time series analysis problem. This work aims at developing machine learning approaches for automated surgical skill assessment based on hand motion analysis. Specifically, this work presents several contributions to the field of objective surgical techniques using multi-dimensional time series, such as 1) introduce a new distance measure for the surgical activities based on the alignment of two multi-dimensional time series, 2) develop an automated classification framework to identify the surgeon proficiency level using wrist worn sensors, 3) develop a classification technique to identify elementary surgical tasks: suturing, needle passing, and knot tying , 4) introduce a new surgemes mean feature reduction technique which help improve the machine learning algorithms, 5) develop a framework for surgical gesture classification by employing the mean feature reduction method, 6) design an unsupervised method to identify the surgemes in a given procedure.Includes bibliographical references

Computational Modeling Approaches For Task Analysis In Robotic-Assisted Surgery

Surgery is continuously subject to technological innovations including the introduction of robotic surgical devices. The ultimate goal is to program the surgical robot to perform certain difficult or complex surgical tasks in an autonomous manner. The feasibility of current robotic surgery systems to record quantitative motion and video data motivates developing descriptive mathematical models to recognize, classify and analyze surgical tasks. Recent advances in machine learning research for uncovering concealed patterns in huge data sets, like kinematic and video data, offer a possibility to better understand surgical procedures from a system point of view. This dissertation focuses on bridging the gap between these two lines of the research by developing computational models for task analysis in robotic-assisted surgery. The key step for advance study in robotic-assisted surgery and autonomous skill assessment is to develop techniques that are capable of recognizing fundamental surgical tasks intelligently. Surgical tasks and at a more granular level, surgical gestures, need to be quantified to make them amenable for further study. To answer to this query, we introduce a new framework, namely DTW-kNN, to recognize and classify three important surgical tasks including suturing, needle passing and knot tying based on kinematic data captured using da Vinci robotic surgery system. Our proposed method needs minimum preprocessing that results in simple, straightforward and accurate framework which can be applied for any autonomous control system. We also propose an unsupervised gesture segmentation and recognition (UGSR) method which has the ability to automatically segment and recognize temporal sequence of gestures in RMIS task. We also extent our model by applying soft boundary segmentation (Soft-UGSR) to address some of the challenges that exist in the surgical motion segmentation. The proposed algorithm can effectively model gradual transitions between surgical activities. Additionally, surgical training is undergoing a paradigm shift with more emphasis on the development of technical skills earlier in training. Thus metrics for the skills, especially objective metrics, become crucial. One field of surgery where such techniques can be developed is robotic surgery, as here all movements are already digitalized and therefore easily susceptible to analysis. Robotic surgery requires surgeons to perform a much longer and difficult training process which create numerous new challenges for surgical training. Hence, a new method of surgical skill assessment is required to ensure that surgeons have adequate skill level to be allowed to operate freely on patients. Among many possible approaches, those that provide noninvasive monitoring of expert surgeon and have the ability to automatically evaluate surgeon\u27s skill are of increased interest. Therefore, in this dissertation we develop a predictive framework for surgical skill assessment to automatically evaluate performance of surgeon in RMIS. Our classification framework is based on the Global Movement Features (GMFs) which extracted from kinematic movement data. The proposed method addresses some of the limitations in previous work and gives more insight about underlying patterns of surgical skill levels

Unsupervised Trajectory Segmentation for Surgical Gesture Recognition in Robotic Training

International audienceDexterity and procedural knowledge are two critical skills that surgeons need to master to perform accurate and safe surgical interventions. However, current training systems do not allow us to provide an in-depth analysis of surgical gestures to precisely assess these skills. Our objective is to develop a method for the automatic and quantitative assessment of surgical gestures. To reach this goal, we propose a new unsupervised algorithm that can automatically segment kinematic data from robotic training sessions. Without relying on any prior information or model, this algorithm detects critical points in the kinematic data that define relevant spatio-temporal segments. Based on the association of these segments, we obtain an accurate recognition of the gestures involved in the surgical training task. We, then, perform an advanced analysis and assess our algorithm using datasets recorded during real expert training sessions. After comparing our approach with the manual annotations of the surgical gestures, we observe 97.4% accuracy for the learning purpose and an average matching score of 81.9% for the fully automated gesture recognition process. Our results show that trainees workflow can be followed and surgical gestures may be automatically evaluated according to an expert database. This approach tends toward improving training efficiency by minimizing the learning curve

End-to-End Real-time Catheter Segmentation with Optical Flow-Guided Warping during Endovascular Intervention

Accurate real-time catheter segmentation is an important pre-requisite for robot-assisted endovascular intervention. Most of the existing learning-based methods for catheter segmentation and tracking are only trained on small-scale datasets or synthetic data due to the difficulties of ground-truth annotation. Furthermore, the temporal continuity in intraoperative imaging sequences is not fully utilised. In this paper, we present FW-Net, an end-to-end and real-time deep learning framework for endovascular intervention. The proposed FW-Net has three modules: a segmentation network with encoder-decoder architecture, a flow network to extract optical flow information, and a novel flow-guided warping function to learn the frame-to-frame temporal continuity. We show that by effectively learning temporal continuity, the network can successfully segment and track the catheters in real-time sequences using only raw ground-truth for training. Detailed validation results confirm that our FW-Net outperforms state-of-the-art techniques while achieving real-time performance.Comment: ICRA 202

Autonomous Camera Movement for Robotic-Assisted Surgery: A Survey

In the past decade, Robotic-Assisted Surgery (RAS) has become a widely accepted technique as an alternative to traditional open surgery procedures. The best robotic assistant system should combine both human and robot capabilities under the human control. As a matter of fact robot should collaborate with surgeons in a natural and autonomous way, thus requiring less of the surgeons\u27 attention. In this survey, we provide a comprehensive and structured review of the robotic-assisted surgery and autonomous camera movement for RAS operation. We also discuss several topics, including but not limited to task and gesture recognition, that are closely related to robotic-assisted surgery automation and illustrate several successful applications in various real-world application domains. We hope that this paper will provide a more thorough understanding of the recent advances in camera automation in RSA and offer some future research directions

Analysis of a Modern Voice Morphing Approach using Gaussian Mixture Models for Laryngectomees

This paper proposes a voice morphing system for people suffering from Laryngectomy, which is the surgical removal of all or part of the larynx or the voice box, particularly performed in cases of laryngeal cancer. A primitive method of achieving voice morphing is by extracting the source's vocal coefficients and then converting them into the target speaker's vocal parameters. In this paper, we deploy Gaussian Mixture Models (GMM) for mapping the coefficients from source to destination. However, the use of the traditional/conventional GMM-based mapping approach results in the problem of over-smoothening of the converted voice. Thus, we hereby propose a unique method to perform efficient voice morphing and conversion based on GMM,which overcomes the traditional-method effects of over-smoothening. It uses a technique of glottal waveform separation and prediction of excitations and hence the result shows that not only over-smoothening is eliminated but also the transformed vocal tract parameters match with the target. Moreover, the synthesized speech thus obtained is found to be of a sufficiently high quality. Thus, voice morphing based on a unique GMM approach has been proposed and also critically evaluated based on various subjective and objective evaluation parameters. Further, an application of voice morphing for Laryngectomees which deploys this unique approach has been recommended by this paper.Comment: 6 pages, 4 figures, 4 tables; International Journal of Computer Applications Volume 49, Number 21, July 201

Activity profiling for minimally invasive surgery

Imperial Users onl