14 research outputs found
DeepPhase: Surgical Phase Recognition in CATARACTS Videos
Automated surgical workflow analysis and understanding can assist surgeons to
standardize procedures and enhance post-surgical assessment and indexing, as
well as, interventional monitoring. Computer-assisted interventional (CAI)
systems based on video can perform workflow estimation through surgical
instruments' recognition while linking them to an ontology of procedural
phases. In this work, we adopt a deep learning paradigm to detect surgical
instruments in cataract surgery videos which in turn feed a surgical phase
inference recurrent network that encodes temporal aspects of phase steps within
the phase classification. Our models present comparable to state-of-the-art
results for surgical tool detection and phase recognition with accuracies of 99
and 78% respectively.Comment: 8 pages, 3 figures, 1 table, MICCAI 201
Can surgical simulation be used to train detection and classification of neural networks?
Computer-assisted interventions (CAI) aim to increase the effectiveness, precision and repeatability of procedures to improve surgical outcomes. The presence and motion of surgical tools is a key information input for CAI surgical phase recognition algorithms. Vision-based tool detection and recognition approaches are an attractive solution and can be designed to take advantage of the powerful deep learning paradigm that is rapidly advancing image recognition and classification. The challenge for such algorithms is the availability and quality of labelled data used for training. In this Letter, surgical simulation is used to train tool detection and segmentation based on deep convolutional neural networks and generative adversarial networks. The authors experiment with two network architectures for image segmentation in tool classes commonly encountered during cataract surgery. A commercially-available simulator is used to create a simulated cataract dataset for training models prior to performing transfer learning on real surgical data. To the best of authors' knowledge, this is the first attempt to train deep learning models for surgical instrument detection on simulated data while demonstrating promising results to generalise on real data. Results indicate that simulated data does have some potential for training advanced classification methods for CAI systems
Plug-in for visualizing 3D tool tracking from videos of Minimally Invasive Surgeries
This paper tackles instrument tracking and 3D visualization challenges in
minimally invasive surgery (MIS), crucial for computer-assisted interventions.
Conventional and robot-assisted MIS encounter issues with limited 2D camera
projections and minimal hardware integration. The objective is to track and
visualize the entire surgical instrument, including shaft and metallic clasper,
enabling safe navigation within the surgical environment. The proposed method
involves 2D tracking based on segmentation maps, facilitating creation of
labeled dataset without extensive ground-truth knowledge. Geometric changes in
2D intervals express motion, and kinematics based algorithms process results
into 3D tracking information. Synthesized and experimental results in 2D and 3D
motion estimates demonstrate negligible errors, validating the method for
labeling and motion tracking of instruments in MIS videos. The conclusion
underscores the proposed 2D segmentation technique's simplicity and
computational efficiency, emphasizing its potential as direct plug-in for 3D
visualization in instrument tracking and MIS practices
Articulated Multi-Instrument 2D Pose Estimation Using Fully Convolutional Networks
Instrument detection, pose estimation and tracking in surgical videos is an important vision component for computer assisted interventions. While significant advances have been made in recent years, articulation detection is still a major challenge. In this paper, we propose a deep neural network for articulated multi-instrument 2D pose estimation, which is trained on a detailed annotations of endoscopic and microscopic datasets. Our model is formed by a fully convolutional detection-regression network. Joints and associations between joint pairs in our instrument model are located by the detection subnetwork and are subsequently refined through a regression subnetwork. Based on the output from the model, the poses of the instruments are inferred using maximum bipartite graph matching. Our estimation framework is powered by deep learning techniques without any direct kinematic information from a robot. Our framework is tested on single-instrument RMIT data, and also on multi-instrument EndoVis and in vivo data with promising results. In addition, the dataset annotations are publicly released along with our code and model
Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit
Robotic-assisted surgery is now well-established in clinical practice and has
become the gold standard clinical treatment option for several clinical
indications. The field of robotic-assisted surgery is expected to grow
substantially in the next decade with a range of new robotic devices emerging
to address unmet clinical needs across different specialities. A vibrant
surgical robotics research community is pivotal for conceptualizing such new
systems as well as for developing and training the engineers and scientists to
translate them into practice. The da Vinci Research Kit (dVRK), an academic and
industry collaborative effort to re-purpose decommissioned da Vinci surgical
systems (Intuitive Surgical Inc, CA, USA) as a research platform for surgical
robotics research, has been a key initiative for addressing a barrier to entry
for new research groups in surgical robotics. In this paper, we present an
extensive review of the publications that have been facilitated by the dVRK
over the past decade. We classify research efforts into different categories
and outline some of the major challenges and needs for the robotics community
to maintain this initiative and build upon it
Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery
PURPOSE: Computer-assisted interventions for enhanced minimally invasive surgery (MIS) require tracking of the surgical instruments. Instrument tracking is a challenging problem in both conventional and robotic-assisted MIS, but vision-based approaches are a promising solution with minimal hardware integration requirements. However, vision-based methods suffer from drift, and in the case of occlusions, shadows and fast motion, they can be subject to complete tracking failure. METHODS: In this paper, we develop a 2D tracker based on a Generalized Hough Transform using SIFT features which can both handle complex environmental changes and recover from tracking failure. We use this to initialize a 3D tracker at each frame which enables us to recover 3D instrument pose over long sequences and even during occlusions. RESULTS: We quantitatively validate our method in 2D and 3D with ex vivo data collected from a DVRK controller as well as providing qualitative validation on robotic-assisted in vivo data. CONCLUSIONS: We demonstrate from our extended sequences that our method provides drift-free robust and accurate tracking. Our occlusion-based sequences additionally demonstrate that our method can recover from occlusion-based failure. In both cases, we show an improvement over using 3D tracking alone suggesting that combining 2D and 3D tracking is a promising solution to challenges in surgical instrument tracking. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1007/s11548-016-1393-4) contains supplementary material, which is available to authorized users