382 research outputs found
Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery
Intraoperative segmentation and tracking of minimally invasive instruments is
a prerequisite for computer- and robotic-assisted surgery. Since additional
hardware like tracking systems or the robot encoders are cumbersome and lack
accuracy, surgical vision is evolving as promising techniques to segment and
track the instruments using only the endoscopic images. However, what is
missing so far are common image data sets for consistent evaluation and
benchmarking of algorithms against each other. The paper presents a comparative
validation study of different vision-based methods for instrument segmentation
and tracking in the context of robotic as well as conventional laparoscopic
surgery. The contribution of the paper is twofold: we introduce a comprehensive
validation data set that was provided to the study participants and present the
results of the comparative validation study. Based on the results of the
validation study, we arrive at the conclusion that modern deep learning
approaches outperform other methods in instrument segmentation tasks, but the
results are still not perfect. Furthermore, we show that merging results from
different methods actually significantly increases accuracy in comparison to
the best stand-alone method. On the other hand, the results of the instrument
tracking task show that this is still an open challenge, especially during
challenging scenarios in conventional laparoscopic surgery
A comprehensive survey on recent deep learning-based methods applied to surgical data
Minimally invasive surgery is highly operator dependant with a lengthy
procedural time causing fatigue to surgeon and risks to patients such as injury
to organs, infection, bleeding, and complications of anesthesia. To mitigate
such risks, real-time systems are desired to be developed that can provide
intra-operative guidance to surgeons. For example, an automated system for tool
localization, tool (or tissue) tracking, and depth estimation can enable a
clear understanding of surgical scenes preventing miscalculations during
surgical procedures. In this work, we present a systematic review of recent
machine learning-based approaches including surgical tool localization,
segmentation, tracking, and 3D scene perception. Furthermore, we provide a
detailed overview of publicly available benchmark datasets widely used for
surgical navigation tasks. While recent deep learning architectures have shown
promising results, there are still several open research problems such as a
lack of annotated datasets, the presence of artifacts in surgical scenes, and
non-textured surfaces that hinder 3D reconstruction of the anatomical
structures. Based on our comprehensive review, we present a discussion on
current gaps and needed steps to improve the adaptation of technology in
surgery.Comment: This paper is to be submitted to International journal of computer
visio
Artificial intelligence and automation in endoscopy and surgery
Modern endoscopy relies on digital technology, from high-resolution imaging sensors and displays to electronics connecting configurable illumination and actuation systems for robotic articulation. In addition to enabling more effective diagnostic and therapeutic interventions, the digitization of the procedural toolset enables video data capture of the internal human anatomy at unprecedented levels. Interventional video data encapsulate functional and structural information about a patientβs anatomy as well as events, activity and action logs about the surgical process. This detailed but difficult-to-interpret record from endoscopic procedures can be linked to preoperative and postoperative records or patient imaging information. Rapid advances in artificial intelligence, especially in supervised deep learning, can utilize data from endoscopic procedures to develop systems for assisting procedures leading to computer-assisted interventions that can enable better navigation during procedures, automation of image interpretation and robotically assisted tool manipulation. In this Perspective, we summarize state-of-the-art artificial intelligence for computer-assisted interventions in gastroenterology and surgery
Artificial intelligence surgery: how do we get to autonomous actions in surgery?
Most surgeons are skeptical as to the feasibility of autonomous actions in surgery. Interestingly, many examples of autonomous actions already exist and have been around for years. Since the beginning of this millennium, the field of artificial intelligence (AI) has grown exponentially with the development of machine learning (ML), deep learning (DL), computer vision (CV) and natural language processing (NLP). All of these facets of AI will be fundamental to the development of more autonomous actions in surgery, unfortunately, only a limited number of surgeons have or seek expertise in this rapidly evolving field. As opposed to AI in medicine, AI surgery (AIS) involves autonomous movements. Fortuitously, as the field of robotics in surgery has improved, more surgeons are becoming interested in technology and the potential of autonomous actions in procedures such as interventional radiology, endoscopy and surgery. The lack of haptics, or the sensation of touch, has hindered the wider adoption of robotics by many surgeons; however, now that the true potential of robotics can be comprehended, the embracing of AI by the surgical community is more important than ever before. Although current complete surgical systems are mainly only examples of tele-manipulation, for surgeons to get to more autonomously functioning robots, haptics is perhaps not the most important aspect. If the goal is for robots to ultimately become more and more independent, perhaps research should not focus on the concept of haptics as it is perceived by humans, and the focus should be on haptics as it is perceived by robots/computers. This article will discuss aspects of ML, DL, CV and NLP as they pertain to the modern practice of surgery, with a focus on current AI issues and advances that will enable us to get to more autonomous actions in surgery. Ultimately, there may be a paradigm shift that needs to occur in the surgical community as more surgeons with expertise in AI may be needed to fully unlock the potential of AIS in a safe, efficacious and timely manner
ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools
Real-time tool segmentation from endoscopic videos is an essential part of
many computer-assisted robotic surgical systems and of critical importance in
robotic surgical data science. We propose two novel deep learning architectures
for automatic segmentation of non-rigid surgical instruments. Both methods take
advantage of automated deep-learning-based multi-scale feature extraction while
trying to maintain an accurate segmentation quality at all resolutions. The two
proposed methods encode the multi-scale constraint inside the network
architecture. The first proposed architecture enforces it by cascaded
aggregation of predictions and the second proposed network does it by means of
a holistically-nested architecture where the loss at each scale is taken into
account for the optimization process. As the proposed methods are for real-time
semantic labeling, both present a reduced number of parameters. We propose the
use of parametric rectified linear units for semantic labeling in these small
architectures to increase the regularization ability of the design and maintain
the segmentation accuracy without overfitting the training sets. We compare the
proposed architectures against state-of-the-art fully convolutional networks.
We validate our methods using existing benchmark datasets, including ex vivo
cases with phantom tissue and different robotic surgical instruments present in
the scene. Our results show a statistically significant improved Dice
Similarity Coefficient over previous instrument segmentation methods. We
analyze our design choices and discuss the key drivers for improving accuracy.Comment: Paper accepted at IROS 201
μμμ κΈ° ν₯μμ μν λ₯λ¬λ κΈ°λ² μ°κ΅¬: λμ₯λ΄μκ²½ μ§λ¨ λ° λ‘λ΄μμ μ κΈ° νκ°μ μ μ©
νμλ
Όλ¬Έ (λ°μ¬) -- μμΈλνκ΅ λνμ : 곡과λν νλκ³Όμ μμ©μ체곡νμ 곡, 2020. 8. κΉν¬μ°¬.This paper presents deep learning-based methods for improving performance of clinicians. Novel methods were applied to the following two clinical cases and the results were evaluated.
In the first study, a deep learning-based polyp classification algorithm for improving clinical performance of endoscopist during colonoscopy diagnosis was developed. Colonoscopy is the main method for diagnosing adenomatous polyp, which can multiply into a colorectal cancer and hyperplastic polyps. The classification algorithm was developed using convolutional neural network (CNN), trained with colorectal polyp images taken by a narrow-band imaging colonoscopy. The proposed method is built around an automatic machine learning (AutoML) which searches for the optimal architecture of CNN for colorectal polyp image classification and trains the weights of the architecture. In addition, gradient-weighted class activation mapping technique was used to overlay the probabilistic basis of the prediction result on the polyp location to aid the endoscopists visually. To verify the improvement in diagnostic performance, the efficacy of endoscopists with varying proficiency levels were compared with or without the aid of the proposed polyp classification algorithm. The results confirmed that, on average, diagnostic accuracy was improved and diagnosis time was shortened in all proficiency groups significantly.
In the second study, a surgical instruments tracking algorithm for robotic surgery video was developed, and a model for quantitatively evaluating the surgeons surgical skill based on the acquired motion information of the surgical instruments was proposed. The movement of surgical instruments is the main component of evaluation for surgical skill. Therefore, the focus of this study was develop an automatic surgical instruments tracking algorithm, and to overcome the limitations presented by previous methods. The instance segmentation framework was developed to solve the instrument occlusion issue, and a tracking framework composed of a tracker and a re-identification algorithm was developed to maintain the type of surgical instruments being tracked in the video. In addition, algorithms for detecting the tip position of instruments and arm-indicator were developed to acquire the movement of devices specialized for the robotic surgery video. The performance of the proposed method was evaluated by measuring the difference between the predicted tip position and the ground truth position of the instruments using root mean square error, area under the curve, and Pearsons correlation analysis. Furthermore, motion metrics were calculated from the movement of surgical instruments, and a machine learning-based robotic surgical skill evaluation model was developed based on these metrics. These models were used to evaluate clinicians, and results were similar in the developed evaluation models, the Objective Structured Assessment of Technical Skill (OSATS), and the Global Evaluative Assessment of Robotic Surgery (GEARS) evaluation methods.
In this study, deep learning technology was applied to colorectal polyp images for a polyp classification, and to robotic surgery videos for surgical instruments tracking. The improvement in clinical performance with the aid of these methods were evaluated and verified.λ³Έ λ
Όλ¬Έμ μλ£μ§μ μμμ κΈ° λ₯λ ₯μ ν₯μμν€κΈ° μνμ¬ μλ‘μ΄ λ₯λ¬λ κΈ°λ²λ€μ μ μνκ³ λ€μ λ κ°μ§ μ€λ‘μ λν΄ μ μ©νμ¬ κ·Έ κ²°κ³Όλ₯Ό νκ°νμλ€.
첫 λ²μ§Έ μ°κ΅¬μμλ λμ₯λ΄μκ²½μΌλ‘ κ΄ν μ§λ¨ μ, λ΄μκ²½ μ λ¬Έμμ μ§λ¨ λ₯λ ₯μ ν₯μμν€κΈ° μνμ¬ λ₯λ¬λ κΈ°λ°μ μ©μ’
λΆλ₯ μκ³ λ¦¬μ¦μ κ°λ°νκ³ , λ΄μκ²½ μ λ¬Έμμ μ§λ¨ λ₯λ ₯ ν₯μ μ¬λΆλ₯Ό κ²μ¦νκ³ μ νμλ€. λμ₯λ΄μκ²½ κ²μ¬λ‘ μμ’
μΌλ‘ μ¦μν μ μλ μ μ’
κ³Ό κ³Όμ¦μμ± μ©μ’
μ μ§λ¨νλ κ²μ μ€μνλ€. λ³Έ μ°κ΅¬μμλ νλμ μμ λ΄μκ²½μΌλ‘ 촬μν λμ₯ μ©μ’
μμμΌλ‘ ν©μ±κ³± μ κ²½λ§μ νμ΅νμ¬ λΆλ₯ μκ³ λ¦¬μ¦μ κ°λ°νμλ€. μ μνλ μκ³ λ¦¬μ¦μ μλ κΈ°κ³νμ΅ (AutoML) λ°©λ²μΌλ‘, λμ₯ μ©μ’
μμμ μ΅μ νλ ν©μ±κ³± μ κ²½λ§ κ΅¬μ‘°λ₯Ό μ°Ύκ³ μ κ²½λ§μ κ°μ€μΉλ₯Ό νμ΅νμλ€. λν κΈ°μΈκΈ°-κ°μ€μΉ ν΄λμ€ νμ±ν 맡ν κΈ°λ²μ μ΄μ©νμ¬ κ°λ°ν ν©μ±κ³± μ κ²½λ§ κ²°κ³Όμ νλ₯ μ κ·Όκ±°λ₯Ό μ©μ’
μμΉμ μκ°μ μΌλ‘ λνλλλ‘ ν¨μΌλ‘ λ΄μκ²½ μ λ¬Έμμ μ§λ¨μ λλλ‘ νμλ€. λ§μ§λ§μΌλ‘, μλ ¨λ κ·Έλ£Ήλ³λ‘ λ΄μκ²½ μ λ¬Έμκ° μ©μ’
λΆλ₯ μκ³ λ¦¬μ¦μ κ²°κ³Όλ₯Ό μ°Έκ³ νμμ λ μ§λ¨ λ₯λ ₯μ΄ ν₯μλμλμ§ λΉκ΅ μ€νμ μ§ννμκ³ , λͺ¨λ κ·Έλ£Ήμμ μ μλ―Ένκ² μ§λ¨ μ νλκ° ν₯μλκ³ μ§λ¨ μκ°μ΄ λ¨μΆλμμμ νμΈνμλ€.
λ λ²μ§Έ μ°κ΅¬μμλ λ‘λ΄μμ λμμμμ μμ λꡬ μμΉ μΆμ μκ³ λ¦¬μ¦μ κ°λ°νκ³ , νλν μμ λꡬμ μμ§μ μ 보λ₯Ό λ°νμΌλ‘ μμ μμ μλ ¨λλ₯Ό μ λμ μΌλ‘ νκ°νλ λͺ¨λΈμ μ μνμλ€. μμ λꡬμ μμ§μμ μμ μμ λ‘λ΄μμ μλ ¨λλ₯Ό νκ°νκΈ° μν μ£Όμν μ 보μ΄λ€. λ°λΌμ λ³Έ μ°κ΅¬λ λ₯λ¬λ κΈ°λ°μ μλ μμ λꡬ μΆμ μκ³ λ¦¬μ¦μ κ°λ°νμμΌλ©°, λ€μ λκ°μ§ μ νμ°κ΅¬μ νκ³μ μ 극볡νμλ€. μΈμ€ν΄μ€ λΆν (Instance Segmentation) νλ μμμ κ°λ°νμ¬ νμ (Occlusion) λ¬Έμ λ₯Ό ν΄κ²°νμκ³ , μΆμ κΈ° (Tracker)μ μ¬μλ³ν (Re-Identification) μκ³ λ¦¬μ¦μΌλ‘ ꡬμ±λ μΆμ νλ μμμ κ°λ°νμ¬ λμμμμ μΆμ νλ μμ λꡬμ μ’
λ₯κ° μ μ§λλλ‘ νμλ€. λν λ‘λ΄μμ λμμμ νΉμμ±μ κ³ λ €νμ¬ μμ λꡬμ μμ§μμ νλνκΈ°μν΄ μμ λꡬ λ μμΉμ λ‘λ΄ ν-μΈλμΌμ΄ν° (Arm-Indicator) μΈμ μκ³ λ¦¬μ¦μ κ°λ°νμλ€. μ μνλ μκ³ λ¦¬μ¦μ μ±λ₯μ μμΈ‘ν μμ λꡬ λ μμΉμ μ λ΅ μμΉ κ°μ νκ· μ κ³±κ·Ό μ€μ°¨, 곑μ μλ λ©΄μ , νΌμ΄μ¨ μκ΄λΆμμΌλ‘ νκ°νμλ€. λ§μ§λ§μΌλ‘, μμ λꡬμ μμ§μμΌλ‘λΆν° μμ§μ μ§νλ₯Ό κ³μ°νκ³ μ΄λ₯Ό λ°νμΌλ‘ κΈ°κ³νμ΅ κΈ°λ°μ λ‘λ΄μμ μλ ¨λ νκ° λͺ¨λΈμ κ°λ°νμλ€. κ°λ°ν νκ° λͺ¨λΈμ κΈ°μ‘΄μ Objective Structured Assessment of Technical Skill (OSATS), Global Evaluative Assessment of Robotic Surgery (GEARS) νκ° λ°©λ²κ³Ό μ μ¬ν μ±λ₯μ 보μμ νμΈνμλ€.
λ³Έ λ
Όλ¬Έμ μλ£μ§μ μμμ κΈ° λ₯λ ₯μ ν₯μμν€κΈ° μνμ¬ λμ₯ μ©μ’
μμκ³Ό λ‘λ΄μμ λμμμ λ₯λ¬λ κΈ°μ μ μ μ©νκ³ κ·Έ μ ν¨μ±μ νμΈνμμΌλ©°, ν₯νμ μ μνλ λ°©λ²μ΄ μμμμ μ¬μ©λκ³ μλ μ§λ¨ λ° νκ° λ°©λ²μ λμμ΄ λ κ²μΌλ‘ κΈ°λνλ€.Chapter 1 General Introduction 1
1.1 Deep Learning for Medical Image Analysis 1
1.2 Deep Learning for Colonoscipic Diagnosis 2
1.3 Deep Learning for Robotic Surgical Skill Assessment 3
1.4 Thesis Objectives 5
Chapter 2 Optical Diagnosis of Colorectal Polyps using Deep Learning with Visual Explanations 7
2.1 Introduction 7
2.1.1 Background 7
2.1.2 Needs 8
2.1.3 Related Work 9
2.2 Methods 11
2.2.1 Study Design 11
2.2.2 Dataset 14
2.2.3 Preprocessing 17
2.2.4 Convolutional Neural Networks (CNN) 21
2.2.4.1 Standard CNN 21
2.2.4.2 Search for CNN Architecture 22
2.2.4.3 Searched CNN Training 23
2.2.4.4 Visual Explanation 24
2.2.5 Evaluation of CNN and Endoscopist Performances 25
2.3 Experiments and Results 27
2.3.1 CNN Performance 27
2.3.2 Results of Visual Explanation 31
2.3.3 Endoscopist with CNN Performance 33
2.4 Discussion 45
2.4.1 Research Significance 45
2.4.2 Limitations 47
2.5 Conclusion 49
Chapter 3 Surgical Skill Assessment during Robotic Surgery by Deep Learning-based Surgical Instrument Tracking 50
3.1 Introduction 50
3.1.1 Background 50
3.1.2 Needs 51
3.1.3 Related Work 52
3.2 Methods 56
3.2.1 Study Design 56
3.2.2 Dataset 59
3.2.3 Instance Segmentation Framework 63
3.2.4 Tracking Framework 66
3.2.4.1 Tracker 66
3.2.4.2 Re-identification 68
3.2.5 Surgical Instrument Tip Detection 69
3.2.6 Arm-Indicator Recognition 71
3.2.7 Surgical Skill Prediction Model 71
3.3 Experiments and Results 78
3.3.1 Performance of Instance Segmentation Framework 78
3.3.2 Performance of Tracking Framework 82
3.3.3 Evaluation of Surgical Instruments Trajectory 83
3.3.4 Evaluation of Surgical Skill Prediction Model 86
3.4 Discussion 90
3.4.1 Research Significance 90
3.4.2 Limitations 92
3.5 Conclusion 96
Chapter 4 Summary and Future Works 97
4.1 Thesis Summary 97
4.2 Limitations and Future Works 98
Bibliography 100
Abstract in Korean 116
Acknowledgement 119Docto
- β¦