5,229 research outputs found
Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery
One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-opera- tive morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilites by observ- ing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted in- struments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This paper reviews the state-of-the-art methods for optical intra-operative 3D reconstruction in laparoscopic surgery and discusses the technical challenges and future perspectives towards clinical translation. With the recent paradigm shift of surgical practice towards MIS and new developments in 3D opti- cal imaging, this is a timely discussion about technologies that could facilitate complex CAS procedures in dynamic and deformable anatomical regions
Non-Rigid Liver Registration for Laparoscopy using Data-Driven Biomechanical Models
During laparoscopic liver resection, the limited access to the organ, the small field of view and lack of palpation can obstruct a surgeon’s workflow. Automatic navigation systems could use the images from preoperative volumetric organ scans to help the surgeons find their target (tumors) and risk-structures (vessels) more efficiently. This requires the preoperative data to be fused (or registered) with the intraoperative scene in order to display information at the correct intraoperative position.
One key challenge in this setting is the automatic estimation of the organ’s current intra-operative deformation, which is required in order to predict the position of internal structures. Parameterizing the many patient-specific unknowns (tissue properties, boundary conditions, interactions with other tissues, direction of gravity) is very difficult. Instead, this work explores how to employ deep neural networks to solve the registration problem in a data-driven manner. To this end, convolutional neural networks are trained on synthetic data to estimate an organ’s intraoperative displacement field and thus its current deformation. To drive this estimation, visible surface cues from the intraoperative camera view must be supplied to the networks. Since reliable surface features are very difficult to find, the networks are adapted to also find correspondences between the pre- and intraoperative liver geometry automatically. This combines the search for correspondences with the biomechanical behavior estimation and allows the networks to tackle the full non-rigid registration problem in one single step. The result is a model which can quickly predict the volume deformation of a liver, given only sparse surface information. The model combines the advantages of a physically accurate biomechanical simulation with the speed and powerful feature extraction capabilities of deep neural networks.
To test the method intraoperatively, a registration pipeline is developed which constructs a map of the liver and its surroundings from the laparoscopic video and then uses the neural networks to fuse the preoperative volume data into this map. The deformed organ volume can then be rendered as an overlay directly onto the laparoscopic video stream. The focus of this pipeline is to be applicable to real surgery, where everything should be quick and non-intrusive. To meet these requirements, a SLAM system is used to localize the laparoscopic camera (avoiding setup of an external tracking system), various neural networks are used to quickly interpret the scene and semi-automatic tools let the surgeons guide the system.
Beyond the concrete advantages of the data-driven approach for intraoperative registration, this work also demonstrates general benefits of training a registration system preoperatively on synthetic data. The method lets the engineer decide which values need to be known explicitly and which should be estimated implicitly by the networks, which opens the door to many new possibilities.:1 Introduction
1.1 Motivation
1.1.1 Navigated Liver Surgery
1.1.2 Laparoscopic Liver Registration
1.2 Challenges in Laparoscopic Liver Registration
1.2.1 Preoperative Model
1.2.2 Intraoperative Data
1.2.3 Fusion/Registration
1.2.4 Data
1.3 Scope and Goals of this Work
1.3.1 Data-Driven, Biomechanical Model
1.3.2 Data-Driven Non-Rigid Registration
1.3.3 Building a Working Prototype
2 State of the Art
2.1 Rigid Registration
2.2 Non-Rigid Liver Registration
2.3 Neural Networks for Simulation and Registration
3 Theoretical Background
3.1 Liver
3.2 Laparoscopic Liver Resection
3.2.1 Staging Procedure
3.3 Biomechanical Simulation
3.3.1 Physical Balance Principles
3.3.2 Material Models
3.3.3 Numerical Solver: The Finite Element Method (FEM)
3.3.4 The Lagrangian Specification
3.4 Variables and Data in Liver Registration
3.4.1 Observable
3.4.2 Unknowns
4 Generating Simulations of Deforming Organs
4.1 Organ Volume
4.2 Forces and Boundary Conditions
4.2.1 Surface Forces
4.2.2 Zero-Displacement Boundary Conditions
4.2.3 Surrounding Tissues and Ligaments
4.2.4 Gravity
4.2.5 Pressure
4.3 Simulation
4.3.1 Static Simulation
4.3.2 Dynamic Simulation
4.4 Surface Extraction
4.4.1 Partial Surface Extraction
4.4.2 Surface Noise
4.4.3 Partial Surface Displacement
4.5 Voxelization
4.5.1 Voxelizing the Liver Geometry
4.5.2 Voxelizing the Displacement Field
4.5.3 Voxelizing Boundary Conditions
4.6 Pruning Dataset - Removing Unwanted Results
4.7 Data Augmentation
5 Deep Neural Networks for Biomechanical Simulation
5.1 Training Data
5.2 Network Architecture
5.3 Loss Functions and Training
6 Deep Neural Networks for Non-Rigid Registration
6.1 Training Data
6.2 Architecture
6.3 Loss
6.4 Training
6.5 Mesh Deformation
6.6 Example Application
7 Intraoperative Prototype
7.1 Image Acquisition
7.2 Stereo Calibration
7.3 Image Rectification, Disparity- and Depth- estimation
7.4 Liver Segmentation
7.4.1 Synthetic Image Generation
7.4.2 Automatic Segmentation
7.4.3 Manual Segmentation Modifier
7.5 SLAM
7.6 Dense Reconstruction
7.7 Rigid Registration
7.8 Non-Rigid Registration
7.9 Rendering
7.10 Robotic Operating System
8 Evaluation
8.1 Evaluation Datasets
8.1.1 In-Silico
8.1.2 Phantom Torso and Liver
8.1.3 In-Vivo, Human, Breathing Motion
8.1.4 In-Vivo, Human, Laparoscopy
8.2 Metrics
8.2.1 Mean Displacement Error
8.2.2 Target Registration Error (TRE)
8.2.3 Champfer Distance
8.2.4 Volumetric Change
8.3 Evaluation of the Synthetic Training Data
8.4 Data-Driven Biomechanical Model (DDBM)
8.4.1 Amount of Intraoperative Surface
8.4.2 Dynamic Simulation
8.5 Volume to Surface Registration Network (V2S-Net)
8.5.1 Amount of Intraoperative Surface
8.5.2 Dependency on Initial Rigid Alignment
8.5.3 Registration Accuracy in Comparison to Surface Noise
8.5.4 Registration Accuracy in Comparison to Material Stiffness
8.5.5 Champfer-Distance vs. Mean Displacement Error
8.5.6 In-vivo, Human Breathing Motion
8.6 Full Intraoperative Pipeline
8.6.1 Intraoperative Reconstruction: SLAM and Intraoperative Map
8.6.2 Full Pipeline on Laparoscopic Human Data
8.7 Timing
9 Discussion
9.1 Intraoperative Model
9.2 Physical Accuracy
9.3 Limitations in Training Data
9.4 Limitations Caused by Difference in Pre- and Intraoperative Modalities
9.5 Ambiguity
9.6 Intraoperative Prototype
10 Conclusion
11 List of Publications
List of Figures
Bibliograph
A Survey on Deep Learning in Medical Image Analysis
Deep learning algorithms, in particular convolutional networks, have rapidly
become a methodology of choice for analyzing medical images. This paper reviews
the major deep learning concepts pertinent to medical image analysis and
summarizes over 300 contributions to the field, most of which appeared in the
last year. We survey the use of deep learning for image classification, object
detection, segmentation, registration, and other tasks and provide concise
overviews of studies per application area. Open challenges and directions for
future research are discussed.Comment: Revised survey includes expanded discussion section and reworked
introductory section on common deep architectures. Added missed papers from
before Feb 1st 201
Comparing Regularized Kelvinlet Functions and the Finite Element Method for Registration of Medical Images to Sparse Organ Data
Image-guided surgery collocates patient-specific data with the physical
environment to facilitate surgical decision making in real-time. Unfortunately,
these guidance systems commonly become compromised by intraoperative
soft-tissue deformations. Nonrigid image-to-physical registration methods have
been proposed to compensate for these deformations, but intraoperative clinical
utility requires compatibility of these techniques with data sparsity and
temporal constraints in the operating room. While linear elastic finite element
models are effective in sparse data scenarios, the computation time for finite
element simulation remains a limitation to widespread deployment. This paper
proposes a registration algorithm that uses regularized Kelvinlets, which are
analytical solutions to linear elasticity in an infinite domain, to overcome
these barriers. This algorithm is demonstrated and compared to finite
element-based registration on two datasets: a phantom dataset representing
liver deformations and an in vivo dataset representing breast deformations. The
regularized Kelvinlets algorithm resulted in a significant reduction in
computation time compared to the finite element method. Accuracy as evaluated
by target registration error was comparable between both methods. Average
target registration errors were 4.6 +/- 1.0 and 3.2 +/- 0.8 mm on the liver
dataset and 5.4 +/- 1.4 and 6.4 +/- 1.5 mm on the breast dataset for the
regularized Kelvinlets and finite element method models, respectively. This
work demonstrates the generalizability of using a regularized Kelvinlets
registration algorithm on multiple soft tissue elastic organs. This method may
improve and accelerate registration for image-guided surgery applications, and
it shows the potential of using regularized Kelvinlets solutions on medical
imaging data.Comment: 17 pages, 9 figure
Adversarial training with cycle consistency for unsupervised super-resolution in endomicroscopy
In recent years, endomicroscopy has become increasingly used for diagnostic
purposes and interventional guidance. It can provide intraoperative aids for
real-time tissue characterization and can help to perform visual investigations
aimed for example to discover epithelial cancers. Due to physical constraints
on the acquisition process, endomicroscopy images, still today have a low
number of informative pixels which hampers their quality. Post-processing
techniques, such as Super-Resolution (SR), are a potential solution to increase
the quality of these images. SR techniques are often supervised, requiring
aligned pairs of low-resolution (LR) and high-resolution (HR) images patches to
train a model. However, in our domain, the lack of HR images hinders the
collection of such pairs and makes supervised training unsuitable. For this
reason, we propose an unsupervised SR framework based on an adversarial deep
neural network with a physically-inspired cycle consistency, designed to impose
some acquisition properties on the super-resolved images. Our framework can
exploit HR images, regardless of the domain where they are coming from, to
transfer the quality of the HR images to the initial LR images. This property
can be particularly useful in all situations where pairs of LR/HR are not
available during the training. Our quantitative analysis, validated using a
database of 238 endomicroscopy video sequences from 143 patients, shows the
ability of the pipeline to produce convincing super-resolved images. A Mean
Opinion Score (MOS) study also confirms this quantitative image quality
assessment.Comment: Accepted for publication on Medical Image Analysis journa
- …