24 research outputs found
Placental vessel-guided hybrid framework for fetoscopic mosaicking
Fetoscopic laser photocoagulation is used to treat twin-to-twin transfusion syndrome; however, this procedure is hindered because of difficulty in visualising the intraoperative surgical environment due to limited surgical field-of-view, unusual placenta position, limited manoeuvrability of the fetoscope and poor visibility due to fluid turbidity and occlusions. Fetoscopic video mosaicking can create an expanded field-of-view image of the fetoscopic intraoperative environment, which could support the surgeons in localising the vascular anastomoses during the fetoscopic procedure. However, classical handcrafted feature matching methods fail on in vivo fetoscopic videos. An existing state-of-the-art method on fetoscopic mosaicking relies on vessel presence and fails when vessels are not present in the view. We propose a vessel-guided hybrid fetoscopic mosaicking framework that mutually benefits from a placental vessel-based registration and a deep learning-based dense matching method to optimise the overall performance. A selection mechanism is implemented based on vessels’ appearance consistency and photometric error minimisation for choosing the best pairwise transformation. Using the extended fetoscopy placenta dataset, we experimentally show the robustness of the proposed framework, over the state-of-the-art methods, even in vessel-free, low-textured, or low illumination non-planar fetoscopic views
Deep Sequential Mosaicking of Fetoscopic Videos
Twin-to-twin transfusion syndrome treatment requires fetoscopic laser
photocoagulation of placental vascular anastomoses to regulate blood flow to
both fetuses. Limited field-of-view (FoV) and low visual quality during
fetoscopy make it challenging to identify all vascular connections. Mosaicking
can align multiple overlapping images to generate an image with increased FoV,
however, existing techniques apply poorly to fetoscopy due to the low visual
quality, texture paucity, and hence fail in longer sequences due to the drift
accumulated over time. Deep learning techniques can facilitate in overcoming
these challenges. Therefore, we present a new generalized Deep Sequential
Mosaicking (DSM) framework for fetoscopic videos captured from different
settings such as simulation, phantom, and real environments. DSM extends an
existing deep image-based homography model to sequential data by proposing
controlled data augmentation and outlier rejection methods. Unlike existing
methods, DSM can handle visual variations due to specular highlights and
reflection across adjacent frames, hence reducing the accumulated drift. We
perform experimental validation and comparison using 5 diverse fetoscopic
videos to demonstrate the robustness of our framework.Comment: Accepted at MICCAI 201
Robust fetoscopic mosaicking from deep learned flow fields
PURPOSE: Fetoscopic laser photocoagulation is a minimally invasive procedure to treat twin-to-twin transfusion syndrome during pregnancy by stopping irregular blood flow in the placenta. Building an image mosaic of the placenta and its network of vessels could assist surgeons to navigate in the challenging fetoscopic environment during the procedure. METHODOLOGY: We propose a fetoscopic mosaicking approach by combining deep learning-based optical flow with robust estimation for filtering inconsistent motions that occurs due to floating particles and specularities. While the current state of the art for fetoscopic mosaicking relies on clearly visible vessels for registration, our approach overcomes this limitation by considering the motion of all consistent pixels within consecutive frames. We also overcome the challenges in applying off-the-shelf optical flow to fetoscopic mosaicking through the use of robust estimation and local refinement. RESULTS: We compare our proposed method against the state-of-the-art vessel-based and optical flow-based image registration methods, and robust estimation alternatives. We also compare our proposed pipeline using different optical flow and robust estimation alternatives. CONCLUSIONS: Through analysis of our results, we show that our method outperforms both the vessel-based state of the art and LK, noticeably when vessels are either poorly visible or too thin to be reliably identified. Our approach is thus able to build consistent placental vessel mosaics in challenging cases where currently available alternatives fail
Globally Optimal Fetoscopic Mosaicking Based on Pose Graph Optimisation With Affine Constraints
Fetoscopic laser ablation surgery could be guided using a high-quality panorama of the operating site, representing a map of the placental vasculature. This can be achieved during the initial inspection phase of the procedure using image mosaicking techniques. Due to the lack of camera calibration in the operating room, it has been mostly modelled as an affine registration problem. While previous work mostly focuses on image feature extraction for visual odometry, the challenges related to large-scale reconstruction (re-localisation, loop closure, drift correction) remain largely unaddressed in this context. This letter proposes using pose graph optimisation to produce globally optimal image mosaics of placental vessels. Our approach follows the SLAM framework with a front-end for visual odometry and a back-end for long-term refinement. Our front-end uses a recent state-of-the-art odometry approach based on vessel segmentation, which is then managed by a key-frame structure and the bag-of-words (BoW) scheme to retrieve loop closures. The back-end, which is our key contribution, models odometry and loop closure constraints as a pose graph with affine warpings between states. This problem in the special Euclidean space cannot be solved by existing pose graph algorithms and available libraries such as G2O. We model states on affine Lie group with local linearisation in its Lie algebra. The cost function is established using Mahalanobis distance with the vectorisation of the Lie algebra. Finally, an iterative optimisation algorithm is adopted to minimise the cost function. The proposed pose graph optimisation is first validated on simulation data with a synthetic trajectory that has different levels of noise and different numbers of loop closures. Then the whole system is validated using real fetoscopic data that has three sequences with different numbers of frames and loop closures. Experimental results validate the advantage of the proposed method compared with baselines
Medical Image Analysis using Deep Relational Learning
In the past ten years, with the help of deep learning, especially the rapid
development of deep neural networks, medical image analysis has made remarkable
progress. However, how to effectively use the relational information between
various tissues or organs in medical images is still a very challenging
problem, and it has not been fully studied. In this thesis, we propose two
novel solutions to this problem based on deep relational learning. First, we
propose a context-aware fully convolutional network that effectively models
implicit relation information between features to perform medical image
segmentation. The network achieves the state-of-the-art segmentation results on
the Multi Modal Brain Tumor Segmentation 2017 (BraTS2017) and Multi Modal Brain
Tumor Segmentation 2018 (BraTS2018) data sets. Subsequently, we propose a new
hierarchical homography estimation network to achieve accurate medical image
mosaicing by learning the explicit spatial relationship between adjacent
frames. We use the UCL Fetoscopy Placenta dataset to conduct experiments and
our hierarchical homography estimation network outperforms the other
state-of-the-art mosaicing methods while generating robust and meaningful
mosaicing result on unseen frames.Comment: arXiv admin note: substantial text overlap with arXiv:2007.0778
Robust endoscopic image mosaicking via fusion of multimodal estimation
We propose an endoscopic image mosaicking algorithm that is robust to light conditioning changes, specular reflections, and feature-less scenes. These conditions are especially common in minimally invasive surgery where the light source moves with the camera to dynamically illuminate close range scenes. This makes it difficult for a single image registration method to robustly track camera motion and then generate consistent mosaics of the expanded surgical scene across different and heterogeneous environments. Instead of relying on one specialised feature extractor or image registration method, we propose to fuse different image registration algorithms according to their uncertainties, formulating the problem as affine pose graph optimisation. This allows to combine landmarks, dense intensity registration, and learning-based approaches in a single framework. To demonstrate our application we consider deep learning-based optical flow, hand-crafted features, and intensity-based registration, however, the framework is general and could take as input other sources of motion estimation, including other sensor modalities. We validate the performance of our approach on three datasets with very different characteristics to highlighting its generalisability, demonstrating the advantages of our proposed fusion framework. While each individual registration algorithm eventually fails drastically on certain surgical scenes, the fusion approach flexibly determines which algorithms to use and in which proportion to more robustly obtain consistent mosaics
Deep Placental Vessel Segmentation for Fetoscopic Mosaicking
During fetoscopic laser photocoagulation, a treatment for twin-to-twin
transfusion syndrome (TTTS), the clinician first identifies abnormal placental
vascular connections and laser ablates them to regulate blood flow in both
fetuses. The procedure is challenging due to the mobility of the environment,
poor visibility in amniotic fluid, occasional bleeding, and limitations in the
fetoscopic field-of-view and image quality. Ideally, anastomotic placental
vessels would be automatically identified, segmented and registered to create
expanded vessel maps to guide laser ablation, however, such methods have yet to
be clinically adopted. We propose a solution utilising the U-Net architecture
for performing placental vessel segmentation in fetoscopic videos. The obtained
vessel probability maps provide sufficient cues for mosaicking alignment by
registering consecutive vessel maps using the direct intensity-based technique.
Experiments on 6 different in vivo fetoscopic videos demonstrate that the
vessel intensity-based registration outperformed image intensity-based
registration approaches showing better robustness in qualitative and
quantitative comparison. We additionally reduce drift accumulation to
negligible even for sequences with up to 400 frames and we incorporate a scheme
for quantifying drift error in the absence of the ground-truth. Our paper
provides a benchmark for fetoscopy placental vessel segmentation and
registration by contributing the first in vivo vessel segmentation and
fetoscopic videos dataset.Comment: Accepted at MICCAI 202
Vision based robot assistance in TTTS fetal surgery
© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.This paper presents an accurate and robust tracking vision algorithm for Fetoscopic Laser Photo-coagulation (FLP) surgery for Twin-Twin Transfusion Syndrome (TTTS). The aim of the proposed method is to assist surgeons during anastomosis localization, coagulation and review using a tele-operated robotic system. The algorithm computes the relative position of the fetoscope tool tip with respect to the placenta, via local vascular structure registration.The algorithm uses image features (local superficial vascular structures of the placenta’s surface) to automatically match consecutive fetoscopic images. It is composed of three sequential steps: image processing (filtering, binarization and vascular structures segmentation); relevant Points Of Interest (POIs) seletion; and image registration between consecutive images.The algorithm has to deal with the low quality of fetoscopic images, the liquid and dirty environment inside the placenta jointly with the thin diameter of the fetoscope optics and low amount of environment light reduces the image quality. The obtained images are blurred, noisy and with very poor color components.The tracking system has been tested using real video sequences of FLP surgery for TTTS. The computational performance enables real time tracking, locally guiding the robot over the placenta’s surface with enough accuracy.Peer ReviewedPostprint (author's final draft
FetReg: Placental Vessel Segmentation and Registration in Fetoscopy Challenge Dataset
Fetoscopy laser photocoagulation is a widely used procedure for the treatment of Twin-to-Twin Transfusion Syndrome (TTTS), that occur in mono-chorionic multiple pregnancies due to placental vascular anastomoses. This procedure is particularly challenging due to limited field of view, poor manoeuvrability of the fetoscope, poor visibility due to fluid turbidity, variability in light source, and unusual position of the placenta. This may lead to increased procedural time and incomplete ablation, resulting in persistent TTTS. Computer-assisted intervention may help overcome these challenges by expanding the fetoscopic field of view through video mosaicking and providing better visualization of the vessel network. However, the research and development in this domain remain limited due to unavailability of high-quality data to encode the intra- and inter-procedure variability. Through the \textit{Fetoscopic Placental Vessel Segmentation and Registration (FetReg)} challenge, we present a large-scale multi-centre dataset for the development of generalized and robust semantic segmentation and video mosaicking algorithms for the fetal environment with a focus on creating drift-free mosaics from long duration fetoscopy videos. In this paper, we provide an overview of the FetReg dataset, challenge tasks, evaluation metrics and baseline methods for both segmentation and registration. Baseline methods results on the FetReg dataset shows that our dataset poses interesting challenges, offering large opportunity for the creation of novel methods and models through a community effort initiative guided by the FetReg challenge
FetReg: Placental Vessel Segmentation and Registration in Fetoscopy Challenge Dataset
Fetoscopy laser photocoagulation is a widely used procedure for the treatment
of Twin-to-Twin Transfusion Syndrome (TTTS), that occur in mono-chorionic
multiple pregnancies due to placental vascular anastomoses. This procedure is
particularly challenging due to limited field of view, poor manoeuvrability of
the fetoscope, poor visibility due to fluid turbidity, variability in light
source, and unusual position of the placenta. This may lead to increased
procedural time and incomplete ablation, resulting in persistent TTTS.
Computer-assisted intervention may help overcome these challenges by expanding
the fetoscopic field of view through video mosaicking and providing better
visualization of the vessel network. However, the research and development in
this domain remain limited due to unavailability of high-quality data to encode
the intra- and inter-procedure variability. Through the \textit{Fetoscopic
Placental Vessel Segmentation and Registration (FetReg)} challenge, we present
a large-scale multi-centre dataset for the development of generalized and
robust semantic segmentation and video mosaicking algorithms for the fetal
environment with a focus on creating drift-free mosaics from long duration
fetoscopy videos. In this paper, we provide an overview of the FetReg dataset,
challenge tasks, evaluation metrics and baseline methods for both segmentation
and registration. Baseline methods results on the FetReg dataset shows that our
dataset poses interesting challenges, offering large opportunity for the
creation of novel methods and models through a community effort initiative
guided by the FetReg challenge