9 research outputs found
Deep Reinforcement Learning Based System for Intraoperative Hyperspectral Video Autofocusing
Hyperspectral imaging (HSI) captures a greater level of spectral detail than
traditional optical imaging, making it a potentially valuable intraoperative
tool when precise tissue differentiation is essential. Hardware limitations of
current optical systems used for handheld real-time video HSI result in a
limited focal depth, thereby posing usability issues for integration of the
technology into the operating room. This work integrates a focus-tunable liquid
lens into a video HSI exoscope, and proposes novel video autofocusing methods
based on deep reinforcement learning. A first-of-its-kind robotic focal-time
scan was performed to create a realistic and reproducible testing dataset. We
benchmarked our proposed autofocus algorithm against traditional policies, and
found our novel approach to perform significantly () better than
traditional techniques ( mean absolute focal error compared to
). In addition, we performed a blinded usability trial by having
two neurosurgeons compare the system with different autofocus policies, and
found our novel approach to be the most favourable, making our system a
desirable addition for intraoperative HSI.Comment: To be presented at MICCAI 202
Augmented Reality needle ablation guidance tool for Irreversible Electroporation in the pancreas
Irreversible electroporation (IRE) is a soft tissue ablation technique
suitable for treatment of inoperable tumours in the pancreas. The process
involves applying a high voltage electric field to the tissue containing the
mass using needle electrodes, leaving cancerous cells irreversibly damaged and
vulnerable to apoptosis. Efficacy of the treatment depends heavily on the
accuracy of needle placement and requires a high degree of skill from the
operator. In this paper, we describe an Augmented Reality (AR) system designed
to overcome the challenges associated with planning and guiding the needle
insertion process. Our solution, based on the HoloLens (Microsoft, USA)
platform, tracks the position of the headset, needle electrodes and ultrasound
(US) probe in space. The proof of concept implementation of the system uses
this tracking data to render real-time holographic guides on the HoloLens,
giving the user insight into the current progress of needle insertion and an
indication of the target needle trajectory. The operator's field of view is
augmented using visual guides and real-time US feed rendered on a holographic
plane, eliminating the need to consult external monitors. Based on these early
prototypes, we are aiming to develop a system that will lower the skill level
required for IRE while increasing overall accuracy of needle insertion and,
hence, the likelihood of successful treatment.Comment: 6 pages, 5 figures. Proc. SPIE 10576 (2018) Copyright 2018 Society of
Photo Optical Instrumentation Engineers (SPIE). One print or electronic copy
may be made for personal use only. Systematic reproduction and distribution,
duplication of any material in this publication for a fee or for commercial
purposes, or modification of the contents of the publication are prohibite
Synthetic white balancing for intra-operative hyperspectral imaging
Hyperspectral imaging shows promise for surgical applications to
non-invasively provide spatially-resolved, spectral information. For
calibration purposes, a white reference image of a highly-reflective Lambertian
surface should be obtained under the same imaging conditions. Standard white
references are not sterilizable, and so are unsuitable for surgical
environments. We demonstrate the necessity for in situ white references and
address this by proposing a novel, sterile, synthetic reference construction
algorithm. The use of references obtained at different distances and lighting
conditions to the subject were examined. Spectral and color reconstructions
were compared with standard measurements qualitatively and quantitatively,
using and normalised RMSE respectively. The algorithm forms a
composite image from a video of a standard sterile ruler, whose imperfect
reflectivity is compensated for. The reference is modelled as the product of
independent spatial and spectral components, and a scalar factor accounting for
gain, exposure, and light intensity. Evaluation of synthetic references against
ideal but non-sterile references is performed using the same metrics alongside
pixel-by-pixel errors. Finally, intraoperative integration is assessed though
cadaveric experiments. Improper white balancing leads to increases in all
quantitative and qualitative errors. Synthetic references achieve median
pixel-by-pixel errors lower than 6.5% and produce similar reconstructions and
errors to an ideal reference. The algorithm integrated well into surgical
workflow, achieving median pixel-by-pixel errors of 4.77%, while maintaining
good spectral and color reconstruction.Comment: 22 pages, 10 figure
Lightfield hyperspectral imaging in neuro-oncology surgery: an IDEAL 0 and 1 study
IntroductionHyperspectral imaging (HSI) has shown promise in the field of intra-operative imaging and tissue differentiation as it carries the capability to provide real-time information invisible to the naked eye whilst remaining label free. Previous iterations of intra-operative HSI systems have shown limitations, either due to carrying a large footprint limiting ease of use within the confines of a neurosurgical theater environment, having a slow image acquisition time, or by compromising spatial/spectral resolution in favor of improvements to the surgical workflow. Lightfield hyperspectral imaging is a novel technique that has the potential to facilitate video rate image acquisition whilst maintaining a high spectral resolution. Our pre-clinical and first-in-human studies (IDEAL 0 and 1, respectively) demonstrate the necessary steps leading to the first in-vivo use of a real-time lightfield hyperspectral system in neuro-oncology surgery.MethodsA lightfield hyperspectral camera (Cubert Ultris ×50) was integrated in a bespoke imaging system setup so that it could be safely adopted into the open neurosurgical workflow whilst maintaining sterility. Our system allowed the surgeon to capture in-vivo hyperspectral data (155 bands, 350–1,000 nm) at 1.5 Hz. Following successful implementation in a pre-clinical setup (IDEAL 0), our system was evaluated during brain tumor surgery in a single patient to remove a posterior fossa meningioma (IDEAL 1). Feedback from the theater team was analyzed and incorporated in a follow-up design aimed at implementing an IDEAL 2a study.ResultsFocusing on our IDEAL 1 study results, hyperspectral information was acquired from the cerebellum and associated meningioma with minimal disruption to the neurosurgical workflow. To the best of our knowledge, this is the first demonstration of HSI acquisition with 100+ spectral bands at a frame rate over 1Hz in surgery.DiscussionThis work demonstrated that a lightfield hyperspectral imaging system not only meets the design criteria and specifications outlined in an IDEAL-0 (pre-clinical) study, but also that it can translate into clinical practice as illustrated by a successful first in human study (IDEAL 1). This opens doors for further development and optimisation, given the increasing evidence that hyperspectral imaging can provide live, wide-field, and label-free intra-operative imaging and tissue differentiation
Synthetic white balancing for intra-operative hyperspectral imaging
PURPOSE
Hyperspectral imaging shows promise for surgical applications to non-invasively provide spatially resolved, spectral information. For calibration purposes, a white reference image of a highly reflective Lambertian surface should be obtained under the same imaging conditions. Standard white references are not sterilizable and so are unsuitable for surgical environments. We demonstrate the necessity for in situ white references and address this by proposing a novel, sterile, synthetic reference construction algorithm.
APPROACH
The use of references obtained at different distances and lighting conditions to the subject were examined. Spectral and color reconstructions were compared with standard measurements qualitatively and quantitatively, using and normalized RMSE, respectively. The algorithm forms a composite image from a video of a standard sterile ruler, whose imperfect reflectivity is compensated for. The reference is modeled as the product of independent spatial and spectral components, and a scalar factor accounting for gain, exposure, and light intensity. Evaluation of synthetic references against ideal but non-sterile references is performed using the same metrics alongside pixel-by-pixel errors. Finally, intraoperative integration is assessed though cadaveric experiments.
RESULTS
Improper white balancing leads to increases in all quantitative and qualitative errors. Synthetic references achieve median pixel-by-pixel errors lower than 6.5% and produce similar reconstructions and errors to an ideal reference. The algorithm integrated well into surgical workflow, achieving median pixel-by-pixel errors of 4.77% while maintaining good spectral and color reconstruction.
CONCLUSIONS
We demonstrate the importance of in situ white referencing and present a novel synthetic referencing algorithm. This algorithm is suitable for surgery while maintaining the quality of classical data reconstruction