4 research outputs found
Distilled Visual and Robot Kinematics Embeddings for Metric Depth Estimation in Monocular Scene Reconstruction
Estimating precise metric depth and scene reconstruction from monocular
endoscopy is a fundamental task for surgical navigation in robotic surgery.
However, traditional stereo matching adopts binocular images to perceive the
depth information, which is difficult to transfer to the soft robotics-based
surgical systems due to the use of monocular endoscopy. In this paper, we
present a novel framework that combines robot kinematics and monocular
endoscope images with deep unsupervised learning into a single network for
metric depth estimation and then achieve 3D reconstruction of complex anatomy.
Specifically, we first obtain the relative depth maps of surgical scenes by
leveraging a brightness-aware monocular depth estimation method. Then, the
corresponding endoscope poses are computed based on non-linear optimization of
geometric and photometric reprojection residuals. Afterwards, we develop a
Depth-driven Sliding Optimization (DDSO) algorithm to extract the scaling
coefficient from kinematics and calculated poses offline. By coupling the
metric scale and relative depth data, we form a robust ensemble that represents
the metric and consistent depth. Next, we treat the ensemble as supervisory
labels to train a metric depth estimation network for surgeries (i.e.,
MetricDepthS-Net) that distills the embeddings from the robot kinematics,
endoscopic videos, and poses. With accurate metric depth estimation, we utilize
a dense visual reconstruction method to recover the 3D structure of the whole
surgical site. We have extensively evaluated the proposed framework on public
SCARED and achieved comparable performance with stereo-based depth estimation
methods. Our results demonstrate the feasibility of the proposed approach to
recover the metric depth and 3D structure with monocular inputs
Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery
Objective: The computation of anatomical information and laparoscope position
is a fundamental block of surgical navigation in Minimally Invasive Surgery
(MIS). Recovering a dense 3D structure of surgical scene using visual cues
remains a challenge, and the online laparoscopic tracking primarily relies on
external sensors, which increases system complexity. Methods: Here, we propose
a learning-driven framework, in which an image-guided laparoscopic localization
with 3D reconstructions of complex anatomical structures is obtained. To
reconstruct the 3D structure of the whole surgical environment, we first
fine-tune a learning-based stereoscopic depth perception method, which is
robust to the texture-less and variant soft tissues, for depth estimation.
Then, we develop a dense visual reconstruction algorithm to represent the scene
by surfels, estimate the laparoscope poses and fuse the depth maps into a
unified reference coordinate for tissue reconstruction. To estimate poses of
new laparoscope views, we achieve a coarse-to-fine localization method, which
incorporates our reconstructed 3D model. Results: We evaluate the
reconstruction method and the localization module on three datasets, namely,
the stereo correspondence and reconstruction of endoscopic data (SCARED), the
ex-vivo phantom and tissue data collected with Universal Robot (UR) and Karl
Storz Laparoscope, and the in-vivo DaVinci robotic surgery dataset, where the
reconstructed 3D structures have rich details of surface texture with an
accuracy error under 1.71 mm and the localization module can accurately track
the laparoscope with only images as input. Conclusions: Experimental results
demonstrate the superior performance of the proposed method in 3D anatomy
reconstruction and laparoscopic localization. Significance: The proposed
framework can be potentially extended to the current surgical navigation
system
Three-Dimensional Collision Avoidance Method for Robot-Assisted Minimally Invasive Surgery
In the robot-assisted minimally invasive surgery, if a collision occurs, the robot system program could be damaged, and normal tissues could be injured. To avoid collisions during surgery, a 3-dimensional collision avoidance method is proposed in this paper. The proposed method is predicated on the design of 3 strategic vectors: the collision-with-instrument-avoidance (CI) vector, the collision-with-tissues-avoidance (CT) vector, and the constrained-control (CC) vector. The CI vector demarcates 3 specific directions to forestall collision among the surgical instruments. The CT vector, on the other hand, comprises 2 components tailored to prevent inadvertent contact between the robot-controlled instrument and nontarget tissues. Meanwhile, the CC vector is introduced to guide the endpoint of the robot-controlled instrument toward the desired position, ensuring precision in its movements, in alignment with the surgical goals. Simulation results verify the proposed collision avoidance method for robot-assisted minimally invasive surgery. The code and data are available at https://github.com/cynerelee/collision-avoidance
Design of a LiquidâDriven Laser Scanner with Low Voltage Based on LiquidâInfused Membrane
Laser energy is commonly used in tissue ablation, wound suturing, and other precise manipulations during surgery. However, currently available laser scanners require further improvements in terms of miniaturization, driving voltage, and stability to steer the laser beam accurately within a constrained environment. Herein, the development of a liquidâdriven laser scanner installed on the end effector of a continuum endoscope to perform fast and reliable laser steering is proposed. The developed laser scanner is 7âmm in diameter and 7âmm in length, and it is actuated with a voltage lower than 15âV due to the liquidâinfused membrane. The miniature size and low driving voltage of the proposed laser scanner facilitate safe laserâassisted surgery in confined spaces. A theoretical model is established to predict laser spot position quantitatively, and laser steering ability is also tested experimentally. The fiberâdelivered laser beam can be steered for 21.2° (±10.6°) with a standard deviation of 0.3° in 1000 cycles, demonstrating excellent stability. A laser steering speed of up to 27.3âmmâsâ1 and a reflection loss of less than 3.1% are achieved