64 research outputs found
Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources
Many tasks in robot-assisted surgeries (RAS) can be represented by finite-state machines (FSMs), where each state represents either an action (such as picking up a needle) or an observation (such as bleeding). A crucial step towards the automation of such surgical tasks is the temporal perception of the current surgical scene, which requires a real-time estimation of the states in the FSMs. The objective of this work is to estimate the current state of the surgical task based on the actions performed or events occurred as the task progresses. We propose Fusion-KVE, a unified surgical state estimation model that incorporates multiple data sources including the Kinematics, Vision, and system Events. Additionally, we examine the strengths and weaknesses of different state estimation models in segmenting states with different representative features or levels of granularity. We evaluate our model on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), as well as a more complex dataset involving robotic intra-operative ultrasound (RIOUS) imaging, created using the da Vinci® Xi surgical system. Our model achieves a superior frame-wise state estimation accuracy up to 89.4%, which improves the state-of-the-art surgical state estimation models in both JIGSAWS suturing dataset and our RIOUS dataset
Recommended from our members
Broad role for YBX1 in defining the small noncoding RNA composition of exosomes.
RNA is secreted from cells enclosed within extracellular vesicles (EVs). Defining the RNA composition of EVs is challenging due to their coisolation with contaminants, lack of knowledge of the mechanisms of RNA sorting into EVs, and limitations of conventional RNA-sequencing methods. Here we present our observations using thermostable group II intron reverse transcriptase sequencing (TGIRT-seq) to characterize the RNA extracted from HEK293T cell EVs isolated by flotation gradient ultracentrifugation and from exosomes containing the tetraspanin CD63 further purified from the gradient fractions by immunoisolation. We found that EV-associated transcripts are dominated by full-length, mature transfer RNAs (tRNAs) and other small noncoding RNAs (ncRNAs) encapsulated within vesicles. A substantial proportion of the reads mapping to protein-coding genes, long ncRNAs, and antisense RNAs were due to DNA contamination on the surface of vesicles. Nevertheless, sequences mapping to spliced mRNAs were identified within HEK293T cell EVs and exosomes, among the most abundant being transcripts containing a 5' terminal oligopyrimidine (5' TOP) motif. Our results indicate that the RNA-binding protein YBX1, which is required for the sorting of selected miRNAs into exosomes, plays a role in the sorting of highly abundant small ncRNA species, including tRNAs, Y RNAs, and Vault RNAs. Finally, we obtained evidence for an EV-specific tRNA modification, perhaps indicating a role for posttranscriptional modification in the sorting of some RNA species into EVs. Our results suggest that EVs and exosomes could play a role in the purging and intercellular transfer of excess free RNAs, including full-length tRNAs and other small ncRNAs
daVinciNet: Joint Prediction of Motion and Surgical State in Robot-Assisted Surgery
This paper presents a technique to concurrently and jointly predict the
future trajectories of surgical instruments and the future state(s) of surgical
subtasks in robot-assisted surgeries (RAS) using multiple input sources. Such
predictions are a necessary first step towards shared control and supervised
autonomy of surgical subtasks. Minute-long surgical subtasks, such as suturing
or ultrasound scanning, often have distinguishable tool kinematics and visual
features, and can be described as a series of fine-grained states with
transition schematics. We propose daVinciNet - an end-to-end dual-task model
for robot motion and surgical state predictions. daVinciNet performs concurrent
end-effector trajectory and surgical state predictions using features extracted
from multiple data streams, including robot kinematics, endoscopic vision, and
system events. We evaluate our proposed model on an extended Robotic
Intra-Operative Ultrasound (RIOUS+) imaging dataset collected on a da Vinci Xi
surgical system and the JHU-ISI Gesture and Skill Assessment Working Set
(JIGSAWS). Our model achieves up to 93.85% short-term (0.5s) and 82.11%
long-term (2s) state prediction accuracy, as well as 1.07mm short-term and
5.62mm long-term trajectory prediction error.Comment: Accepted to IROS 202
daVinciNet: Joint Prediction of Motion and Surgical State in Robot-Assisted Surgery
This paper presents a technique to concurrently and jointly predict the future trajectories of surgical instruments and the future state(s) of surgical subtasks in robot-assisted surgeries (RAS) using multiple input sources. Such predictions are a necessary first step towards shared control and supervised autonomy of surgical subtasks. Minute-long surgical subtasks, such as suturing or ultrasound scanning, often have distinguishable tool kinematics and visual features, and can be described as a series of fine-grained states with transition schematics. We propose daVinciNet - an end-to-end dual-task model for robot motion and surgical state predictions. daVinciNet performs concurrent end-effector trajectory and surgical state predictions using features extracted from multiple data streams, including robot kinematics, endoscopic vision, and system events. We evaluate our proposed model on an extended Robotic Intra-Operative Ultrasound (RIOUS+) imaging dataset collected on a da Vinci Xi surgical system and the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our model achieves up to 93.85% short-term (0.5s) and 82.11% long-term (2s) state prediction accuracy, as well as 1.07mm short-term and 5.62mm long-term trajectory prediction error
Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources
Many tasks in robot-assisted surgeries (RAS) can be represented by finite-state machines (FSMs), where each state represents either an action (such as picking up a needle) or an observation (such as bleeding). A crucial step towards the automation of such surgical tasks is the temporal perception of the current surgical scene, which requires a real-time estimation of the states in the FSMs. The objective of this work is to estimate the current state of the surgical task based on the actions performed or events occurred as the task progresses. We propose Fusion-KVE, a unified surgical state estimation model that incorporates multiple data sources including the Kinematics, Vision, and system Events. Additionally, we examine the strengths and weaknesses of different state estimation models in segmenting states with different representative features or levels of granularity. We evaluate our model on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), as well as a more complex dataset involving robotic intra-operative ultrasound (RIOUS) imaging, created using the da Vinci® Xi surgical system. Our model achieves a superior frame-wise state estimation accuracy up to 89.4%, which improves the state-of-the-art surgical state estimation models in both JIGSAWS suturing dataset and our RIOUS dataset
- …