Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources

Many tasks in robot-assisted surgeries (RAS) can be represented by finite-state machines (FSMs), where each state represents either an action (such as picking up a needle) or an observation (such as bleeding). A crucial step towards the automation of such surgical tasks is the temporal perception of the current surgical scene, which requires a real-time estimation of the states in the FSMs. The objective of this work is to estimate the current state of the surgical task based on the actions performed or events occurred as the task progresses. We propose Fusion-KVE, a unified surgical state estimation model that incorporates multiple data sources including the Kinematics, Vision, and system Events. Additionally, we examine the strengths and weaknesses of different state estimation models in segmenting states with different representative features or levels of granularity. We evaluate our model on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), as well as a more complex dataset involving robotic intra-operative ultrasound (RIOUS) imaging, created using the da Vinci® Xi surgical system. Our model achieves a superior frame-wise state estimation accuracy up to 89.4%, which improves the state-of-the-art surgical state estimation models in both JIGSAWS suturing dataset and our RIOUS dataset

daVinciNet: Joint Prediction of Motion and Surgical State in Robot-Assisted Surgery

This paper presents a technique to concurrently and jointly predict the future trajectories of surgical instruments and the future state(s) of surgical subtasks in robot-assisted surgeries (RAS) using multiple input sources. Such predictions are a necessary first step towards shared control and supervised autonomy of surgical subtasks. Minute-long surgical subtasks, such as suturing or ultrasound scanning, often have distinguishable tool kinematics and visual features, and can be described as a series of fine-grained states with transition schematics. We propose daVinciNet - an end-to-end dual-task model for robot motion and surgical state predictions. daVinciNet performs concurrent end-effector trajectory and surgical state predictions using features extracted from multiple data streams, including robot kinematics, endoscopic vision, and system events. We evaluate our proposed model on an extended Robotic Intra-Operative Ultrasound (RIOUS+) imaging dataset collected on a da Vinci Xi surgical system and the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our model achieves up to 93.85% short-term (0.5s) and 82.11% long-term (2s) state prediction accuracy, as well as 1.07mm short-term and 5.62mm long-term trajectory prediction error.Comment: Accepted to IROS 202

daVinciNet: Joint Prediction of Motion and Surgical State in Robot-Assisted Surgery

This paper presents a technique to concurrently and jointly predict the future trajectories of surgical instruments and the future state(s) of surgical subtasks in robot-assisted surgeries (RAS) using multiple input sources. Such predictions are a necessary first step towards shared control and supervised autonomy of surgical subtasks. Minute-long surgical subtasks, such as suturing or ultrasound scanning, often have distinguishable tool kinematics and visual features, and can be described as a series of fine-grained states with transition schematics. We propose daVinciNet - an end-to-end dual-task model for robot motion and surgical state predictions. daVinciNet performs concurrent end-effector trajectory and surgical state predictions using features extracted from multiple data streams, including robot kinematics, endoscopic vision, and system events. We evaluate our proposed model on an extended Robotic Intra-Operative Ultrasound (RIOUS+) imaging dataset collected on a da Vinci Xi surgical system and the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our model achieves up to 93.85% short-term (0.5s) and 82.11% long-term (2s) state prediction accuracy, as well as 1.07mm short-term and 5.62mm long-term trajectory prediction error

Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources

Many tasks in robot-assisted surgeries (RAS) can be represented by finite-state machines (FSMs), where each state represents either an action (such as picking up a needle) or an observation (such as bleeding). A crucial step towards the automation of such surgical tasks is the temporal perception of the current surgical scene, which requires a real-time estimation of the states in the FSMs. The objective of this work is to estimate the current state of the surgical task based on the actions performed or events occurred as the task progresses. We propose Fusion-KVE, a unified surgical state estimation model that incorporates multiple data sources including the Kinematics, Vision, and system Events. Additionally, we examine the strengths and weaknesses of different state estimation models in segmenting states with different representative features or levels of granularity. We evaluate our model on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), as well as a more complex dataset involving robotic intra-operative ultrasound (RIOUS) imaging, created using the da Vinci® Xi surgical system. Our model achieves a superior frame-wise state estimation accuracy up to 89.4%, which improves the state-of-the-art surgical state estimation models in both JIGSAWS suturing dataset and our RIOUS dataset

(Cyclopentadienone)iron-Catalyzed Transfer Dehydrogenation of Symmetrical and Unsymmetrical Diols to Lactones

Air-stable iron carbonyl compounds bearing cyclopentadienone ligands with varying substitution were explored as catalysts in dehydrogenative diol lactonization reactions using acetone as both the solvent and hydrogen acceptor. Two catalysts with trimethylsilyl groups in the 2- and 5-positions, [2,5-(SiMe3)2-3,4-(CH2)4(η4-C4C═O)]Fe(CO)3 (1) and [2,5-(SiMe3)2-3,4-(CH2)3(η4-C4C═O)]Fe(CO)3 (2), were found to be the most active, with 2 being the most selective in the lactonization of diols containing both primary and secondary alcohols. Lactones containing five-, six-, and seven-membered rings were successfully synthesized, and no over-oxidations to carboxylic acids were detected. The lactonization of unsymmetrical diols containing two primary alcohols occurred with catalyst 1, but selectivity was low based on alcohol electronics and modest based on alcohol sterics. Evidence for a transfer dehydrogenation mechanism was found, and insight into the origin of selectivity in the lactonization of 1°/2° diols was obtained. Additionally, spectroscopic evidence for a trimethylamine-ligated iron species formed in solution during the reaction was discovered

Splenic release of platelets contributes to increased circulating platelet size and inflammation after myocardial infarction

Abstract Acute myocardial infarction (AMI) is characterized by a rapid increase in circulating platelet size but the mechanism for this is unclear. Large platelets are hyperactive and associated with adverse clinical outcomes. We determined mean platelet volume (MPV) and platelet-monocyte conjugation (PMC) using blood samples from patients, and blood and the spleen from mice with AMI. We further measured changes in platelet size, PMC, cardiac and splenic contents of platelets and leucocyte infiltration into the mouse heart. In AMI patients, circulating MPV and PMC increased at 1-3 h post-MI and MPV returned to reference levels within 24 h after admission. In mice with MI, increases in platelet size and PMC became evident within 12 h and were sustained up to 72 h. Splenic platelets are bigger than circulating platelets in normal or infarct mice. At 24 h post-MI, splenic platelet storage was halved whereas cardiac platelets increased by 4-fold. Splenectomy attenuated all changes observed in the blood, reduced leucocyte and platelet accumulation in the infarct myocardium, limited infarct size and alleviated cardiac dilatation and dysfunction. AMI-induced elevated circulating levels of adenosine diphosphate and catecholamines in both human and the mouse, which may trigger splenic platelet release. Pharmacological inhibition of angiotensin-converting enzyme, β 1 -adrenergic receptor or platelet P2Y 12 receptor reduced platelet abundance in the murine infarct myocardium albeit having diverse effects on platelet size and PMC. In conclusion, AMI evokes release of splenic platelets, which contributes to the increase in platelet size and PMC and facilitates myocardial accumulation of platelets and leucocytes, thereby promoting post-infarct inflammation

Computational fluid dynamics modeling patterns and force characteristics of flow over in-line four square cylinders

The flow over four square cylinders in an in-line, square arrangement was numerically investigated by using the finite volume method with CFD techniques. The working fluid is an incompressible ideal gas. The length of the sides of the array, L, is equal. The analysis is carried out for a Reynolds number of 300, with center-to-center distance ratios, L/D, ranging from 1.5 to 8.0. To fully understand the flow mechanism, details in terms of lift and drag coefficients and Strouhal numbers of the unsteady wake frequencies are analyzed, and the vortex shedding patterns around the four square cylinders are described. It is concluded that L/D has important effects on the drag and lift coefficients, vortex shedding frequencies, and flow field characteristics