ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools

Real-time tool segmentation from endoscopic videos is an essential part of many computer-assisted robotic surgical systems and of critical importance in robotic surgical data science. We propose two novel deep learning architectures for automatic segmentation of non-rigid surgical instruments. Both methods take advantage of automated deep-learning-based multi-scale feature extraction while trying to maintain an accurate segmentation quality at all resolutions. The two proposed methods encode the multi-scale constraint inside the network architecture. The first proposed architecture enforces it by cascaded aggregation of predictions and the second proposed network does it by means of a holistically-nested architecture where the loss at each scale is taken into account for the optimization process. As the proposed methods are for real-time semantic labeling, both present a reduced number of parameters. We propose the use of parametric rectified linear units for semantic labeling in these small architectures to increase the regularization ability of the design and maintain the segmentation accuracy without overfitting the training sets. We compare the proposed architectures against state-of-the-art fully convolutional networks. We validate our methods using existing benchmark datasets, including ex vivo cases with phantom tissue and different robotic surgical instruments present in the scene. Our results show a statistically significant improved Dice Similarity Coefficient over previous instrument segmentation methods. We analyze our design choices and discuss the key drivers for improving accuracy.Comment: Paper accepted at IROS 201

Deep learning-based fetoscopic mosaicking for field-of-view expansion

PURPOSE: Fetoscopic laser photocoagulation is a minimally invasive surgical procedure used to treat twin-to-twin transfusion syndrome (TTTS), which involves localization and ablation of abnormal vascular connections on the placenta to regulate the blood flow in both fetuses. This procedure is particularly challenging due to the limited field of view, poor visibility, occasional bleeding, and poor image quality. Fetoscopic mosaicking can help in creating an image with the expanded field of view which could facilitate the clinicians during the TTTS procedure. METHODS: We propose a deep learning-based mosaicking framework for diverse fetoscopic videos captured from different settings such as simulation, phantoms, ex vivo, and in vivo environments. The proposed mosaicking framework extends an existing deep image homography model to handle video data by introducing the controlled data generation and consistent homography estimation modules. Training is performed on a small subset of fetoscopic images which are independent of the testing videos. RESULTS: We perform both quantitative and qualitative evaluations on 5 diverse fetoscopic videos (2400 frames) that captured different environments. To demonstrate the robustness of the proposed framework, a comparison is performed with the existing feature-based and deep image homography methods. CONCLUSION: The proposed mosaicking framework outperformed existing methods and generated meaningful mosaic, while reducing the accumulated drift, even in the presence of visual challenges such as specular highlights, reflection, texture paucity, and low video resolution

Body Wall Force Sensor for Simulated Minimally Invasive Surgery: Application to Fetal Surgery

Surgical interventions are increasingly executed minimal invasively. Surgeons insert instruments through tiny incisions in the body and pivot slender instruments to treat organs or tissue below the surface. While a blessing for patients, surgeons need to pay extra attention to overcome the fulcrum effect, reduced haptic feedback and deal with lost hand-eye coordination. The mental load makes it difficult to pay sufficient attention to the forces that are exerted on the body wall. In delicate procedures such as fetal surgery, this might be problematic as irreparable damage could cause premature delivery. As a first attempt to quantify the interaction forces applied on the patient's body wall, a novel 6 degrees of freedom force sensor was developed for an ex-vivo set up. The performance of the sensor was characterised. User experiments were conducted by 3 clinicians on a set up simulating a fetal surgical intervention. During these simulated interventions, the interaction forces were recorded and analysed when a normal instrument was employed. These results were compared with a session where a flexible instrument under haptic guidance was used. The conducted experiments resulted in interesting insights in the interaction forces and stresses that develop during such difficult surgical intervention. The results also implicated that haptic guidance schemes and the use of flexible instruments rather than rigid ones could have a significant impact on the stresses that occur at the body wall

Haptic Guidance in Comanipulated Laser Surgery for Fetal Disorders

The current techniques in minimal invasive surgery allow to treat fetal disorders. In fetal interventions very precise instrument manipulation is required from the surgeon. For instance in the treatment of the twin-to-twin transfusion syndrome (TTTS) it is crucial that the surgeon maintains a specific distance between the tip of the employed instrument and the placenta, while lasering target sites on the placenta. To facilitate this procedure, we suggest a new approach where the surgeon comanipulates the instruments together with a robotic stabilizer arm. The stabilizer arm provides haptic guidance to the surgeon, augmenting the surgeon's dexterity and precision. The first results show that this approach is promising

Quantification of nitrotyrosine in nitrated proteins

For kinetic studies of protein nitration reactions, we have developed a method for the quantification of nitrotyrosine residues in protein molecules by liquid chromatography coupled to a diode array detector of ultraviolet-visible absorption. Nitrated bovine serum albumin (BSA) and nitrated ovalbumin (OVA) were synthesized and used as standards for the determination of the protein nitration degree (ND), which is defined as the average number of nitrotyrosine residues divided by the total number of tyrosine residues in a protein molecule. The obtained calibration curves of the ratio of chromatographic peak areas of absorbance at 357 and at 280 nm vs. nitration degree are nearly the same for BSA and OVA (relative deviations <5%). They are near-linear at low ND (< 0.1) and can be described by a second-order polynomial fit up to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hbox{ND}} = 0.5\left( {{R^2} > 0.99} \right)$$\end{document}. A change of chromatographic column led to changes in absolute peak areas but not in the peak area ratios and related calibration functions, which confirms the robustness of the analytical method. First results of laboratory experiments confirm that the method is applicable for the investigation of the reaction kinetics of protein nitration. The main advantage over alternative methods is that nitration degrees can be efficiently determined without hydrolysis or digestion of the investigated protein molecules

Haptic Guidance Based on All-Optical Ultrasound Distance Sensing for Safer Minimally Invasive Fetal Surgery

By intervening during the early stage of gestation, fetal surgeons aim to correct or minimize the effects of congenital disorders. As compared to postnatal treatment of these disorders, such early interventions can often actually save the life of the fetus and also improve the quality of life of the newborn. However, fetal surgery is considered one of the most challenging disciplines within Minimally Invasive Surgery (MIS), owing to factors such as the fragility of the anatomic features, poor visibility, limited manoeuvrability, and extreme requirements in terms of instrument handling with precise positioning. This work is centered on a fetal laser surgery procedure treating placental disorders. It proposes the use of haptic guidance to enhance the overall safety of this procedure and to simplify instrument handling. A method is described that provides effective guidance by installing a forbidden region virtual fixture over the placenta, thereby safeguarding adequate clearance between the instrument tip and the placenta. With a novel application of all-optical ultrasound distance sensing in which transmission and reception are performed with fibre optics, this method can be used with a sole reliance on intraoperatively acquired data. The added value of the guidance approach, in terms of safety and performance, is demonstrated in a series of experiments with a robotic platform

Nitration of the Pollen Allergen Bet v 1.0101 Enhances the Presentation of Bet v 1-Derived Peptides by HLA-DR on Human Dendritic Cells

Nitration of pollen derived allergens can occur by NO2 and ozone in polluted air and it has already been shown that nitrated major birch (Betula verrucosa) pollen allergen Bet v 1.0101 (Bet v 1) exhibits an increased potency to trigger an immune response. However, the mechanisms by which nitration might contribute to the induction of allergy are still unknown. In this study, we assessed the effect of chemically induced nitration of Bet v 1 on the generation of HLA-DR associated peptides. Human dendritic cells were loaded with unmodified Bet v 1 or nitrated Bet v 1, and the naturally processed HLA-DR associated peptides were subsequently identified by liquid chromatography-mass spectrometry. Nitration of Bet v 1 resulted in enhanced presentation of allergen-derived HLA-DR-associated peptides. Both the copy number of Bet v 1 derived peptides as well as the number of nested clusters was increased. Our study shows that nitration of Bet v 1 alters antigen processing and presentation via HLA-DR, by enhancing both the quality and the quantity of the Bet v 1-specific peptide repertoire. These findings indicate that air pollution can contribute to allergic diseases and might also shed light on the analogous events concerning the nitration of self-proteins

Nitration of the Egg-Allergen Ovalbumin Enhances Protein Allergenicity but Reduces the Risk for Oral Sensitization in a Murine Model of Food Allergy

Nitration of proteins on tyrosine residues, which can occur due to polluted air under "summer smog" conditions, has been shown to increase the allergic potential of allergens. Since nitration of tyrosine residues is also observed during inflammatory responses, this modification could directly influence protein immunogenicity and might therefore contribute to food allergy induction. In the current study we have analyzed the impact of protein nitration on sensitization via the oral route.BALB/c mice were immunized intragastrically by feeding untreated ovalbumin (OVA), sham-nitrated ovalbumin (snOVA) or nitrated ovalbumin (nOVA) with or without concomitant acid-suppression. To analyze the impact of the sensitization route, the allergens were also injected intraperitoneally. Animals being fed OVA or snOVA under acid-suppressive medication developed significantly elevated levels of IgE, and increased titers of specific IgG1 and IgG2a antibodies. Interestingly, oral immunizations of nOVA under anti-acid treatment did not result in IgG and IgE formation. In contrast, intraperitoneal immunization induced high levels of OVA specific IgE, which were significantly increased in the group that received nOVA by injection. Furthermore, nOVA triggered significantly enhanced mediator release from RBL cells passively sensitized with sera from allergic mice. Gastric digestion experiments demonstrated protein nitration to interfere with protein stability as nOVA was easily degraded, whereas OVA and snOVA remained stable up to 120 min. Additionally, HPLC-chip-MS/MS analysis showed that one tyrosine residue (Y(107)) being very efficiently nitrated is part of an ovalbumin epitope recognized exclusively after oral sensitization.These data indicated that despite the enhanced triggering capacity in existing allergy, nitration of OVA may be associated with a reduced de novo sensitizing capability via the oral route due to enhanced protein digestibility and/or changes in antibody epitopes

Synergistic Robotic Surgery: Novel Techniques to Harmonize Surgeon and Robot

During the last 150 years, surgery has evolved from a last resort solution to the highly specialized discipline that it is today. Technological advancement has substantially contributed to this evolution. Thanks to the combined efforts of doctors and engineers, minimally invasive surgery has become a reality, where trauma to the patient is kept so small that recovery is much faster. Yet, this is not the end. The continuing technological progress provides the surgeon with more and more options to improve the quality of treatment. Today, robotic surgery is a prominent representative of this technological innovation. Robots offer the possibility to operate with higher accuracy, speed up the learning process, enhance surgeon comfort and control the motion of complex tools. However, the clinical deployment of these advantages is not straightforward and also substantially differs between surgical disciplines. The focus of this dissertation lies on robotic laparoscopy. Within this medical specialization, telemanipulation systems are presently the standard. With this type of system, the robot copies the movements that the remotely located surgeon executes with a sophisticated joystick. So far, it has however proven difficult to justify the hefty price tag of these robots. The existing systems only employ a fraction of the full potential of robotics. Especially the opportunity to embed intelligent behavior in these robots is hardly exploited. At the same time, the surgeon needs to make compromises to integrate the system by completely revisiting his/her normal approach and workflow. This can for instance be witnessed by the long set-up time of the devices and the distance they create between the surgeon, the team and the patient. This dissertation aims to reduce the distance, both physically and psychologically, that exists between surgeon and robot. It proposes methods that optimize the synergy between both. This allows the surgeon to operate more effectively and safely, supported by a robot that provides assistance in a way that seamlessly fits in with his/her expectations. Based on this starting point, a number of important design guidelines for synergistic robotic surgery are formulated. They promote the comanipulation principle for robotic laparoscopy. In comanipulation, the surgeon and the robot jointly hold and manipulate the surgical instruments, in contrast to the remote control in telemanipulation. To further reduce the impact of the comanipulation system on the surgical practices, it is of importance that the robot can be moved without resistance and that it can automatically align itself with the incision point in the patient. If synergy is to be guaranteed, a careful approach is also required towards robot perception, which is necessary for more advanced robot assistance. The surgeon cannot be burdened with extra steps to collect or interpret data. While these design guidelines are valuable for the surgeon, their practical implementation implies a number of technical challenges, which are studied and tackled in the remainder of this dissertation. The outcomes are every time applied to the particularly demanding discipline of fetal surgery. The first part of the challenges is related to the incision point in laparoscopy and has an impact on the robot control strategy. To ensure that the robot can automatically align itself with the incision point, a robust and responsive method is developed to measure the position of the incision point. Knowledge of this point is subsequently utilized to enhance the performance of the proposed type of comanipulation robot. On the one hand, it is shown how this knowledge of the incision point allows for improved accuracy, precision and bandwidth of instrument tip position measurements. On the other hand, an algorithm is designed that applies this knowledge to expand the stability margins of the robot. The combination of the developed techniques results in safer, more effective and more synergistic robot assistance. The second part of the challenges is related to robot perception. Perception-related problems are elaborated for two clinical use cases: contactless surgery and endoscope guidance. For these use cases, sensors are selected that can be integrated in laparoscopic instruments suitable for fetal surgery. Techniques to process the sensor data into actionable computer models are then developed. These techniques work entirely automatically, without requiring the attention of the surgeon to collect, enrich or verify the data. Through a series of in-vitro and in-vivo experiments, the feasibility of robot assistance for both clinical cases is demonstrated. From this dissertation, it thus follows that intelligent and effective robot assistance can be successfully realized while taking into account all design guidelines for synergy, even for demanding fetal interventions.status: publishe