ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools

Real-time tool segmentation from endoscopic videos is an essential part of many computer-assisted robotic surgical systems and of critical importance in robotic surgical data science. We propose two novel deep learning architectures for automatic segmentation of non-rigid surgical instruments. Both methods take advantage of automated deep-learning-based multi-scale feature extraction while trying to maintain an accurate segmentation quality at all resolutions. The two proposed methods encode the multi-scale constraint inside the network architecture. The first proposed architecture enforces it by cascaded aggregation of predictions and the second proposed network does it by means of a holistically-nested architecture where the loss at each scale is taken into account for the optimization process. As the proposed methods are for real-time semantic labeling, both present a reduced number of parameters. We propose the use of parametric rectified linear units for semantic labeling in these small architectures to increase the regularization ability of the design and maintain the segmentation accuracy without overfitting the training sets. We compare the proposed architectures against state-of-the-art fully convolutional networks. We validate our methods using existing benchmark datasets, including ex vivo cases with phantom tissue and different robotic surgical instruments present in the scene. Our results show a statistically significant improved Dice Similarity Coefficient over previous instrument segmentation methods. We analyze our design choices and discuss the key drivers for improving accuracy.Comment: Paper accepted at IROS 201

Soft Robot-Assisted Minimally Invasive Surgery and Interventions: Advances and Outlook

Since the emergence of soft robotics around two decades ago, research interest in the field has escalated at a pace. It is fuelled by the industry's appreciation of the wide range of soft materials available that can be used to create highly dexterous robots with adaptability characteristics far beyond that which can be achieved with rigid component devices. The ability, inherent in soft robots, to compliantly adapt to the environment, has significantly sparked interest from the surgical robotics community. This article provides an in-depth overview of recent progress and outlines the remaining challenges in the development of soft robotics for minimally invasive surgery

Robotic manipulators for in situ inspections of jet engines

Jet engines need to be inspected periodically and, in some instances, repaired. Currently, some of these maintenance operations require the engine to be removed from the wing and dismantled, which has a significant associated cost. The capability of performing some of these inspections and repairs while the engine is on-wing could lead to important cost savings. However, existing technology for on-wing operations is limited, and does not suffice to satisfy some of the needs. In this work, the problem of performing on-wing operations such as inspection and repair is analysed, and after an extensive literature review, a novel robotic system for the on-wing insertion and deployment of probes or other tools is proposed. The system consists of a fine-positioner, which is a miniature and dexterous robotic manipulator; a gross-positioner, which is a device to insert the fine-positioner to the engine region of interest; an end-effector, such as a probe; a deployment mechanism, which is a passive device to ensure correct contact between probe and component; and a feedback system that provides information about the robot state for control. The research and development work conducted to address the main challenges to create this robotic system is presented in this thesis. The work is focussed on the fine-positioner, as it is the most relevant and complex part of the system. After a literature review of relevant work, and as part of the exploration of potential robot concepts for the system, the kinematic capabilities of concentric tube robots (CTRs) are first investigated. The complete set of stable trajectories that can be traced in follow-the-leader motion is discovered. A case study involving simulations and an experiment is then presented to showcase and verify the work. The research findings indicate that CTRs are not suitable for the fine-positioner. However, they show that CTRs with non-annular cross section can be used for the gross-positioner. In addition, the new trajectories discovered show promise in minimally invasive surgery (MIS). Soft robotic manipulators with fluidic actuation are then selected as the most suitable concept for the fine-positioner. The design of soft robotic manipulators with fluidic actuation is investigated from a general perspective. A general framework for the design of these devices is proposed, and a set of design principles are derived. These principles are first applied in a MIS case study to illustrate and verify the work. Finite element (FE) simulations are then reported to perform design optimisation, and thus complete the case study. The design study is then applied to determine the most suitable design for the fine-positioner. An additional analytical derivation is developed, followed by FE simulations, which extend those of the case study. Eventually, this work yields a final design of the fine-positioner. The final design found is different from existing ones, and is shown to provide an important performance improvement with respect to existing soft robots in terms of wrenches it can support. The control of soft and continuum robots relevant to the fine-positioner is also studied. The full kinematics of continuum robots with constant curvature bending and extending capabilities are first investigated, which correspond to a preliminary design concept conceived for the fine-positioner. Closed-form solutions are derived, closing an open problem. These kinematics, however, do not exactly match the final fine-positioner design selected. Thus, an alternative control approach based on closed-loop control laws is then adopted. For this, a mechanical model is first developed. Closed-loop control laws are then derived based on this mechanical model for planar operation of a segment of the fine-positioner. The control laws obtained represent the foundation for the subsequent development of control laws for a full fine-positioner operating in 3D. Furthermore, work on path planning for nonholonomic systems is also reported, and a new algorithm is presented, which can be applied for the insertion of the overall robotic system. Solutions to the other parts of the robotic system for on-wing operations are also reported. A gross-positioner consisting of a non-annular CTR is proposed. Solutions for a deployment mechanism are also presented. Potential feedback systems are outlined. In addition, methods for the fabrication of the systems are reported, and the electronics and systems required for the assembly of the different parts are described. Finally, the use of the robotic system to perform on-wing inspections in a representative case study is studied to determine the viability. Inspection strategies are shortlisted, and simulations and experiments are used to study them. The results, however, indicate that inspection is not viable since the signal to noise ratio is excessively low. Nonetheless, the robotic system proposed, and the research conducted, are still expected to be useful to perform a range of on-wing operations that require the insertion and deployment of a probe or other end-effector. In addition, the trajectories discovered for CTRs, the design found for the fine-positioner, and the advances on control, also have significant potential in MIS, where there is an important need for miniature robotic manipulators and similar devices.Open Acces

Implantable Microsystem Technologies For Nanoliter-Resolution Inner Ear Drug Delivery

Advances in protective and restorative biotherapies have created new opportunities to use site-directed, programmable drug delivery systems to treat auditory and vestibular disorders. Successful therapy development that leverages the transgenic, knock-in, and knock-out variants of mouse models of human disease requires advanced microsystems specifically designed to function with nanoliter precision and with system volumes suitable for implantation. The present work demonstrates a novel biocompatible, implantable, and scalable microsystem consisted of a thermal phase-change peristaltic micropump with wireless control and a refillable reservoir. The micropump is fabricated around a catheter microtubing (250 μm OD, 125 μm ID) that provided a biocompatible leak-free flow path while avoiding complicated microfluidic interconnects. Direct-write micro-scale printing technology was used to build the mechanical components of the pump around the microtubing directly on the back of a printed circuit board assembly. In vitro characterization results indicated nanoliter resolution control over the desired flow rates of 10–100 nL/min by changing the actuation frequency, with negligible deviations in presence of up to 10× greater than physiological backpressures and ±3°C ambient temperature variation. A biocompatibility study was performed to evaluate material suitability for chronic subcutaneous implantation and clinical translational development. A stand-alone, refillable, in-plane, scalable, and fully implantable microreservoir platform was designed and fabricated to be integrated with the micropump. The microreservoir consists two main components: a cavity for storing the drug and a septum for refilling. The cavity membrane is fabricated with thin Parylene-C layers, using a polyethylene glycol (PEG) sacrificial layer. The septum thickness is minimized by pre-compression down to 1 mm. The results of in vitro characterization indicated negligible restoring force for the optimized cavity membrane and thousands of punctures through the septum without leakage. The micropump and microreservoir were integrated into microsystems which were implanted in mice. The microtubing was implanted into the round window membrane niche for infusion of a known ototoxic compound (sodium salicylate) at 50 nL/min for 20 min. Real-time shifts in distortion product otoacoustic emission thresholds and amplitudes were measured during the infusion. The results match with syringe pump gold standard. For the first time a miniature and yet scalable microsystem for inner ear drug delivery was developed, enabling drug discovery opportunities and translation to human

Bio-Micro-Systems for Diagnostic Applications, Disease Prevention and Creating Tools for Biological Research

This thesis, divided into two parts, describes the development of 5 novel Bio-Micro-System devices. The term Bio-Micro-System has been used here to describe BioMEMS and 3D printed devices, with the dimensions of key components ranging from micrometers to a millimeter. Part A is focused on ‘Medical’ Micro-System devices that can potentially solve common medical problems. Part B is focused on ‘Biological’ Micro-System devices/tools for facilitating/enabling biological research. Specifically, Part A describes two implantable, electronics-free intraocular pressure (IOP) microsensors for the medical management of glaucoma: 1) Near Infrared Fluorescence-based Optomechanical (NiFO) technology - Consists of an implantable, pressure sensor that ‘optically encodes’ pressure in the near infrared (NIR) regime. A non-implantable, portable and compact optical head is used to excite the sensor and collect the emitted NIR light. The thesis discusses optimized device architecture and microfabrication approaches for best performance commercialization. 2) Displacement based Contrast Imaging (DCI) technology - A proof of concept, fluid pressure sensing scheme is shown to operate over a pressure range of 0–100 mbar (∼2 mbar resolution between 0–20 mbar,∼10 mbar resolution between 20–100 mbar), with a maximum error of <7% throughout its dynamic range. The thesis introduces the DCI technology and discusses its application as an IOP sensor. Moreover, Part A also describes a Touch-activated Sanitizer Dispensing (TSD) system for combating community acquired infections. The TSD can be mounted on any surface that is exposed to high human traffic and consists of an array of human-powered, miniaturized valves that deliver a small amount of disinfectant when touch actuated. The device disinfects the person’s hand that is touching it while being self-sterilized at the same time. The thesis describes the design and implementation of a proof of concept TSD that can disinfect an area equivalent to the size of a thumb. A significant (~ 10 fold) reduction in microbiological load is demonstrated on the fingertip and device surface within the first 24 hours. The size and footprint of the TSD can be scaled up as needed to improve hand hygiene compliance. In Part B, we developed a microfluidic chip for immobilizing Drosophila melanogaster larva by creating a cold micro-environment around the larva. After characterizing on chip temperature distribution and larval body movement, results indicate that the method is appropriate for repetitive and reversible, short-term (several minutes) immobilization. The method offers the added advantage of using the same chip to accommodate and immobilize larvae across all developmental stages (1st instar-late 3rd instar). Besides the demonstrated applications of the chip in high resolution observation of sub cellular events such as mitochondrial trafficking in neurons and neuro-synaptic growth, we envision the use of this method in a wide variety of biological imaging studies employing the Drosophila larval system, including cellular development and other studies. Finally, Part B also describes a 3D printed millifluidic device for CO2 immobilization of Caenorhabditis elegans populations. We developed a novel 3D printed device for immobilizing populations of Caenorhabditis elegans by creating a localized CO2 environment while the animals are maintained on the surface of agar. The results indicate that the method is easy to implement, is appropriate for short-term (20 minutes) immobilization and allows recovery within a few minutes. We envision its use in a wide variety of biological studies in Caenorhabditis elegan, including cellular development and neuronal regeneration studies.PHDBiomedical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/144050/1/amritarc_1.pd

High-resolution 3D printing enabled, minimally invasive fibre optic sensing and imaging probes

Minimally invasive surgical procedures have become more favourable to their traditional surgical counterparts due to their reduced risks, faster recovery times and decreased trauma. Despite this, there are still some limitations involved with these procedures, such as the spatial confinement of operating through small incisions and the intrinsic lack of visual or tactile feedback. Specialised tools and imaging equipment are required to overcome these issues. Providing better feedback to surgeons is a key area of research to enhance the outcomes and safety profiles of minimally invasive procedures. This thesis is centred on the development of new microfabrication methods to create novel fibre optic imaging and sensing probes that could ultimately be used for improving the guidance of minimally invasive surgeries. Several themes emerged in this process. The first theme involved the use and optimisation of high-resolution 3D injection of polymers as sacrificial layers onto which parylene-C was deposited. One outcome from this theme was a series of miniaturised parylene-C based membranes to create fibre optic pressure sensors for physiological pressure measurements and for ultrasound reception. The pressure sensor sensitivity was found to vary from 0.02 to 0.14 radians/mmHg, as the thickness of parylene was decreased from 2 to 0.5 μm. The ultrasound receivers were characterised and exhibited a noise equivalent pressure (NEP) value of ~100 Pa (an order of magnitude improvement compared to similarly sized piezoelectric hydrophones). A second theme employed high-resolution 3D printing to create microstructures of polydimethylsiloxane (PDMS) and subsequently formed nanocomposites, to create microscale acoustic hologram structures. This theme included the development of innovative manufacturing processes such as printing directly onto optical fibres, micro moulding and precise deposition which enabled the creation of such devices. These microstructures were investigated for reducing the divergence of photoacoustically-generated ultrasound beams. Taken together, the developments in this thesis pave the way for 3D microfabricated polymer-based fibre optic sensors that could find broad clinical utility in minimally invasive procedures

Dna studies: Latest spectroscopic and structural approaches

This review looks at the different approaches, techniques, and materials devoted to DNA studies. In the past few decades, DNA nanotechnology, micro-fabrication, imaging, and spectro-scopies have been tailored and combined for a broad range of medical-oriented applications. The continuous advancements in miniaturization of the devices, as well as the continuous need to study biological material structures and interactions, down to single molecules, have increase the interdisciplinarity of emerging technologies. In the following paragraphs, we will focus on recent sensing approaches, with a particular effort attributed to cutting-edge techniques for structural and mechanical studies of nucleic acids

Snake Robots for Surgical Applications: A Review

Although substantial advancements have been achieved in robot-assisted surgery, the blueprint to existing snake robotics predominantly focuses on the preliminary structural design, control, and human–robot interfaces, with features which have not been particularly explored in the literature. This paper aims to conduct a review of planning and operation concepts of hyper-redundant serpentine robots for surgical use, as well as any future challenges and solutions for better manipulation. Current researchers in the field of the manufacture and navigation of snake robots have faced issues, such as a low dexterity of the end-effectors around delicate organs, state estimation and the lack of depth perception on two-dimensional screens. A wide range of robots have been analysed, such as the i2Snake robot, inspiring the use of force and position feedback, visual servoing and augmented reality (AR). We present the types of actuation methods, robot kinematics, dynamics, sensing, and prospects of AR integration in snake robots, whilst addressing their shortcomings to facilitate the surgeon’s task. For a smoother gait control, validation and optimization algorithms such as deep learning databases are examined to mitigate redundancy in module linkage backlash and accidental self-collision. In essence, we aim to provide an outlook on robot configurations during motion by enhancing their material compositions within anatomical biocompatibility standards