Visual kinematic force estimation in robot-assisted surgery – application to knot tying

Robot-assisted surgery has potential advantages but lacks force feedback, which can lead to errors such as broken stitches or tissue damage. More experienced surgeons can judge the tool-tissue forces visually and an automated way of capturing this skill is desirable. Methods to measure force tend to involve complex measurement devices or visual tracking of tissue deformation. We investigate whether surgical forces can be estimated simply from the discrepancy between kinematic and visual measurement of the tool position. We show that combined visual and kinematic force estimation can be achieved without external measurements or modelling of tissue deformation. After initial alignment when no force is applied to the tool, visual and kinematic estimates of tool position diverge under force. We plot visual/kinematic displacement with force using vision and marker-based tracking. We demonstrate the ability to discern the forces involved in knot tying and visualize the displacement force using the publicly available JIGSAWS dataset as well as clinical examples of knot tying with the da Vinci surgical system. The ability to visualize or feel forces using this method may offer an advantage to those learning robotic surgery as well as adding to the information available to more experienced surgeons

Sensory substitution for force feedback recovery: A perception experimental study

Robotic-assisted surgeries are commonly used today as a more efficient alternative to traditional surgical options. Both surgeons and patients benefit from those systems, as they offer many advantages, including less trauma and blood loss, fewer complications, and better ergonomics. However, a remaining limitation of currently available surgical systems is the lack of force feedback due to the teleoperation setting, which prevents direct interaction with the patient. Once the force information is obtained by either a sensing device or indirectly through vision-based force estimation, a concern arises on how to transmit this information to the surgeon. An attractive alternative is sensory substitution, which allows transcoding information from one sensory modality to present it in a different sensory modality. In the current work, we used visual feedback to convey interaction forces to the surgeon. Our overarching goal was to address the following question: How should interaction forces be displayed to support efficient comprehension by the surgeon without interfering with the surgeon’s perception and workflow during surgery? Until now, the use the visual modality for force feedback has not been carefully evaluated. For this reason, we conducted an experimental study with two aims: (1) to demonstrate the potential benefits of using this modality and (2) to understand the surgeons’ perceptual preferences. The results derived from our study of 28 surgeons revealed a strong positive acceptance of the users (96%) using this modality. Moreover, we found that for surgeons to easily interpret the information, their mental model must be considered, meaning that the design of the visualizations should fit the perceptual and cognitive abilities of the end user. To our knowledge, this is the first time that these principles have been analyzed for exploring sensory substitution in medical robotics. Finally, we provide user-centered recommendations for the design of visual displays for robotic surgical systems.Peer ReviewedPostprint (author's final draft

The benefits of haptic feedback in robot assisted surgery and their moderators: a metaanalysis

Robot assisted surgery (RAS) provides medical practitioners with valuable tools, decreasing strain during surgery and leading to better patient outcomes. While the loss of haptic sensation is a commonly cited disadvantage of RAS, new systems aim to address this problem by providing artificial haptic feedback. N = 56 papers that compared robotic surgery systems with and without haptic feedback were analyzed to quantify the performance benefits of restoring the haptic modality. Additionally, this study identifies factors moderating the effect of restoring haptic sensation. Overall results showed haptic feedback was effective in reducing average forces (Hedges' g = 0.83) and peak forces (Hedges' g = 0.69) applied during surgery, as well as reducing the completion time (Hedges' g = 0.83). Haptic feedback has also been found to lead to higher accuracy (Hedges' g = 1.50) and success rates (Hedges' g = 0.80) during surgical tasks. Effect sizes on several measures varied between tasks, the type of provided feedback, and the subjects' levels of surgical expertise, with higher levels of expertise generally associated with smaller effect sizes. No significant differences were found between virtual fixtures and rendering contact forces. Implications for future research are discussed

Adverse Events in Robotic Surgery: A Retrospective Study of 14 Years of FDA Data

Understanding the causes and patient impacts of surgical adverse events will help improve systems and operational practices to avoid incidents in the future. We analyzed the adverse events data related to robotic systems and instruments used in minimally invasive surgery, reported to the U.S. FDA MAUDE database from January 2000 to December 2013. We determined the number of events reported per procedure and per surgical specialty, the most common types of device malfunctions and their impact on patients, and the causes for catastrophic events such as major complications, patient injuries, and deaths. During the study period, 144 deaths (1.4% of the 10,624 reports), 1,391 patient injuries (13.1%), and 8,061 device malfunctions (75.9%) were reported. The numbers of injury and death events per procedure have stayed relatively constant since 2007 (mean = 83.4, 95% CI, 74.2-92.7). Surgical specialties, for which robots are extensively used, such as gynecology and urology, had lower number of injuries, deaths, and conversions per procedure than more complex surgeries, such as cardiothoracic and head and neck (106.3 vs. 232.9, Risk Ratio = 2.2, 95% CI, 1.9-2.6). Device and instrument malfunctions, such as falling of burnt/broken pieces of instruments into the patient (14.7%), electrical arcing of instruments (10.5%), unintended operation of instruments (8.6%), system errors (5%), and video/imaging problems (2.6%), constituted a major part of the reports. Device malfunctions impacted patients in terms of injuries or procedure interruptions. In 1,104 (10.4%) of the events, the procedure was interrupted to restart the system (3.1%), to convert the procedure to non-robotic techniques (7.3%), or to reschedule it to a later time (2.5%). Adoption of advanced techniques in design and operation of robotic surgical systems may reduce these preventable incidents in the future.Comment: Presented as the J. Maxwell Chamberlain Memorial Paper for adult cardiac surgery at the 50th Annual Meeting of the Society of Thoracic Surgeons in January. See Appendix for more detailed results, discussions, and related work. Updated the header

Medical Robotics

The first generation of surgical robots are already being installed in a number of operating rooms around the world. Robotics is being introduced to medicine because it allows for unprecedented control and precision of surgical instruments in minimally invasive procedures. So far, robots have been used to position an endoscope, perform gallbladder surgery and correct gastroesophogeal reflux and heartburn. The ultimate goal of the robotic surgery field is to design a robot that can be used to perform closed-chest, beating-heart surgery. The use of robotics in surgery will expand over the next decades without any doubt. Minimally Invasive Surgery (MIS) is a revolutionary approach in surgery. In MIS, the operation is performed with instruments and viewing equipment inserted into the body through small incisions created by the surgeon, in contrast to open surgery with large incisions. This minimizes surgical trauma and damage to healthy tissue, resulting in shorter patient recovery time. The aim of this book is to provide an overview of the state-of-art, to present new ideas, original results and practical experiences in this expanding area. Nevertheless, many chapters in the book concern advanced research on this growing area. The book provides critical analysis of clinical trials, assessment of the benefits and risks of the application of these technologies. This book is certainly a small sample of the research activity on Medical Robotics going on around the globe as you read it, but it surely covers a good deal of what has been done in the field recently, and as such it works as a valuable source for researchers interested in the involved subjects, whether they are currently “medical roboticists” or not

Mental and physical workload in laparoscopic surgery

Imperial Users onl

Osseointegration for Amputees: Past, Present and Future: Basic Science, Innovations in Surgical Technique, Implant Design and Rehabilitation Strategies

Loss of a leg or arm is a tremendous disability. Immediate and obvious impairments are decreased mobility or diminished functional capacity. Not quite as obvious are the difficulties associated with activities of daily living, quality of life impairments, sometimes loss of independence or employment, and the mental health issues which often accompany limb loss. The interface between native tissue and the prosthetic limb presents the greatest challenge to amputee rehabilitation. Computer-controlled robotic limbs have been widely available since the 1990s. However, the weight of prosthetic limbs, coupled with the difficulty of where to locate the components, requires substantial loads to be transferred through the humanimplant interface. This interface has always been a skin-squeezing mechanism which results in repetitive soft-tissue loading and trauma, in both compression and shear, which inevitably causes multiple problems (pain, skin breakdown and infection, hyperhidrosis, allergic reaction to the material) leading to periodic or prolonged prosthesis disuse. So unfortunately, despite all the effort and expense invested in the prosthetic limb itself, patients often were unable to benefit. Percutaneous EndoProsthetic Osseointegration for Limbs (PEPOL) is a revolutionary technique that involves anchoring a metal implant directly to a patient’s skeleton, then permanently passed through the patient’s skin, and attached to a prosthetic limb. By doing this, the weight of the prosthesis is borne by the patient’s skeleton and is directly powered by muscles, leading to a lighter and more native experience. The skin is no longer compressed and traumatised, eliminating the aforementioned issues. Since learning about this technology in the mid-2000s and performing my first independent procedure in 2009, I have investigated and pioneered the world’s leading surgical techniques and rehabilitative methods for PEPOL. Treating nearly 1000 amputees via the Osseointegration Group of Australia and the MQ Health Limb Reconstruction Centre at Macquarie University has allowed research to be performed on this technology, documented, and discussed in the 2 Body of Work. Patients almost always improve their objective and assessed mobility performance (Overall 38.6% distance improvement on the 6MWT), they wear their prosthetic limb more (Overall 38.1% increase in the Q-TFA Prosthetic Use Score), and they are subjectively more satisfied with their condition as an amputee (Overall 41.1% increase in the Q-TFA Global Score) . While these benefits are consistent, my research has also identified the fortunately limited problems with infection and soft tissue management (29% of all patients required re-operations due to direct or indirect complications). PEPOL clearly provides excellent improvement for the vast majority of patients, and the continued investigation of this technology should lead to even greater improvements in progressing from what is already successful, make it more readily available, and ameliorate its existing challenges

Data-driven resiliency assessment of medical cyber-physical systems

Advances in computing, networking, and sensing technologies have resulted in the ubiquitous deployment of medical cyber-physical systems in various clinical and personalized settings. The increasing complexity and connectivity of such systems, the tight coupling between their cyber and physical components, and the inevitable involvement of human operators in supervision and control have introduced major challenges in ensuring system reliability, safety, and security. This dissertation takes a data-driven approach to resiliency assessment of medical cyber-physical systems. Driven by large-scale studies of real safety incidents involving medical devices, we develop techniques and tools for (i) deeper understanding of incident causes and measurement of their impacts, (ii) validation of system safety mechanisms in the presence of realistic hazard scenarios, and (iii) preemptive real-time detection of safety hazards to mitigate adverse impacts on patients. We present a framework for automated analysis of structured and unstructured data from public FDA databases on medical device recalls and adverse events. This framework allows characterization of the safety issues originated from computer failures in terms of fault classes, failure modes, and recovery actions. We develop an approach for constructing ontology models that enable automated extraction of safety-related features from unstructured text. The proposed ontology model is defined based on device-specific human-in-the-loop control structures in order to facilitate the systems-theoretic causality analysis of adverse events. Our large-scale analysis of FDA data shows that medical devices are often recalled because of failure to identify all potential safety hazards, use of safety mechanisms that have not been rigorously validated, and limited capability in real-time detection and automated mitigation of hazards. To address those problems, we develop a safety hazard injection framework for experimental validation of safety mechanisms in the presence of accidental failures and malicious attacks. To reduce the test space for safety validation, this framework uses systems-theoretic accident causality models in order to identify the critical locations within the system to target software fault injection. For mitigation of safety hazards at run time, we present a model-based analysis framework that estimates the consequences of control commands sent from the software to the physical system through real-time computation of the system’s dynamics, and preemptively detects if a command is unsafe before its adverse consequences manifest in the physical system. The proposed techniques are evaluated on a real-world cyber-physical system for robot-assisted minimally invasive surgery and are shown to be more effective than existing methods in identifying system vulnerabilities and deficiencies in safety mechanisms as well as in preemptive detection of safety hazards caused by malicious attacks

Augmentation Of Human Skill In Microsurgery

Surgeons performing highly skilled microsurgery tasks can benefit from information and manual assistance to overcome technological and physiological limitations to make surgery safer, efficient, and more successful. Vitreoretinal surgery is particularly difficult due to inherent micro-scale and fragility of human eye anatomy. Additionally, surgeons are challenged by physiological hand tremor, poor visualization, lack of force sensing, and significant cognitive load while executing high-risk procedures inside the eye, such as epiretinal membrane peeling. This dissertation presents the architecture and the design principles for a surgical augmentation environment which is used to develop innovative functionality to address the fundamental limitations in vitreoretinal surgery. It is an inherently information driven modular system incorporating robotics, sensors, and multimedia components. The integrated nature of the system is leveraged to create intuitive and relevant human-machine interfaces and generate a particular system behavior to provide active physical assistance and present relevant sensory information to the surgeon. These include basic manipulation assistance, audio-visual and haptic feedback, intraoperative imaging and force sensing. The resulting functionality, and the proposed architecture and design methods generalize to other microsurgical procedures. The system's performance is demonstrated and evaluated using phantoms and in vivo experiments