A gaze-contingent framework for perceptually-enabled applications in healthcare

Patient safety and quality of care remain the focus of the smart operating room of the future. Some of the most influential factors with a detrimental effect are related to suboptimal communication among the staff, poor flow of information, staff workload and fatigue, ergonomics and sterility in the operating room. While technological developments constantly transform the operating room layout and the interaction between surgical staff and machinery, a vast array of opportunities arise for the design of systems and approaches, that can enhance patient safety and improve workflow and efficiency. The aim of this research is to develop a real-time gaze-contingent framework towards a "smart" operating suite, that will enhance operator's ergonomics by allowing perceptually-enabled, touchless and natural interaction with the environment. The main feature of the proposed framework is the ability to acquire and utilise the plethora of information provided by the human visual system to allow touchless interaction with medical devices in the operating room. In this thesis, a gaze-guided robotic scrub nurse, a gaze-controlled robotised flexible endoscope and a gaze-guided assistive robotic system are proposed. Firstly, the gaze-guided robotic scrub nurse is presented; surgical teams performed a simulated surgical task with the assistance of a robot scrub nurse, which complements the human scrub nurse in delivery of surgical instruments, following gaze selection by the surgeon. Then, the gaze-controlled robotised flexible endoscope is introduced; experienced endoscopists and novice users performed a simulated examination of the upper gastrointestinal tract using predominately their natural gaze. Finally, a gaze-guided assistive robotic system is presented, which aims to facilitate activities of daily living. The results of this work provide valuable insights into the feasibility of integrating the developed gaze-contingent framework into clinical practice without significant workflow disruptions.Open Acces

A flexible access platform for robot-assisted minimally invasive surgery

Advances in Minimally Invasive Surgery (MIS) are driven by the clinical demand to reduce the invasiveness of surgical procedures so patients undergo less trauma and experience faster recoveries. These well documented benefits of MIS have been achieved through parallel advances in the technology and instrumentation used during procedures. The new and evolving field of Flexible Access Surgery (FAS), where surgeons access the operative site through a single incision or a natural orifice incision, is being promoted as the next potential step in the evolution of surgery. In order to achieve similar levels of success and adoption as MIS, technology again has its role to play in developing new instruments to solve the unmet clinical challenges of FAS. As procedures become less invasive, these instruments should not just address the challenges presented by the complex access routes of FAS, but should also build on the recent advances in pre- and intraoperative imaging techniques to provide surgeons with new diagnostic and interventional decision making capabilities. The main focus of this thesis is the development and applications of a flexible robotic device that is capable of providing controlled flexibility along curved pathways inside the body. The principal component of the device is its modular mechatronic joint design which utilises an embedded micromotor-tendon actuation scheme to provide independently addressable degrees of freedom and three internal working channels. Connecting multiple modules together allows a seven degree-of-freedom (DoF) flexible access platform to be constructed. The platform is intended for use as a research test-bed to explore engineering and surgical challenges of FAS. Navigation of the platform is realised using a handheld controller optimised for functionality and ergonomics, or in a "hands-free" manner via a gaze contingent control framework. Under this framework, the operator's gaze fixation point is used as feedback to close the servo control loop. The feasibility and potential of integrating multi-spectral imaging capabilities into flexible robotic devices is also demonstrated. A force adaptive servoing mechanism is developed to simplify the deployment, and improve the consistency of probe-based optical imaging techniques by automatically controlling the contact force between the probe tip and target tissue. The thesis concludes with the description of two FAS case studies performed with the platform during in-vivo porcine experiments. These studies demonstrate the ability of the platform to perform large area explorations within the peritoneal cavity and to provide a stable base for the deployment of interventional instruments and imaging probes

Mobile robot teleoperation through eye-gaze (telegaze)

In most teleoperation applications the human operator is required to monitor the status of the robot, as well as, issue controlling commands for the whole duration of the operation. Using a vision based feedback system, monitoring the robot requires the operator to look at a continuous stream of images displayed on an interaction screen. The eyes of the operator therefore, are fully engaged in monitoring and the hands in controlling. Since the eyes of the operator are engaged in monitoring anyway, inputs from their gaze can be used to aid in controlling. This frees the hands of the operator, either partially or fully, from controlling which can then be used to perform any other necessary tasks. However, the challenge here lies in distinguishing between the inputs that can be used for controlling and the inputs that can be used for monitoring. In mobile robot teleoperation, controlling is mainly composed of issuing locomotion commands to drive the robot. Monitoring on the other hand, is looking where the robot goes and looking for any obstacles in the route. Interestingly, there exist a strong correlation between human's gazing behaviours and their moving intentions. This correlation has been exploited in this thesis to investigate novel means for mobile robot teleoperation through eye-gaze, which has been named TeleGaze for short

Investigating Ultrasound-Guided Autonomous Assistance during Robotic Minimally Invasive Surgery

Despite it being over twenty years since the first introduction of robotic surgical systems in common surgical practice, they are still far from widespread across all healthcare systems, surgical disciplines and procedures. At the same time, the systems that are used act as mere tele-manipulators with motion scaling and have yet to make use of the immense potential of their sensory data in providing autonomous assistance during surgery or perform tasks themselves in a semi-autonomous fashion. Equivalently, the potential of using intracorporeal imaging, particularly Ultrasound (US) during surgery for improved tumour localisation remains largely unused. Aside from the cost factors, this also has to do with the necessity of adequate training for scan interpretation and the difficulty of handling an US probe near the surgical sight. Additionally, the potential for automation that is being explored in extracorporeal US using serial manipulators does not yet translate into ultrasound-enabled autonomous assistance in a surgical robotic setting. Motivated by this research gap, this work explores means to enable autonomous intracorporeal ultrasound in a surgical robotic setting. Based around the the da Vinci Research Kit (dVRK), it first develops a surgical robotics platform that allows for precise evaluation of the robot’s performance using Infrared (IR) tracking technology. Based on this initial work, it then explores the possibility to provide autonomous ultrasound guidance during surgery. Therefore, it develops and assesses means to improve kinematic accuracy despite manipulator backlash as well as enabling adequate probe position with respect to the tissue surface and anatomy. Founded on the acquired anatomical information, this thesis explores the integration of a second robotic arm and its usage for autonomous assistance. Starting with an autonomously acquired tumor scan, the setup is extended and methods devised to enable the autonomous marking of margined tumor boundaries on the tissue surface both in a phantom as well as in an ex-vivo experiment on porcine liver. Moving towards increased autonomy, a novel minimally invasive High Intensity Focused Ultrasound (HIFUS) transducer is integrated into the robotic setup including a sensorised, water-filled membrane for sensing interaction forces with the tissue surface. For this purpose an extensive material characterisation is caried out, exploring different surface material pairings. Finally, the proposed system, including trajectory planning and a hybrid-force position control scheme are evaluated in a benchtop ultrasound phantom trial

Appearance Modelling and Reconstruction for Navigation in Minimally Invasive Surgery

Minimally invasive surgery is playing an increasingly important role for patient care. Whilst its direct patient benefit in terms of reduced trauma, improved recovery and shortened hospitalisation has been well established, there is a sustained need for improved training of the existing procedures and the development of new smart instruments to tackle the issue of visualisation, ergonomic control, haptic and tactile feedback. For endoscopic intervention, the small field of view in the presence of a complex anatomy can easily introduce disorientation to the operator as the tortuous access pathway is not always easy to predict and control with standard endoscopes. Effective training through simulation devices, based on either virtual reality or mixed-reality simulators, can help to improve the spatial awareness, consistency and safety of these procedures. This thesis examines the use of endoscopic videos for both simulation and navigation purposes. More specifically, it addresses the challenging problem of how to build high-fidelity subject-specific simulation environments for improved training and skills assessment. Issues related to mesh parameterisation and texture blending are investigated. With the maturity of computer vision in terms of both 3D shape reconstruction and localisation and mapping, vision-based techniques have enjoyed significant interest in recent years for surgical navigation. The thesis also tackles the problem of how to use vision-based techniques for providing a detailed 3D map and dynamically expanded field of view to improve spatial awareness and avoid operator disorientation. The key advantage of this approach is that it does not require additional hardware, and thus introduces minimal interference to the existing surgical workflow. The derived 3D map can be effectively integrated with pre-operative data, allowing both global and local 3D navigation by taking into account tissue structural and appearance changes. Both simulation and laboratory-based experiments are conducted throughout this research to assess the practical value of the method proposed

Smart Camera Robotic Assistant for Laparoscopic Surgery

The cognitive architecture also includes learning mechanisms to adapt the behavior of the robot to the different ways of working of surgeons, and to improve the robot behavior through experience, in a similar way as a human assistant would do. The theoretical concepts of this dissertation have been validated both through in-vitro experimentation in the labs of medical robotics of the University of Malaga and through in-vivo experimentation with pigs in the IACE Center (Instituto Andaluz de Cirugía Experimental), performed by expert surgeons.In the last decades, laparoscopic surgery has become a daily practice in operating rooms worldwide, which evolution is tending towards less invasive techniques. In this scenario, robotics has found a wide field of application, from slave robotic systems that replicate the movements of the surgeon to autonomous robots able to assist the surgeon in certain maneuvers or to perform autonomous surgical tasks. However, these systems require the direct supervision of the surgeon, and its capacity of making decisions and adapting to dynamic environments is very limited. This PhD dissertation presents the design and implementation of a smart camera robotic assistant to collaborate with the surgeon in a real surgical environment. First, it presents the design of a novel camera robotic assistant able to augment the capacities of current vision systems. This robotic assistant is based on an intra-abdominal camera robot, which is completely inserted into the patient’s abdomen and it can be freely moved along the abdominal cavity by means of magnetic interaction with an external magnet. To provide the camera with the autonomy of motion, the external magnet is coupled to the end effector of a robotic arm, which controls the shift of the camera robot along the abdominal wall. This way, the robotic assistant proposed in this dissertation has six degrees of freedom, which allow providing a wider field of view compared to the traditional vision systems, and also to have different perspectives of the operating area. On the other hand, the intelligence of the system is based on a cognitive architecture specially designed for autonomous collaboration with the surgeon in real surgical environments. The proposed architecture simulates the behavior of a human assistant, with a natural and intuitive human-robot interface for the communication between the robot and the surgeon

Cable-driven parallel mechanisms for minimally invasive robotic surgery

Minimally invasive surgery (MIS) has revolutionised surgery by providing faster recovery times, less post-operative complications, improved cosmesis and reduced pain for the patient. Surgical robotics are used to further decrease the invasiveness of procedures, by using yet smaller and fewer incisions or using natural orifices as entry point. However, many robotic systems still suffer from technical challenges such as sufficient instrument dexterity and payloads, leading to limited adoption in clinical practice. Cable-driven parallel mechanisms (CDPMs) have unique properties, which can be used to overcome existing challenges in surgical robotics. These beneficial properties include high end-effector payloads, efficient force transmission and a large configurable instrument workspace. However, the use of CDPMs in MIS is largely unexplored. This research presents the first structured exploration of CDPMs for MIS and demonstrates the potential of this type of mechanism through the development of multiple prototypes: the ESD CYCLOPS, CDAQS, SIMPLE, neuroCYCLOPS and microCYCLOPS. One key challenge for MIS is the access method used to introduce CDPMs into the body. Three different access methods are presented by the prototypes. By focusing on the minimally invasive access method in which CDPMs are introduced into the body, the thesis provides a framework, which can be used by researchers, engineers and clinicians to identify future opportunities of CDPMs in MIS. Additionally, through user studies and pre-clinical studies, these prototypes demonstrate that this type of mechanism has several key advantages for surgical applications in which haptic feedback, safe automation or a high payload are required. These advantages, combined with the different access methods, demonstrate that CDPMs can have a key role in the advancement of MIS technology.Open Acces

Robotic surgery, human fallibility, and the politics of care

Robotic Surgery, Human Fallibility, and the Politics of Care leverages the methods and theoretical paradigms of performance, visual, and new media studies to explore the contradictions, aspirations, and failures of modern technologized medicine. In particular, I consider the use of robots in the operating rooms of a large research hospital. University Hospital illuminates a contemporary articulation of human bodies and robotic technology that focuses and amplifies existing and emergent tensions and contradictions in modern medicine's investment in providing both care and cure. Intuitive Surgical, Inc.'s da Vinci Surgical System provides a platform for this exploration, both as a concrete, material, and particular assemblage of hardware, software and human wetware, and as a technology that offers a specific and perhaps more productive vantage point--a modest step stool--for understanding the contemporary politics of surgical pedagogy and practice. I locate the dVSS in a broader context of ambivalence that surgeons experience with regard to the manual practices of their craft, an ambivalence amplified by the increasing sophistication and automation of surgical tools and the changing ontologies of surgical practice. The surgical interface of the dVSS prosthetically enhances--as well as displaces and replaces--embodied surgical skill. At a time when all facets of medical care grapple with the problem of medical error, I outline an emergent sensibility of machinic virtuosity, articulated to both human and robotic surgical practice alike, geared toward addressing and overcoming the perceived pitfalls of human fallibility. Rather than simply enacting a technological dehumanization of medicine, robotic surgery suggests a more complicated terrain where the nature of the human and the machine bleed into each other. What I term the becoming machine of the surgeon and the becoming surgeon of the medical device occurs on the cutting edge of the robot-surgeon interface. The implications of this emergent medical sensibility are far from clear or unilateral. In closing, I reflect on the uncertain impact of the ideal of machinic virtuosity on the politics of care. This reflection considers software and machine ethics alongside medicine's aspiration to manage contingency according to the procedurality of medical and surgical protocols

Motor learning induced neuroplasticity in minimally invasive surgery

Technical skills in surgery have become more complex and challenging to acquire since the introduction of technological aids, particularly in the arena of Minimally Invasive Surgery. Additional challenges posed by reforms to surgical careers and increased public scrutiny, have propelled identification of methods to assess and acquire MIS technical skills. Although validated objective assessments have been developed to assess motor skills requisite for MIS, they poorly understand the development of expertise. Motor skills learning, is indirectly observable, an internal process leading to relative permanent changes in the central nervous system. Advances in functional neuroimaging permit direct interrogation of evolving patterns of brain function associated with motor learning due to the property of neuroplasticity and has been used on surgeons to identify the neural correlates for technical skills acquisition and the impact of new technology. However significant gaps exist in understanding neuroplasticity underlying learning complex bimanual MIS skills. In this thesis the available evidence on applying functional neuroimaging towards assessment and enhancing operative performance in the field of surgery has been synthesized. The purpose of this thesis was to evaluate frontal lobe neuroplasticity associated with learning a complex bimanual MIS skill using functional near-infrared spectroscopy an indirect neuroimaging technique. Laparoscopic suturing and knot-tying a technically challenging bimanual skill is selected to demonstrate learning related reorganisation of cortical behaviour within the frontal lobe by shifts in activation from the prefrontal cortex (PFC) subserving attention to primary and secondary motor centres (premotor cortex, supplementary motor area and primary motor cortex) in which motor sequences are encoded and executed. In the cross-sectional study, participants of varying expertise demonstrate frontal lobe neuroplasticity commensurate with motor learning. The longitudinal study involves tracking evolution in cortical behaviour of novices in response to receipt of eight hours distributed training over a fortnight. Despite novices achieving expert like performance and stabilisation on the technical task, this study demonstrates that novices displayed persistent PFC activity. This study establishes for complex bimanual tasks, that improvements in technical performance do not accompany a reduced reliance in attention to support performance. Finally, least-squares support vector machine is used to classify expertise based on frontal lobe functional connectivity. Findings of this thesis demonstrate the value of interrogating cortical behaviour towards assessing MIS skills development and credentialing.Open Acces

Doctor of Philosophy

dissertationUsing eye-tracking technology to capture the visual scanpaths of a sample of laypersons (N = 92), the current study employed a 2 (training condition: ABCDE vs. Ugly Duckling Sign) Ã- 2 (visual condition: photorealistic images vs. illustrations) factorial design to assess whether SSE training succeeds or fails in facilitating increases in sensitivity and specificity. Self-efficacy and perceived importance were tested as moderators, and eye-tracking fixation metrics as mediators, within the framework of Visual Skill Acquisition Theory (VSAT). For sensitivity, results indicated a significant main effect for visual condition, F(1,88) = 7.102, p = .009, wherein illustrations (M = .524, SD = .197) resulted in greater sensitivity than photos (M = .425, SD = .159, d = .55). For specificity, the main effect for training was not significant, F(1,88) = 2.120, p = .149; however, results indicated a significant main effect for visual condition, F(1,88) = 4.079, p = .046, wherein photos (M = .821, SD = .108) resulted in greater specificity than illustrations (M = .770, SD = .137, d = .41). The interaction for training Ã- visual condition, F(1,88) = 3.554, p = .063, was significant within a 90% confidence interval, such that those within the UDS Photo condition displayed greater specificity than all other combinations of training and visual condition. No significant moderated mediation manifested for sensitivity, but for specificity, the model was significant, r = .59, R2 = .34, F(9,82) = 4.7783, p =.001, with Percent of Time in Lookzone serving as a significant mediator, and both self-efficacy and visual condition significantly moderating the mediation. For those in the photo condition with very high self-efficacy, UDS increased specificity directly. For those in the photo condition with self-efficacy levels at the mean or lower, there was a conditional indirect effect through Percent of Time in Lookzoneâ€"which is to say that these individuals spent a larger amount of their viewing time on target (observing the atypical nevi)â€"and time on target is positively related to specificity. Findings suggest that existing SSE training techniques may be enhanced by maximizing visual processing efficiency