A gaze-contingent framework for perceptually-enabled applications in healthcare

Patient safety and quality of care remain the focus of the smart operating room of the future. Some of the most influential factors with a detrimental effect are related to suboptimal communication among the staff, poor flow of information, staff workload and fatigue, ergonomics and sterility in the operating room. While technological developments constantly transform the operating room layout and the interaction between surgical staff and machinery, a vast array of opportunities arise for the design of systems and approaches, that can enhance patient safety and improve workflow and efficiency. The aim of this research is to develop a real-time gaze-contingent framework towards a "smart" operating suite, that will enhance operator's ergonomics by allowing perceptually-enabled, touchless and natural interaction with the environment. The main feature of the proposed framework is the ability to acquire and utilise the plethora of information provided by the human visual system to allow touchless interaction with medical devices in the operating room. In this thesis, a gaze-guided robotic scrub nurse, a gaze-controlled robotised flexible endoscope and a gaze-guided assistive robotic system are proposed. Firstly, the gaze-guided robotic scrub nurse is presented; surgical teams performed a simulated surgical task with the assistance of a robot scrub nurse, which complements the human scrub nurse in delivery of surgical instruments, following gaze selection by the surgeon. Then, the gaze-controlled robotised flexible endoscope is introduced; experienced endoscopists and novice users performed a simulated examination of the upper gastrointestinal tract using predominately their natural gaze. Finally, a gaze-guided assistive robotic system is presented, which aims to facilitate activities of daily living. The results of this work provide valuable insights into the feasibility of integrating the developed gaze-contingent framework into clinical practice without significant workflow disruptions.Open Acces

A Cognitive Robot Control Architecture for Autonomous Execution of Surgical Tasks

The research on medical robotics is starting to address the autonomous execution of surgical tasks, without effective intervention of humans apart from supervision and task configuration. This paper addresses the complete automation of a surgical robot by combining advanced sensing, cognition and control capabilities, developed according to rigorous assessment of surgical require- ments, formal specification of robotic system behavior and software design and implementation based on solid tools and frame- works. In particular, the paper focuses on the cognitive control architecture and its development process, based on formal modeling and verification methods as best practices to ensure safe and reliable behavior. Full implementation of the proposed architecture has been tested on an experimental setup including a novel robot specifically designed for surgical applications, but adaptable to different selected tasks (i.e. needle insertion, wound suturing)

Augmentation Of Human Skill In Microsurgery

Surgeons performing highly skilled microsurgery tasks can benefit from information and manual assistance to overcome technological and physiological limitations to make surgery safer, efficient, and more successful. Vitreoretinal surgery is particularly difficult due to inherent micro-scale and fragility of human eye anatomy. Additionally, surgeons are challenged by physiological hand tremor, poor visualization, lack of force sensing, and significant cognitive load while executing high-risk procedures inside the eye, such as epiretinal membrane peeling. This dissertation presents the architecture and the design principles for a surgical augmentation environment which is used to develop innovative functionality to address the fundamental limitations in vitreoretinal surgery. It is an inherently information driven modular system incorporating robotics, sensors, and multimedia components. The integrated nature of the system is leveraged to create intuitive and relevant human-machine interfaces and generate a particular system behavior to provide active physical assistance and present relevant sensory information to the surgeon. These include basic manipulation assistance, audio-visual and haptic feedback, intraoperative imaging and force sensing. The resulting functionality, and the proposed architecture and design methods generalize to other microsurgical procedures. The system's performance is demonstrated and evaluated using phantoms and in vivo experiments

Control and Estimation Methods Towards Safe Robot-assisted Eye Surgery

Vitreoretinal surgery is among the most delicate surgical tasks in which physiological hand tremor may severely diminish surgeon performance and put the eye at high risk of injury. Unerring targeting accuracy is required to perform precise operations on micro-scale tissues. Tool tip to tissue interaction forces are usually below human tactile perception, which may result in exertion of excessive forces to the retinal tissue leading to irreversible damages. Notable challenges during retinal surgery lend themselves to robotic assistance which has proven beneficial in providing a safe steady-hand manipulation. Efficient assistance from the robots heavily relies on accurate sensing and intelligent control algorithms of important surgery states and situations (e.g. instrument tip position measurements and control of interaction forces). This dissertation provides novel control and state estimation methods to improve safety during robot-assisted eye surgery. The integration of robotics into retinal microsurgery leads to a reduction in surgeon perception of tool-to-tissue forces at sclera. This blunting of human tactile sensory input, which is due to the inflexible inertia of the robot, is a potential iatrogenic risk during robotic eye surgery. To address this issue, a sensorized surgical instrument equipped with Fiber Bragg Grating (FBG) sensors, which is capable of measuring the sclera forces and instrument insertion depth into the eye, is integrated to the Steady-Hand Eye Robot (SHER). An adaptive control scheme is then customized and implemented on the robot that is intended to autonomously mitigate the risk of unsafe scleral forces and excessive insertion of the instrument. Various preliminary and multi-user clinician studies are then conducted to evaluate the effectiveness of the control method during mock retinal surgery procedures. In addition, due to inherent flexibility and the resulting deflection of eye surgical instruments as well as the need for targeting accuracy, we have developed a method to enhance deflected instrument tip position estimation. Using an iterative method and microscope data, we develop a calibration- and registration-independent (RI) framework to provide online estimates of the instrument stiffness (least squares and adaptive). The estimations are then combined with a state-space model for tip position evolution obtained based on the forward kinematics (FWK) of the robot and FBG sensor measurements. This is accomplished using a Kalman Filtering (KF) approach to improve the instrument tip position estimation during robotic surgery. The entire framework is independent of camera-to-robot coordinate frame registration and is evaluated during various phantom experiments to demonstrate its effectiveness

A Sensorized Instrument for Minimally Invasive Surgery for the Measurement of Forces during Training and Surgery: Development and Applications

The reduced access conditions present in Minimally Invasive Surgery (MIS) affect the feel of interaction forces between the instruments and the tissue being treated. This loss of haptic information compromises the safety of the procedure and must be overcome through training. Haptics in MIS is the subject of extensive research, focused on establishing force feedback mechanisms and developing appropriate sensors. This latter task is complicated by the need to place the sensors as close as possible to the instrument tip, as the measurement of forces outside of the patient\u27s body does not represent the true tool--tissue interaction. Many force sensors have been proposed, but none are yet available for surgery. The objectives of this thesis were to develop a set of instruments capable of measuring tool--tissue force information in MIS, and to evaluate the usefulness of force information during surgery and for training and skills assessment. To address these objectives, a set of laparoscopic instruments was developed that can measure instrument position and tool--tissue interaction forces in multiple degrees of freedom. Different design iterations and the work performed towards the development of a sterilizable instrument are presented. Several experiments were performed using these instruments to establish the usefulness of force information in surgery and training. The results showed that the combination of force and position information can be used in the development of realistic tissue models or haptic interfaces specifically designed for MIS. This information is also valuable in order to create tactile maps to assist in the identification of areas of different stiffness. The real-time measurement of forces allows visual force feedback to be presented to the surgeon. When applied to training scenarios, the results show that experience level correlates better with force-based metrics than those currently used in training simulators. The proposed metrics can be automatically computed, are completely objective, and measure important aspects of performance. The primary contribution of this thesis is the design and development of highly versatile instruments capable of measuring force and position during surgery. A second contribution establishes the importance and usefulness of force data during skills assessment, training and surgery

Optical Microsystems for Static and Dynamic Tactile Sensing: Design, Modeling, Fabrication and Testing

Minimally invasive surgical operations encompass various surgical tasks ranging from conventional endoscopic/laparoscopic methods to recent sophisticated minimally invasive surgical techniques. In such sophisticated techniques, surgeons use equipment varying from robotic-assisted surgical platforms for abdominal surgery to computer-controlled catheters for catheter-based cardiovascular surgery. Presently, the countless advantages that minimally invasive surgery offers for both patients and surgeons have made the use of such surgical operations routine and reliable. However, in such operations, unlike conventional surgical operations, surgeons still suffer from the lack of tactile perception while interacting with the biological tissues using surgical instruments. To address this issue, it is necessary to develop a tactile sensor that can mimic the fingertip tactile perceptions of surgeons. In doing so and to satisfy the needs of surgeons, a number of considerations should be implemented in the design of the tactile sensors. First, the sensor should be magnetic resonance compatible to perform measurements even in the presence of magnetic resonance imaging (MRI) devices. Currently, such devices are in wide-spread use in surgical operation rooms. Second, the sensor should be electrically-passive because introducing electrical current into the patients’ body is not desirable in various surgical operations such as cardiovascular operations. Third, the sensor should perform measurements under both static and dynamic loading conditions during the sensor-tissue interactions. Such a capability of the sensor ensures that surgeons receive tactile feedback even when there is continuous static contact between surgical tools and tissues. Essentially, surgeons need such feedback to make surgical tasks safer. In addition, the size of the sensor should be miniaturized to address the size restrictions. In fact, the combination of intensity-based optical fiber sensing principles and micro-systems technology is one of the limited choices that address all the required considerations to develop such tactile sensors in a variety of ways. The present thesis deals with the design, modeling, manufacturing, testing, and characterizing of different tactile sensor configurations based on detection and integration methods. The various stages of design progress and principles are developed into different design configurations and presented in different chapters. The main sensing principle applied is based on the intensity modulation principle of optical fibers using micro-systems technology. In addition, a hybrid sensing principle is also studied by integrating both optical and non-optical detection methods. The micromachined sensors are categorized into five different generations. Each generation has advantages by comparison with its counterpart from the previous generation. The initial development of micromachined sensors is based on optical fiber coupling loss. In the second phase, a hybrid optical-piezoresistive sensing principle is studied. The success of these phases was instrumental in realizing a micromachined sensor that has the advantage of being fully optical. This sensor measures the magnitude of concentrated and distributed force, the position of a concentrated force, the variations in the force distribution along its length, the relative hardness of soft contact objects, and the local discontinuities in the hardness of the contact objects along the length of the contact area. Unlike most electrical-based commercially-available sensors, it performs all of these measurements under both static and dynamic loading conditions. Moreover, it is electrically passive and potentially MRI-compatible. The performances of the sensors were experimentally characterized for specific conditions presented in this thesis. However, these performances are easily tunable and adjustable depending upon the requirements of specific surgical tasks. Although the sensors were initially designed for surgical applications, they can have numerous other applications in the areas of robotics, automation, tele-display, and material testing

NASA Tech Briefs, December 1991

Topics include: Electronic Components and Circuits. Electronic Systems, Physical Sciences, Materials, Computer Programs, Mechanics, Machinery, Fabrication Technology, Mathematics and Information Sciences