A gaze-contingent framework for perceptually-enabled applications in healthcare

Patient safety and quality of care remain the focus of the smart operating room of the future. Some of the most influential factors with a detrimental effect are related to suboptimal communication among the staff, poor flow of information, staff workload and fatigue, ergonomics and sterility in the operating room. While technological developments constantly transform the operating room layout and the interaction between surgical staff and machinery, a vast array of opportunities arise for the design of systems and approaches, that can enhance patient safety and improve workflow and efficiency. The aim of this research is to develop a real-time gaze-contingent framework towards a "smart" operating suite, that will enhance operator's ergonomics by allowing perceptually-enabled, touchless and natural interaction with the environment. The main feature of the proposed framework is the ability to acquire and utilise the plethora of information provided by the human visual system to allow touchless interaction with medical devices in the operating room. In this thesis, a gaze-guided robotic scrub nurse, a gaze-controlled robotised flexible endoscope and a gaze-guided assistive robotic system are proposed. Firstly, the gaze-guided robotic scrub nurse is presented; surgical teams performed a simulated surgical task with the assistance of a robot scrub nurse, which complements the human scrub nurse in delivery of surgical instruments, following gaze selection by the surgeon. Then, the gaze-controlled robotised flexible endoscope is introduced; experienced endoscopists and novice users performed a simulated examination of the upper gastrointestinal tract using predominately their natural gaze. Finally, a gaze-guided assistive robotic system is presented, which aims to facilitate activities of daily living. The results of this work provide valuable insights into the feasibility of integrating the developed gaze-contingent framework into clinical practice without significant workflow disruptions.Open Acces

TRAINING AND ASSESSMENT OF HAND-EYE COORDINATION WITH ELECTROENCEPHALOGRAPHY

Ph.DDOCTOR OF PHILOSOPH

Vitreo-retinal eye surgery robot : sustainable precision

Vitreo-retinal eye surgery encompasses the surgical procedures performed on the vitreous humor and the retina. A procedure typically consists of the removal of the vitreous humor, the peeling of a membrane and/or the repair of a retinal detachment. Vitreo-retinal surgery is performed minimal invasively. Small needle shaped instruments are inserted into the eye. Instruments are manipulated by hand in four degrees of freedom about the insertion point. Two rotations move the instrument tip laterally, in addition to a translation in axial instrument direction and a rotation about its longitudinal axis. The manipulation of the instrument tip, e.g. a gripping motion can be considered as a fifth degree of freedom. While performing vitreo-retinal surgery manually, the surgeon faces various challenges. Typically, delicate micrometer range thick tissue is operated, for which steady hand movements and high accuracy instrument manipulation are required. Lateral instrument movements are inverted by the pivoting insertion point and scaled depending on the instrument insertion depth. A maximum of two instruments can be used simultaneously. There is nearly no perception of surgical forces, since most forces are below the human detection limit. Therefore, the surgeon relies only on visual feedback, obtained via a microscope or endoscope. Both vision systems force the surgeon to work in a static and non ergonomic body posture. Although the surgeon’s proficiency improves throughout his career, hand tremor will become a problem at higher age. Robotically assisted surgery with a master-slave system can assist the surgeon in these challenges. The slave system performs the actual surgery, by means of instrument manipulators which handle the instruments. The surgeon remains in control of the instruments by operating haptic interfaces via a master. Using electronic hardware and control software, the master and slave are connected. Amongst others, advantages as tremor filtering, up-scaled force feedback, down-scaled motions and stabilized instrument positioning will enhance dexterity on surgical tasks. Furthermore, providing the surgeon an ergonomic body posture will prolong the surgeon’s career. This thesis focuses on the design and realization of a high precision slave system for eye surgery. The master-slave system uses a table mounted design, where the system is compact, lightweight, easy to setup and equipped to perform a complete intervention. The slave system consists of two main parts: the instrument manipulators and their passive support system. Requirements are derived from manual eye surgery, conversations with medical specialists and analysis of the human anatomy and vitreo-retinal interventions. The passive support system provides a stiff connection between the instrument manipulator, patient and surgical table. Given the human anatomical diversity, presurgical adjustments can be made to allow the instrument manipulators to be positioned over each eye. Most of the support system is integrated within the patient’s headrest. On either the left or right side, two exchangeable manipulator-support arms can be installed onto the support system, depending on the eye being operated upon. The compact, lightweight and easy to install design, allows for a short setup time and quick removal in case of a complication. The slave system’s surgical reach is optimized to emulate manually performed surgery. For bimanual instrument operation, two instrument manipulators are used. Additional instrument manipulators can be used for non-active tools e.g. an illumination probe or an endoscope. An instrument manipulator allows the same degrees of freedom and a similar reach as manually performed surgery. Instrument forces are measured to supply force feedback to the surgeon via haptic interfaces. The instrument manipulator is designed for high stiffness, is play free and has low friction to allow tissue manipulation with high accuracy. Each instrument manipulator is equipped with an on board instrument change system, by which instruments can be changed in a fast and secure way. A compact design near the instrument allows easy access to the surgical area, leaving room for the microscope and peripheral equipment. The acceptance of a surgical robot for eye surgery mostly relies on equipment safety and reliability. The design of the slave system features various safety measures, e.g. a quick release mechanism for the instrument manipulator and additional locks on the pre-surgical adjustment fixation clamp. Additional safety measures are proposed, like a hard cover over the instrument manipulator and redundant control loops in the controlling FPGA. A method to fixate the patient’s head to the headrest by use of a custom shaped polymer mask is proposed. Two instrument manipulators and their passive support system have been realized so far, and the first experimental results confirm the designed low actuation torque and high precision performance

A Novel Head-mounted Display based Control in Robotic Surgery

학위논문 (석사) -- 서울대학교 대학원 : 공과대학 협동과정 바이오엔지니어링전공, 2020. 8. Sungwan Kim.현존하는 복강경 로봇 수술은 다양한 이점을 제공하지만 수술 중 집도의는 목, 어깨 그리고 등의 통증을 유발하는 불편한 자세를 유지해야 한다. 본 연구에서는 이러한 단점을 개선하기 위하여 머리 착용형 디스플레이(Head-mounted display, HMD)에 기반한 제어 시스템을 제안한다. 본 연구에서 활용되는 전체 시스템은 da Vinci research kit (dVRK)와 4 자유도의 내시경 제어 시스템 (Endoscope control system, ECS), 내시경 모듈, Attitude and heading reference system (AHRS)이 내장된 HMD로 구성된다. 4 자유도 ECS에 결합되어 사용되는 내시경 모듈은 HMD에 내장된 AHRS에 의해 제어되며, 이 HMD은 dVRK에 있는 Stereo viewer를 대체하여 수술 로봇 시스템의 크기를 축소할 수 있다. 수술 로봇 플랫폼에서 HMD 기반 제어의 적용 가능성을 평가하기 위해 해당 시스템을 다루어 본 경험이 없는 4명의 지원자들을 모집하여 Peg transfer task를 수행하게 하였다. 또한, HMD 기반 제어의 사용성을 평가하기 위해 Line tracking test를 진행하였다. 지원자들은 본 시스템에 빠른 속도로 학습하는 경향을 보여주었으며, 사용자 간 변산도 (Inter-user variability) 또한 매우 작았다. 내시경과 수술 도구를 동시에 제어하는 경우, 내시경 하드웨어와 수술 도구 사이의 충돌에 취약할 수 있다. 이에, HMD을 기반으로 ECS을 조종할 때에 적용되는 충돌 방지 전략을 개발하였다. 수술 기구와 내시경을 둘러싸는 Oriented bounding boxes (OBBs)를 정의하고 박스들 간의 충돌 가능성을 계산함으로써 수술 기구와 내시경 간의 충돌 여부를 추정하였으며, 사용자의 의도와 충돌 방지를 동시에 제어 신호에 반영하였다. dVRK에서는 End-effector의 실시간 위치 추적 데이터를 제공하지 않기 때문에 충돌 방지 전략을 검증하기 위해 MATLAB®을 활용한 컴퓨터 시뮬레이션을 수행하였다. 그 결과, 충돌 방지 전략이 수술 환경의 안전성을 보장함을 확인할 수 있었고, 안전성과 사용자의 의도 간 Trade-off를 적절히 고려한 Blending 파라미터 범위를 제안하였다. 본 연구를 통해 제안하는 HMD 기반 내시경 제어는 집도의의 목, 어깨 그리고 등에 발생하는 통증을 감소시킬 수 있어, 결과적으로 보다 효율적인 수술이 가능할 것이다. 또한, 기존의 Stereo viewer와 비교하여 공간적 효율성 또한 크게 개선할 수 있어, 차세대 수술 로봇의 제어 인터페이스로서 활용이 가능할 것으로 사료된다.Robotic laparoscopic surgery has provided various benefits, but during the surgery, the surgeons are experiencing uncomfortable positioning issue which leads to neck, shoulder, and back pain. For improving this issue, a novel head-mounted display (HMD) based endoscope control system (ECS) considering an ergonomic aspect is proposed in this research. The overall system is composed of a da Vinci research kit (dVRK), 4-degree-of-freedom ECS, endoscope module, and HMD with a built-in attitude and heading reference system (AHRS). The endoscope module is controlled by a built-in AHRS in the HMD. The stereo viewer in dVRK could be replaced by the HMD, so it would reduce the size of surgical robot system. Applicability of the proposed system to surgical robot platform was verified by peg-transfer task with four novice volunteers. Also, line tracking test was conducted to assess usability of the HMD based control. They showed rapid learning to the system and small value of inter-user variability. In the case of simultaneous control of HMD and surgical instruments, the collision issue between them could be raised. Thus, a collision avoidance strategy for HMD based ECS control was developed. Oriented bounding boxes (OBBs) containing the surgical instruments and endoscope were defined. And then, it is estimated whether the surgical instruments and endoscope collide through calculating the possibility between the OBBs. The control signal to endoscope includes both the user intention and collision avoidance strategy. dVRK does not provide real-time position data of its end-effectors, so computer-based simulations through MATLAB® were performed to verify the collision avoidance strategy. As a result, the strategy was assured of safety of surgery, and the range of blending parameter considering a trade-off between the user intention and safety was proposed. The HMD based ECS proposed in this research could reduce surgeons pains in neck, shoulder, and back, so it would lead to more efficient surgery. Additionally, space efficiency could be improved compared with the existing stereo viewer, so it is considered that the proposed system could be used as the control interface of the next-generation surgical robot.1. 서론 1 1.1. 수술 로봇 개요 1 1.1.1. 로봇을 활용한 최소 침습 수술 1 1.1.2. 현존하는 수술 로봇의 문제점 3 1.1.3. 선행 연구들의 문제점 3 1.2. 연구의 목적 4 1.2.1. HMD 기반의 내시경 제어 시스템 4 1.2.3. 충돌 방지 전략 5 1.3. 기대 효과 5 2. 방법 7 2.1. Hardware 구현 7 2.1.1. ECS를 활용한 수술 로봇의 구성 7 2.1.2. ECS 제어를 위한 HMD 9 2.2. HMD 기반의 제어 알고리즘 9 2.3. HMD 기반 제어의 검증 12 2.3.1. Peg transfer task 12 2.3.2. Line tracking test 15 2.4. 충돌 방지 12 2.4.1. 충돌 방지 알고리즘 17 2.4.2. 이론적 고찰 18 2.5. OBB 12 2.5.1. OBB의 정의 19 2.5.2. OBB 간의 거리 20 2.6. 보상 벡터의 계산 23 2.7. 충돌 방지의 검증 24 3. 결과 및 분석 27 3.1. HMD 기반 제어의 평가 27 3.2. 충돌 방지의 평가 29 4. 고찰 40 4.1. Peg transfer task 40 4.2. Line tracking test 41 4.3. 충돌 방지 알고리즘 42 5. 결론 43 5.1. 결론 43 5.2. 향후 연구 44 참고 문헌 45 Abstract 48 감사의 글 51Maste

Hand eye coordination in surgery

The coordination of the hand in response to visual target selection has always been regarded as an essential quality in a range of professional activities. This quality has thus far been elusive to objective scientific measurements, and is usually engulfed in the overall performance of the individuals. Parallels can be drawn to surgery, especially Minimally Invasive Surgery (MIS), where the physical constraints imposed by the arrangements of the instruments and visualisation methods require certain coordination skills that are unprecedented. With the current paradigm shift towards early specialisation in surgical training and shortened focused training time, selection process should identify trainees with the highest potentials in certain specific skills. Although significant effort has been made in objective assessment of surgical skills, it is only currently possible to measure surgeons’ abilities at the time of assessment. It has been particularly difficult to quantify specific details of hand-eye coordination and assess innate ability of future skills development. The purpose of this thesis is to examine hand-eye coordination in laboratory-based simulations, with a particular emphasis on details that are important to MIS. In order to understand the challenges of visuomotor coordination, movement trajectory errors have been used to provide an insight into the innate coordinate mapping of the brain. In MIS, novel spatial transformations, due to a combination of distorted endoscopic image projections and the “fulcrum” effect of the instruments, accentuate movement generation errors. Obvious differences in the quality of movement trajectories have been observed between novices and experts in MIS, however, this is difficult to measure quantitatively. A Hidden Markov Model (HMM) is used in this thesis to reveal the underlying characteristic movement details of a particular MIS manoeuvre and how such features are exaggerated by the introduction of rotation in the endoscopic camera. The proposed method has demonstrated the feasibility of measuring movement trajectory quality by machine learning techniques without prior arbitrary classification of expertise. Experimental results have highlighted these changes in novice laparoscopic surgeons, even after a short period of training. The intricate relationship between the hands and the eyes changes when learning a skilled visuomotor task has been previously studied. Reactive eye movement, when visual input is used primarily as a feedback mechanism for error correction, implies difficulties in hand-eye coordination. As the brain learns to adapt to this new coordinate map, eye movements then become predictive of the action generated. The concept of measuring this spatiotemporal relationship is introduced as a measure of hand-eye coordination in MIS, by comparing the Target Distance Function (TDF) between the eye fixation and the instrument tip position on the laparoscopic screen. Further validation of this concept using high fidelity experimental tasks is presented, where higher cognitive influence and multiple target selection increase the complexity of the data analysis. To this end, Granger-causality is presented as a measure of the predictability of the instrument movement with the eye fixation pattern. Partial Directed Coherence (PDC), a frequency-domain variation of Granger-causality, is used for the first time to measure hand-eye coordination. Experimental results are used to establish the strengths and potential pitfalls of the technique. To further enhance the accuracy of this measurement, a modified Jensen-Shannon Divergence (JSD) measure has been developed for enhancing the signal matching algorithm and trajectory segmentations. The proposed framework incorporates high frequency noise filtering, which represents non-purposeful hand and eye movements. The accuracy of the technique has been demonstrated by quantitative measurement of multiple laparoscopic tasks by expert and novice surgeons. Experimental results supporting visual search behavioural theory are presented, as this underpins the target selection process immediately prior to visual motor action generation. The effects of specialisation and experience on visual search patterns are also examined. Finally, pilot results from functional brain imaging are presented, where the Posterior Parietal Cortical (PPC) activation is measured using optical spectroscopy techniques. PPC has been demonstrated to involve in the calculation of the coordinate transformations between the visual and motor systems, which establishes the possibilities of exciting future studies in hand-eye coordination

Robotically assisted eye surgery : a haptic master console

Vitreo-retinal surgery encompasses the surgical procedures performed on the vitreous humor and the retina. A procedure typically consists of the removal of the vitreous humor, the peeling of a membrane and/or the repair of a retinal detachment. Operations are performed with needle shaped instruments which enter the eye through surgeon made scleral openings. An instrument is moved by hand in four degrees of freedom (three rotations and one translation) through this opening. Two rotations (? and ? ) are for a lateral instrument tip movement. The other two DoFs (z and ?) are the translation and rotation along the instrument axis. Actuation of for example a forceps can be considered as a fifth DoF. Characteristically, the manipulation of delicate, micrometer range thick intraocular tissue is required. Today, eye surgery is performed with a maximum of two instruments simultaneously. The surgeon relies on visual feedback only, since instrument forces are below the human detection limit. A microscope provides the visual feedback. It forces the surgeon to work in a static and non ergonomic body posture. Although the surgeon’s proficiency improves throughout his career, hand tremor may become a problem around his mid-fifties. Robotically assisted surgery with a master-slave system enhances dexterity. The slave with instrument manipulators is placed over the eye. The surgeon controls the instrument manipulators via haptic interfaces at the master. The master and slave are connected by electronic hardware and control software. Implementation of tremor filtering in the control software and downscaling of the hand motion allow prolongation of the surgeon’s career. Furthermore, it becomes possible to do tasks like intraocular cannulation which can not be done by manually performed surgery. This thesis focusses on the master console. Eye surgery procedures are observed in the operating room of different hospitals to gain insight in the requirements for the master. The master console as designed has an adjustable frame, a 3D display and two haptic interfaces with a coarse adjustment arm each. The console is mounted at the head of the operating table and is combined with the slave. It is compact, easy to place and allows the surgeon to have a direct view on and a physical contact with the patient. Furthermore, it fits in today’s manual surgery arrangement. Each haptic interface has the same five degrees of freedom as the instrument inside the eye. Through these interfaces, the surgeon can feel the augmented instrument forces. Downscaling of the hand motion results in a more accurate instrument movement compared to manually performed surgery. Together with the visual feedback, it is like the surgeon grasps the instrument near the tip inside the eye. The similarity between hand motion and motion of the instrument tip as seen on the display results in an intuitive manipulation. Pre-adjustment of the interface is done via the coarse adjustment arm. Mode switching enables to control three or more instruments manipulators with only two interfaces. Two one degree of freedom master-slave systems with force feedback are built to derive the requirements for the haptic interface. Hardware in the loop testing provides valuable insights and shows the possibility of force feedback without the use of force sensors. Two five DoF haptic interfaces are realized for bimanual operation. Each DoF has a position encoder and a force feedback motor. A correct representation of the upscaled instrument forces is only possible if the disturbance forces are low. Actuators are therefore mounted to the fixed world or in the neighborhood of the pivoting point for a low contribution to the inertia. The use of direct drive for ' and and low geared, backdriveable transmissions for the other three DoFs gives a minimum of friction. Disturbance forces are further minimized by a proper cable layout and actuator-amplifier combinations without torque ripple. The similarity in DoFs between vitreo-retinal eye surgery and minimally invasive surgery (MIS) enables the system to be used for MIS as well. Experiments in combination with a slave robot for laparoscopic and thoracoscopic surgery show that an instrument can be manipulated in a comfortable and intuitive way. User experience of surgeons and others is utilized to improve the haptic interface further. A parallel instead of a serial actuation concept for the ' and DoFs reduces the inertia, eliminates the flexible cable connection between frame and motor and allows that the heat of the motor is transferred directly to the frame. A newly designed z-?? module combines the actuation and suspension of the hand held part of the interface and has a three times larger z range than in the first design of the haptic interface

복강경 수술 로봇 시스템의 활용도 향상을 위한 추가적인 마스터 인터페이스 개발과 이를 이용한 응용 시스템 개발 연구

학위논문 (박사)-- 서울대학교 대학원 공과대학 협동과정 바이오엔지니어링전공, 2017. 8. Sungwan Kim.Robot-assisted laparoscopic surgery offers several advantages compared to open surgery and conventional minimally invasive surgery. However, important issues which need to be resolved are the complexity of current operation room environment for laparoscopic robotic surgery and demand for a larger operation room. To overcome these issues, additional interfaces based on Hands-On-Throttle-And-Stick (HOTAS) concept which can be simply attached and integrated with master interface of da Vinci surgical robot system were proposed. HOTAS controller is widely used for flight control in the aerospace field which can manipulate hundreds of functions and provide feedback to the pilot on flight conditions. The implementation of HOTAS controller significantly reduced the complexity of flights and reduced the number of pilots required in a cockpit from two to one. In this study, to provide above benefits to the operation room for robotic laparoscopic surgery, two types of additional interfaces are proposed. Proposed additional interfaces can be easily manipulated by the surgeons index finger, which is currently operated only by finger clutch buttons, and therefore enable the surgeon to use multiple functions. Initially, a novel master interface (NMI) was developed. The NMI mainly consists of a 9-way switch and a microprocessor with a wireless communication module. Thus, the NMI can be also regarded as a 9-way compact HOTAS. The performance test, latency, and power consumption of the developed NMI were verified by repeated experiments. Then, an improved novel master interface (iNMI) was developed to provide more intuitive and convenient manipulation. The iNMI was developed based on a capacitive touch sensor array and a wireless microprocessor to intuitively reflect the surgeons decision. Multiple experiments were performed to evaluate the iNMI performance in terms of performance test, latency, and power consumption. In addition, two application systems based on Surgical-Operation-By-Wire (SOBW) concept are proposed in this research to enhance the function of laparoscopic surgical robot system based on clinical needs that are stated below. The size of the additional interface is small enough to be easily installed to the master tool manipulators (MTMs) of da Vinci research kit (dVRK), which was used as an operation robot arm system, to maximize convenience to the surgeon when using the additional interfaces to simultaneously manipulate the application systems with the MTMs. Firstly, a robotic assistant that can be simultaneously manipulated via a wireless controller is proposed to allow the surgeon to control the assistant instrument. This approach not only decreases surgeon fatigue by eliminating communication process with assistants, but also resolves collision between the operation robot arms and the assistant instruments that can be caused by an inexperienced assistant or miscommunication and misaligned intent between the surgeon and the assistant. The system comprises two additional interfaces, a surgical instrument with a gripper actuated by a micromotor and a 6-axis robot arm. The gripping force of the surgical instrument was comparable to that of conventional systems and was consistent even after 1,000 times of gripping motion. The workspace was calculated to be 8,397.4 cm3. Recruited volunteers were able to execute the simple peg task within the cut-off time and successfully performed the in vitro test. Secondly, a wirelessly controllable stereo endoscope system which enables simultaneous control with the operating robot arm system is proposed. This is able to remove any discontinuous surgical flow that occurs when the control is swapped between the endoscope system and the operating robot arm system, and therefore prevent problems such as increased operation time, collision among surgical instruments, and injury to patients. The proposed system consists of two additional interfaces, a four-degrees of freedom (4-DOFs) endoscope control system (ECS) and a simple three-dimensional (3D) endoscope. The 4-DOFs ECS consists of four servo motors and employs a two-parallel link structure to provide translational and fulcrum point motions to the simple 3D endoscope. The workspace was calculated to be 20,378.3 cm3, which exceeds the reference workspace. The novice volunteers were able to successfully execute the modified peg transfer task. Throughout the various verifications, it has been confirmed that the proposed interfaces could make the surgical robot system more efficiently by overcoming its several limitations.1. Introduction 1 1.1. Robotic Laparoscopic Surgery 1 1.2. Objectives and Scope 8 1.2.1. Additional Master Interfaces 14 1.2.2. Application Systems 15 2. Materials and Methods 20 2.1. Additional Master Interfaces 20 2.1.1. Novel Master Interface: 9-way Compact Hands-On-Throttle-And-Stick 20 2.1.2. improved Novel Master Interface: Capacitive Touch Type Compact Hands-On-Throttle-And-Stick 26 2.2. Application Systems 34 2.2.1. Robotic Assistant 34 2.2.2. Stereo Endoscope System 49 3. Results 57 3.1. Novel Master Interface with Application Systems 57 3.1.1. Novel Master Interface 57 3.1.2. Robotic Assistant 59 3.1.3. Novel Master Interface with Robotic Assistant 67 3.1.4. Stereo Endoscope System 76 3.1.5. Novel Master Interface with Stereo Endoscope System 82 3.2. improved Novel Master Interface with Application Systems 87 3.2.1. improved Novel Master Interface 87 3.2.2. improved Novel Master Interface with Stereo Endoscope System 90 4. Discussion 91 5. Conclusion 102 References 105 Abstract in Korean 117Docto

Technology applications

A summary of NASA Technology Utilization programs for the period of 1 December 1971 through 31 May 1972 is presented. An abbreviated description of the overall Technology Utilization Applications Program is provided as a background for the specific applications examples. Subjects discussed are in the broad headings of: (1) cancer, (2) cardiovascular disease, (2) medical instrumentation, (4) urinary system disorders, (5) rehabilitation medicine, (6) air and water pollution, (7) housing and urban construction, (8) fire safety, (9) law enforcement and criminalistics, (10) transportation, and (11) mine safety