The effect of indentation force and displacement on visual perception of compliance

This paper investigates the effect of maximum indentation force and depth on people's ability to accurately discriminate compliance using indirect visual information only. Participants took part in two psychophysical experiments in which they were asked to choose the 'softest' sample out of a series of presented sample pairs. In the experiments, participants observed a computer-actuated tip indent the sample pairs to one of two conditions; maximum depth (10mm) or maximum force (4N). This indentation process simulates tool operated palpation in laparoscopic surgery. Results were used to plot psychometric functions as a measure of accuracy of compliance discriminability. A comparison indicated that participants performed best in the task where they judged samples being indented to a pre-set maximum force relying solely on visual cues, which demonstrates the effect of visual information on compliance discrimination. Results also show that indentation cues such as force and deformation depth have different effects on our ability to visually discriminate compliance. These findings will inform future work on designing a haptic feedback system capable of augmenting visual and haptic information independently for optimal compliance discrimination performance

Haptic threshold for pulling force feedback on surgeon\u27s fingertip in medical robotic systems

The human fingertip has very high density of the receptor to accept sense of touch stimulation. The corresponding somatic sensory area in a brain is very large, and considered to be a specialized part for palpation. A lot of haptic display system then have been developed with the investigation of human haptic perception. However, the researches about the human perception for pulling force at grasping, namely static frictional force are limited. This paper investigated it, aiming at a future development of pulling and grasping force feedback system for neurosurgical robotic systems. For the purpose, this paper explored the possibility of displaying pulling force to an index finger during grasping. The absolute and difference thresholds for pulling sense were the targets. The results showed that grasping disturbs the pulling sense, and the sides of index fingertip can be used to display pulling sense, relatively large force, namely scaled force feedback is required for the perception. The results provide an important insight at a hardware and controller design of force feedback systems. © 2016 IEEE.42nd Conference of the Industrial Electronics Society, IECON 2016; Palazzo dei CongressiFlorence; Italy; 24 October 2016 through 27 October 2016; Category numberCFP16IEC-ART; Code 12554

Control of Cooperative Haptics-Enabled Teleoperation Systems with Application to Minimally Invasive Surgery

Robot-Assisted Minimally Invasive Surgical (RAMIS) systems frequently have a structure of cooperative teleoperator systems where multiple master-slave pairs are used to collaboratively execute a task. Although multiple studies indicate that haptic feedback improves the realism of tool-tissue interaction to the surgeon and leads to better performance for surgical procedures, current telesurgical systems typically do not provide force feedback, mainly because of the inherent stability issues. The research presented in this thesis is directed towards the development of control algorithms for force reflecting cooperative surgical teleoperator systems with improved stability and transparency characteristics. In the case of cooperative force reflecting teleoperation over networks, conventional passivity based approaches may have limited applicability due to potentially non-passive slave-slave interactions and irregular communication delays imposed by the network. In this thesis, an alternative small gain framework for the design of cooperative network-based force reflecting teleoperator systems is developed. Using the small gain framework, control algorithms for cooperative force-reflecting teleoperator systems are designed that guarantee stability in the presence of multiple network-induced communication constraints. Furthermore, the design conservatism typically associated with the small-gain approach is eliminated by using the Projection-Based Force Reflection (PBFR) algorithms. Stability results are established for networked cooperative teleoperator systems under different types of force reflection algorithms in the presence of irregular communication delays. The proposed control approach is consequently implemented on a dual-arm (two masters/two slaves) robotic MIS testbed. The testbed consists of two Haptic Wand devices as masters and two PA10-7C robots as the slave manipulators equipped with da Vinci laparoscopic surgical instruments. The performance of the proposed control approach is evaluated in three different cooperative surgical tasks, which are knot tightening, pegboard transfer, and object manipulation. The experimental results obtained indicate that the PBFR algorithms demonstrate statistically significant performance improvement in comparison with the conventional direct force reflection algorithms. One possible shortcoming of using PBFR algorithms is that implementation of these algorithms may lead to attenuation of the high-frequency component of the contact force which is important, in particular, for haptic perception of stiff surfaces. In this thesis, a solution to this problem is proposed which is based on the idea of separating the different frequency bands in the force reflection signal and consequently applying the projection-based principle to the low-frequency component, while reflecting the high-frequency component directly. The experimental results demonstrate that substantial improvement in transient fidelity of the force feedback is achieved using the proposed method without negative effects on the stability of the system

Microscope Embedded Neurosurgical Training and Intraoperative System

In the recent years, neurosurgery has been strongly influenced by new technologies. Computer Aided Surgery (CAS) offers several benefits for patients\u27 safety but fine techniques targeted to obtain minimally invasive and traumatic treatments are required, since intra-operative false movements can be devastating, resulting in patients deaths. The precision of the surgical gesture is related both to accuracy of the available technological instruments and surgeon\u27s experience. In this frame, medical training is particularly important. From a technological point of view, the use of Virtual Reality (VR) for surgeon training and Augmented Reality (AR) for intra-operative treatments offer the best results. In addition, traditional techniques for training in surgery include the use of animals, phantoms and cadavers. The main limitation of these approaches is that live tissue has different properties from dead tissue and that animal anatomy is significantly different from the human. From the medical point of view, Low-Grade Gliomas (LGGs) are intrinsic brain tumours that typically occur in younger adults. The objective of related treatment is to remove as much of the tumour as possible while minimizing damage to the healthy brain. Pathological tissue may closely resemble normal brain parenchyma when looked at through the neurosurgical microscope. The tactile appreciation of the different consistency of the tumour compared to normal brain requires considerable experience on the part of the neurosurgeon and it is a vital point. The first part of this PhD thesis presents a system for realistic simulation (visual and haptic) of the spatula palpation of the LGG. This is the first prototype of a training system using VR, haptics and a real microscope for neurosurgery. This architecture can be also adapted for intra-operative purposes. In this instance, a surgeon needs the basic setup for the Image Guided Therapy (IGT) interventions: microscope, monitors and navigated surgical instruments. The same virtual environment can be AR rendered onto the microscope optics. The objective is to enhance the surgeon\u27s ability for a better intra-operative orientation by giving him a three-dimensional view and other information necessary for a safe navigation inside the patient. The last considerations have served as motivation for the second part of this work which has been devoted to improving a prototype of an AR stereoscopic microscope for neurosurgical interventions, developed in our institute in a previous work. A completely new software has been developed in order to reuse the microscope hardware, enhancing both rendering performances and usability. Since both AR and VR share the same platform, the system can be referred to as Mixed Reality System for neurosurgery. All the components are open source or at least based on a GPL license

Haptics-Enabled Teleoperation for Robotics-Assisted Minimally Invasive Surgery

The lack of force feedback (haptics) in robotic surgery can be considered to be a safety risk leading to accidental tissue damage and puncturing of blood vessels due to excessive forces being applied to tissue and vessels or causing inefficient control over the instruments because of insufficient applied force. This project focuses on providing a satisfactory solution for introducing haptic feedback in robotics-assisted minimally invasive surgical (RAMIS) systems. The research addresses several key issues associated with the incorporation of haptics in a master-slave (teleoperated) robotic environment for minimally invasive surgery (MIS). In this project, we designed a haptics-enabled dual-arm (two masters - two slaves) robotic MIS testbed to investigate and validate various single-arm as well as dual-arm teleoperation scenarios. The most important feature of this setup is the capability of providing haptic feedback in all 7 degrees of freedom (DOF) required for RAMIS (3 translations, 3 rotations and pinch motion of the laparoscopic tool). The setup also enables the evaluation of the effect of replacing haptic feedback by other sensory cues such as visual representation of haptic information (sensory substitution) and the hypothesis that surgical outcomes may be improved by substituting or augmenting haptic feedback by such sensory cues

The Role of Visualization, Force Feedback, and Augmented Reality in Minimally Invasive Heart Valve Repair

New cardiovascular techniques have been developed to address the unique requirements of high risk, elderly, surgical patients with heart valve disease by avoiding both sternotomy and cardiopulmonary bypass. However, these technologies pose new challenges in visualization, force application, and intracardiac navigation. Force feedback and augmented reality (AR) can be applied to minimally invasive mitral valve repair and transcatheter aortic valve implantation (TAVI) techniques to potentially surmount these challenges. Our study demonstrated shorter operative times with three dimensional (3D) visualization compared to two dimensional (2D) visualization; however, both experts and novices applied significantly more force to cardiac tissue during 3D robotics-assisted mitral valve annuloplasty than during conventional open mitral valve annuloplasty. This finding suggests that 3D visualization does not fully compensate for the absence of haptic feedback in robotics-assisted cardiac surgery. Subsequently, using an innovative robotics-assisted surgical system design, we determined that direct haptic feedback may improve both expert and trainee performance using robotics-assisted techniques. We determined that during robotics-assisted mitral valve annuloplasty the use of either visual or direct force feedback resulted in a significant decrease in forces applied to cardiac tissue when compared to robotics-assisted mitral valve annuloplasty without force feedback. We presented NeoNav, an AR-enhanced echocardiograpy intracardiac guidance system for NeoChord off-pump mitral valve repair. Our study demonstrated superior tool navigation accuracy, significantly shorter navigation times, and reduced potential for injury with AR enhanced intracardiac navigation for off-pump transapical mitral valve repair with neochordae implantation. In addition, we applied the NeoNav system as a safe and inexpensive alternative imaging modality for TAVI guidance. We found that our proposed AR guidance system may achieve similar or better results than the current standard of care, contrast enhanced fluoroscopy, while eliminating the use of nephrotoxic contrast and ionizing radiation. These results suggest that the addition of both force feedback and augmented reality image guidance can improve both surgical performance and safety during minimally invasive robotics assisted and beating heart valve surgery, respectively

Determining the Contribution of Visual and Haptic Cues during Compliance Discrimination in the Context of Minimally Invasive Surgery

While minimally invasive surgery is replacing open surgery in an increasing number of surgical procedures, it still poses risks such as unintended tissue damage due to reduced visual and haptic feedback. Surgeons assess tissue health by analysing mechanical properties such as compliance. The literature shows that while both types of feedback contribute to the final percept, visual information is dominant during compliance discrimination tasks. The magnitude of that contribution, however, was never quantitatively determined. To determine the effect of the type of visual feedback on compliance discrimination, a psychophysical experiment was set up using different combinations of direct and indirect visual and haptic cues. Results reiterated the significance of visual information and suggested a visio-haptic cross-modal integration. Consequently, to determine which cues contributed most to visual feedback, the impact of force and position on the ability to discriminate compliance using visual information only was assessed. Results showed that isolating force and position cues during indentation enhanced performance. Furthermore, under force and position constraints, visual information was shown to be sufficient to determine the compliance of deformable objects. A pseudo-haptic feedback system was developed to quantitatively determine the contribution of visual feedback during compliance discrimination. A psychophysical experiment showed that the system realistically simulated viscoelastic behaviour of compliant objects. Through a magnitude estimation experiment, the pseudo-haptic system was shown to be successful at generating haptic sensations of compliance during stimuli indentation only by modifying the visual feedback presented to participants. This can be implemented in research and educational facilities where advanced force feedback devices are inaccessible. Moreover, it can be incorporated into virtual reality simulators to enhance force ranges. Future work will assess the value of visual cue augmentation in more complicated surgical tasks