V-ANFIS for Dealing with Visual Uncertainty for Force Estimation in Robotic Surgery

Accurate and robust estimation of applied forces in Robotic-Assisted Minimally Invasive Surgery is a very challenging task. Many vision-based solutions attempt to estimate the force by measuring the surface deformation after contacting the surgical tool. However, visual uncertainty, due to tool occlusion, is a major concern and can highly affect the results' precision. In this paper, a novel design of an adaptive neuro-fuzzy inference strategy with a voting step (V-ANFIS) is used to accommodate with this loss of information. Experimental results show a significant accuracy improvement from 50% to 77% with respect to other proposals.Peer ReviewedPostprint (published version

Towards retrieving force feedback in robotic-assisted surgery: a supervised neuro-recurrent-vision approach

Robotic-assisted minimally invasive surgeries have gained a lot of popularity over conventional procedures as they offer many benefits to both surgeons and patients. Nonetheless, they still suffer from some limitations that affect their outcome. One of them is the lack of force feedback which restricts the surgeon's sense of touch and might reduce precision during a procedure. To overcome this limitation, we propose a novel force estimation approach that combines a vision based solution with supervised learning to estimate the applied force and provide the surgeon with a suitable representation of it. The proposed solution starts with extracting the geometry of motion of the heart's surface by minimizing an energy functional to recover its 3D deformable structure. A deep network, based on a LSTM-RNN architecture, is then used to learn the relationship between the extracted visual-geometric information and the applied force, and to find accurate mapping between the two. Our proposed force estimation solution avoids the drawbacks usually associated with force sensing devices, such as biocompatibility and integration issues. We evaluate our approach on phantom and realistic tissues in which we report an average root-mean square error of 0.02 N.Peer ReviewedPostprint (author's final draft

Sight to touch: 3D diffeomorphic deformation recovery with mixture components for perceiving forces in robotic-assisted surgery

Robotic-assisted minimally invasive surgical sys-tems suffer from one major limitation which is the lack of interaction forces feedback. The restricted sense of touch hinders the surgeons’ performance and reduces their dexterity and precision during a procedure. In this work, we present a sensory substitution approach that relies on visual stimuli to transmit the tool-tissue interaction forces to the operating surgeon. Our approach combines a 3D diffeomorphic defor-mation mapping with a generative model to precisely label the force level. The main highlights of our approach are that the use of diffeomorphic transformation ensures anatomical structure preservation and the label assignment is based on a parametric form of several mixture elements. We performed experimentations on both ex-vivo and in-vivo datasets and offer careful numerical results evaluating our approach. The results show that our solution has an error measure less than 1mm in all directions and an average labeling error of 2.05%. It can also be applicable to other scenarios that require force feedback such as microsurgery, knot tying or needle-based procedures.Peer ReviewedPostprint (author's final draft

Haptic assessment of tissue stiffness in locating and identifying gynaecological cancer in human tissue

Gynaecological surgeons are not able to gather adequate tissue feedback during minimal access surgery for cancer treatment. This can result in failure to locate tumour boundaries and to ensure these are completely resected within tumour-free resection margins. Surgeons achieve significantly better surgical and oncological outcomes if they can identify the precise location of a gynaecological tumour. Indeed, the true nature of tumour, whether benign or cancerous, is often not known prior to surgery. If more details were available in relation to the characteristics that differentiate gynaecological cancer in tumours, this would enable more accurate diagnosis and help in the planning of surgery. HYPOTHESIS: Haptic technology has the potential to enhance the surgeon’s degree of perception during minimal access surgery. Alteration in tissue stiffness in gynaecological tumours, thought to be associated with the accelerated multiplication of cancer cells, should allow their location to be identified and help in determining the likelihood of malignancy. METHOD: Setting: (i) Guy's & St Thomas' Hospital (ii) Dept of Informatics (King's College London).Permission from the National Research Ethics Committee and Research & Development (R&D) approval were sought from the National Health Service. The Phantom Omni, capable of 3D motion tracking, attached to a nano-17 force sensor, was used to capture real-time position data and force data. Uniaxial indentation palpation behaviour was used. The indentation depth was calculated using the displacement of the probe from the surface to the deepest point for each contact. The tissue stiffness (TS) was then calculated.The haptic probe was tested first on silicone models with embedded nodules mimicking tumour(s). This was followed by assessing TS ex-vivo using a haptic probe on fresh human gynaecological organs that had been removed in surgery. Tissue stiffness maps were generated in real time using the haptic device by converting stiffness values into RGB values. Surgeons also manually palpated and recorded the site of the tumour. Histology was used as the gold standard for location and cancer diagnosis. Manual palpation and haptic data were compared for accuracy on tumour location. The tissue stiffness calculated by the haptic probe was compared in cancer and control specimens. Several data analysis techniques were applied to derive results.CONTRIBUTIONS: Haptic indentation probe was tested for the first time on fresh human gynaecological organs to locate cancer in a clinical setting. We are the first one to evaluate the accuracy of cancer diagnosis in human gynaecological organs with a force sensing haptic indentation probe measuring tissue stiffness

Robot Autonomy for Surgery

Autonomous surgery involves having surgical tasks performed by a robot operating under its own will, with partial or no human involvement. There are several important advantages of automation in surgery, which include increasing precision of care due to sub-millimeter robot control, real-time utilization of biosignals for interventional care, improvements to surgical efficiency and execution, and computer-aided guidance under various medical imaging and sensing modalities. While these methods may displace some tasks of surgical teams and individual surgeons, they also present new capabilities in interventions that are too difficult or go beyond the skills of a human. In this chapter, we provide an overview of robot autonomy in commercial use and in research, and present some of the challenges faced in developing autonomous surgical robots

Force Measurement Methods in Telerobotic Surgery: Implications for End-Effector Manufacture

Haptic feedback in telesurgical applications refers to the relaying of position and force information from a remote surgical site to the surgeon in real-time during a surgical procedure. This feedback, coupled with visual information via microscopic cameras, has the potential to provide the surgeon with additional ‘feel’ for the manipulations being performed at the instrument-biological tissue interface. This increased sensitivity has many associated benefits which include, but are not limited to; minimal tissue damage, reduced recuperation periods, and less patient trauma. The inclusion of haptic feedback leads to reduction in surgeon fatigue which contributes to enhanced performance during operation. Commercially available Minimally Invasive Robotic Surgical (MIRS) systems are being widely used, the best-known examples being from the daVinci® by Intuitive Surgical Inc. However, currently these systems do not possess force feedback capability which therefore restricts their use during many delicate and complex procedures. The ideal system would consist of a multi-degree-of-freedom framework which includes end-effector instruments with embedded force sensing included. A force sensing characterisation platform has been developed by this group which facilitates the evaluation of force sensing technologies. Surgical scissors have been chosen as the instrument and biological tissue phantom specimens have been used during testing. This test-bed provides accurate, repeatable measurements of the forces produced at the interface between the tissue and the scissor blades during cutting using conventional sensing technologies. The primary focus of this paper is to provide a review of the traditional and developing force sensing technologies with a view to establishing the most appropriate solution for this application. The impact that an appropriate sensing technology has on the manufacturability of the instrument end-effector is considered. Particular attention is given to the issues of embedding the force sensing transducer into the instrument tip

Multi-axis Stiffness Sensing Device for Medical Palpation

This paper presents an innovative hand-held device able to compute stiffness when interacting with a soft object. The device is composed of four linear indenters and a USB camera. The stiffness is computed in real-time, tracking the movements of spherical features in the image of the camera. Those movements relate to the movements of the four indenters when interacting with a soft surface. Since the indenters are connected to springs with different spring constants, the displacement of the indenters varies when interacting with a soft object. The proposed multi-indenting device allows measuring the object's stiffness as well as the pan and tilt angles between the sensor and the surface of the soft object. Tests were performed to evaluate the accuracy of the proposed palpation mechanism against commercial springs of known stiffness. Results show that the accuracy and sensitivity of the proposed device increases with the softness of the examined object. Preliminary tests with silicon show the ability of the sensing mechanism to characterize phantom soft tissue for small indentation. It is noted that the results are not affected by the orientation of the device when probing the surface. The proposed sensing device can be used in different applications, such as external palpation for diagnosis or, if miniaturized, embedded on an endoscopic camera and used in Minimally Invasive Surgery (MIS)

A Mathematical Model of the Pneumatic Force Sensor for Robot-assisted Surgery

Restoring the sense of touch in robotic surgery is an emerging need several researchers tried to address. In this paper, we focused on the slave side proposing a pneumatic sensor to estimate contact forces occurring during the interaction between surgical instruments and anatomical areas. It consists of a tiny pneumatic balloon, which, after being inflated, appears near the tip of the instrument during the measurement phase only. This paper presents a mathematical method relating the intensity of the contact force to the variation of pressure inside the balloon. The latter was modeled as a spherical elastic membrane, whose behavior during contact was characterized taking into account both the deformation of the membrane and the compression of the contained gas. Geometrical considerations combined with an energetic approach allowed us to compute the force of interest. The effectiveness of our sensing device has been confirmed by experimental results, based on comparison with a high-performance commercial force sensor