DEVELOPMENT OF SPACE-INVARIANT SIGNATURE ALGORITHM (SISA) - AN INNOVATIVE APPROACH FOR PROCESSING THE MEDICAL IMAGES FOR THE DETECTION AND LOCALIZATION OF EARLY ABNORMALITIES IN BIOLOGICAL TISSUES

Early detection, diagnosis and localization are some of the important issues facing the medical profession for diseases such as cancer and cardiac disorders. Therefore, it is vital that a reliable approach, which is economic, safer and less time consuming be developed for the detection and diagnosis of such disorders. In this thesis an innovative approach, Space-Invariant Signature Algorithm (SISA) is proposed and developed to process the medical images for the detection and localization of abnormalities at an early stage in active biological tissues such as cancer, potential tumor growth and damaged tissues. In this proposed SISA approach, if the SISA signature pattern is space-invariant it suggests the absence of any abnormality. A space-variant SISA signature pattern is an indication of the presence of the abnormality. The abnormality in an active system can be defined as the obstacle, which impedes the smooth flow of activities such as blood or electrical signals. In any active system under excitation, abnormalities create extra perturbations, depending on the stage of progression of the abnormality. An abnormality in the final stage or critical stage will create very high perturbations that would largely impede the smooth flow of the excitation, whereas, in early stages, the abnormality will create low perturbations and would slightly impede the smooth flow of the excitation provided to the active system. Using the SISA approach, in the absence of any abnormalities, the signature pattern should have a uniform signature pattern, whereas, in the presences of abnormalities, the SISA signature pattern will be space-variant. The degree and position of the variance in space helps in the detection and localization of the abnormality. The SISA approach was first tested on a liquid vibrating system with various types of obstacles. These abnormalities created perturbations in the system parameters that induced the vibration patterns. Furthermore, the experimental results using the SISA approach were also obtained on ultrasound images to find abnormalities in animal tissues. In each of these cases of ultrasound imaging, the SISA signature patterns were able to localize and detect the tissue abnormality. The experimental results obtained with various types of small and large impedances (obstacles), which represent respectively the early and critical stages of abnormalities in the vibrating liquid system, were very encouraging. This basic SISA study on the liquid vibrating system was extended for processing ultrasound images for the detection and localization of damaged biological tissues. These initial experiments on animal tissues using ultrasound images along with SISA processing indicate that this innovative SISA approach has a great potential for processing other types of medical images such as ultrasound, Magnetic Resonance Elastography (MRE) and Computed Tomography (CT scan) for the detection and localization of abnormalities at an incipient stage

Towards retrieving force feedback in robotic-assisted surgery: a supervised neuro-recurrent-vision approach

Robotic-assisted minimally invasive surgeries have gained a lot of popularity over conventional procedures as they offer many benefits to both surgeons and patients. Nonetheless, they still suffer from some limitations that affect their outcome. One of them is the lack of force feedback which restricts the surgeon's sense of touch and might reduce precision during a procedure. To overcome this limitation, we propose a novel force estimation approach that combines a vision based solution with supervised learning to estimate the applied force and provide the surgeon with a suitable representation of it. The proposed solution starts with extracting the geometry of motion of the heart's surface by minimizing an energy functional to recover its 3D deformable structure. A deep network, based on a LSTM-RNN architecture, is then used to learn the relationship between the extracted visual-geometric information and the applied force, and to find accurate mapping between the two. Our proposed force estimation solution avoids the drawbacks usually associated with force sensing devices, such as biocompatibility and integration issues. We evaluate our approach on phantom and realistic tissues in which we report an average root-mean square error of 0.02 N.Peer ReviewedPostprint (author's final draft

Conditioned haptic perception for 3D localization of nodules in soft tissue palpation with a variable stiffness probe

This paper provides a solution for fast haptic information gain during soft tissue palpation using a Variable Lever Mechanism (VLM) probe. More specifically, we investigate the impact of stiffness variation of the probe to condition likelihood functions of the kinesthetic force and tactile sensors measurements during a palpation task for two sweeping directions. Using knowledge obtained from past probing trials or Finite Element (FE) simulations, we implemented this likelihood conditioning in an autonomous palpation control strategy. Based on a recursive Bayesian inferencing framework, this new control strategy adapts the sweeping direction and the stiffness of the probe to detect abnormal stiff inclusions in soft tissues. This original control strategy for compliant palpation probes shows a sub-millimeter accuracy for the 3D localization of the nodules in a soft tissue phantom as well as a 100% reliability detecting the existence of nodules in a soft phantom

Haptics in Robot-Assisted Surgery: Challenges and Benefits

Robotic surgery is transforming the current surgical practice, not only by improving the conventional surgical methods but also by introducing innovative robot-enhanced approaches that broaden the capabilities of clinicians. Being mainly of man-machine collaborative type, surgical robots are seen as media that transfer pre- and intra-operative information to the operator and reproduce his/her motion, with appropriate filtering, scaling, or limitation, to physically interact with the patient. The field, however, is far from maturity and, more critically, is still a subject of controversy in medical communities. Limited or absent haptic feedback is reputed to be among reasons that impede further spread of surgical robots. In this paper objectives and challenges of deploying haptic technologies in surgical robotics is discussed and a systematic review is performed on works that have studied the effects of providing haptic information to the users in major branches of robotic surgery. It has been tried to encompass both classical works and the state of the art approaches, aiming at delivering a comprehensive and balanced survey both for researchers starting their work in this field and for the experts

A Patient-Specific Approach for Breast Cancer Detection and Tumor Localization Using Infrared Imaging

Breast cancer (BC) is the most common cancer among women in the United States; approximately one out of every 24 women die of related causes. BC screening is a critical factor for improving patient prognosis and survival rate. Infrared (IR) thermography is an accurate, inexpensive and operator independent modality that is not affected by tissue density as it captures surface temperature variations induced by the presence of tumors. A novel patient-specific approach for IR imaging and simulation is proposed. In this work, multi-view IR images of isolated breasts are obtained in the prone position (face down), which allows access to the entire breast surface because the breasts hang freely. The challenge of accurately determining size and location of tumors within the breasts is addressed through numerical simulations of a patient-specific digital breast model. The digital breast models for individual patients are created from clinical images of the breast, such as IR imaging, digital photographs or magnetic resonance images. The numerical simulations of the digital breast model are conducted using ANSYS Fluent, where computed temperature images are generated in the same corresponding views as clinical IRI images. The computed and clinical IRI images are aligned and compared to measure their match. The determination of tumor size and location was conducted through the Levenberg-Marquardt algorithm, which iteratively minimized the mean squared error. The methodology was tested on the breasts of seven patients with biopsy-proven breast cancer with tumor diameters ranging from 8 mm to 27 mm. The method successfully predicted the equivalent tumor diameter within 2 mm and the location was predicted within 6.3 mm in all cases. The time required for the estimation is 48 minutes using a 10-core, 3.41 GHz workstation. The method presented is accurate, fast and has potential to be used as an adjunct modality to mammography in BC screening, especially for dense breasts

Soft Morphological Computation

Soft Robotics is a relatively new area of research, where progress in material science has powered the next generation of robots, exhibiting biological-like properties such as soft/elastic tissues, compliance, resilience and more besides. One of the issues when employing soft robotics technologies is the soft nature of the interactions arising between the robot and its environment. These interactions are complex, and the their dynamics are non-linear and hard to capture with known models. In this thesis we argue that complex soft interactions can actually be beneficial to the robot, and give rise to rich stimuli which can be used for the resolution of robot tasks. We further argue that the usefulness of these interactions depends on statistical regularities, or structure, that appear in the stimuli. To this end, robots should appropriately employ their morphology and their actions, to influence the system-environment interactions such that structure can arise in the stimuli. In this thesis we show that learning processes can be used to perform such a task. Following this rationale, this thesis proposes and supports the theory of Soft Morphological Computation (SoMComp), by which a soft robot should appropriately condition, or ‘affect’, the soft interactions to improve the quality of the physical stimuli arising from it. SoMComp is composed of four main principles, i.e.: Soft Proprioception, Soft Sensing, Soft Morphology and Soft Actuation. Each of these principles is explored in the context of haptic object recognition or object handling in soft robots. Finally, this thesis provides an overview of this research and its future directions.AHDB CP17

Complementary Situational Awareness for an Intelligent Telerobotic Surgical Assistant System

Robotic surgical systems have contributed greatly to the advancement of Minimally Invasive Surgeries (MIS). More specifically, telesurgical robots have provided enhanced dexterity to surgeons performing MIS procedures. However, current robotic teleoperated systems have only limited situational awareness of the patient anatomy and surgical environment that would typically be available to a surgeon in an open surgery. Although the endoscopic view enhances the visualization of the anatomy, perceptual understanding of the environment and anatomy is still lacking due to the absence of sensory feedback. In this work, these limitations are addressed by developing a computational framework to provide Complementary Situational Awareness (CSA) in a surgical assistant. This framework aims at improving the human-robot relationship by providing elaborate guidance and sensory feedback capabilities for the surgeon in complex MIS procedures. Unlike traditional teleoperation, this framework enables the user to telemanipulate the situational model in a virtual environment and uses that information to command the slave robot with appropriate admittance gains and environmental constraints. Simultaneously, the situational model is updated based on interaction of the slave robot with the task space environment. However, developing such a system to provide real-time situational awareness requires that many technical challenges be met. To estimate intraoperative organ information continuous palpation primitives are required. Intraoperative surface information needs to be estimated in real-time while the organ is being palpated/scanned. The model of the task environment needs to be updated in near real-time using the estimated organ geometry so that the force-feedback applied on the surgeon's hand would correspond to the actual location of the model. This work presents a real-time framework that meets these requirements/challenges to provide situational awareness of the environment in the task space. Further, visual feedback is also provided for the surgeon/developer to view the near video frame rate updates of the task model. All these functions are executed in parallel and need to have a synchronized data exchange. The system is very portable and can be incorporated to any existing telerobotic platforms with minimal overhead

Anatomic Characterization and Profilometry of Tissues with Natural Shape: A Real-time Approach for Robotic-Assisted Minimally Invasive Surgery

This master thesis is divided into two major sections. First, anatomic characterization and profilometry of tissues with natural shape: a real-time approach for robotic-assisted minimally invasive surgery (RMIS); and second, design and characterization of a novel tactile array sensor capable of differentiating among different viscoelastic tissues that exhibit time-dependent behaviour. The first part of this thesis is focused on a tissue characterization system for RMIS applications. RMIS has gained immense popularity with the advent of high-precision robotic systems. The lack of haptic feedback, however, is considered as being one of the main drawbacks of present-day RMIS systems. In order to compensate for this deficiency, a novel tissue characterization system is proposed which is inspired from the human haptic system. Hence, kinesthetic and tactile feedback which are constitutive components of human haptic system are used to characterize naturally shaped tissues. Toward this goal, a 5-degree-of-freedom robot which is called Catalys5 is equipped with a ball caster force-cell. The system is used to simulate robotic surgery maneuvers in which an admittance control approach is implemented to design the force feedback controller. The proposed method characterizes naturally shaped tissues, which is capable of touching and palpating to: a) Identify the 2D or 3D surface profile of the target tissue (profilometry), b) Measure the modulus of elasticity of any desired point on the tissue’s surface, c) Find and map the location of any lump in the tissue, and d) Map hardness distribution around the lump. Initially, silicon-rubber materials were used to build tissue phantoms with different curvatures and degrees of softness. The surface profiles were obtained using the developed profilometry algorithm and validated using a 3D scanner. In addition, several experiments were conducted on bovine tissues to evaluate all above mentioned capabilities of the system. The results of experiments on real tissues were also compared to those that are available in current literature. The results indicate that the proposed approach can be used for reliable material characterization for RMIS application. The second part of this thesis is focused on developing an array tactile sensor for distinguishing softness of viscoelastic tissues with time-dependent behaviour for use in MIS and RMIS. Review of literature on tactile sensors reveals that the vast majority deals with determining the applied contact force and object elasticity. In this research, a novel idea is proposed in which a tactile sensor array can measure rate of displacement in addition to force and displacement of any viscoelastic material during the course of a single touch. In order to verify this new array sensor, several experiments were conducted on a range of biological tissues. It was concluded that this novel tactile sensor can distinguish among the softness of real biological tissue with time-dependent behaviour

Estimating and understanding motion : from diagnostic to robotic surgery

Estimating and understanding motion from an image sequence is a central topic in computer vision. The high interest in this topic is because we are living in a world where many events that occur in the environment are dynamic. This makes motion estimation and understanding a natural component and a key factor in a widespread of applications including object recognition , 3D shape reconstruction, autonomous navigation and medica! diagnosis. Particularly, we focus on the medical domain in which understanding the human body for clinical purposes requires retrieving the organs' complex motion patterns, which is in general a hard problem when using only image data. In this thesis, we cope with this problem by posing the question - How to achieve a realistic motion estimation to offer a better clinical understanding? We focus this thesis on answering this question by using a variational formulation as a basis to understand one of the most complex motions in the human's body, the heart motion, through three different applications: (i) cardiac motion estimation for diagnostic, (ii) force estimation and (iii) motion prediction, both for robotic surgery. Firstly, we focus on a central topic in cardiac imaging that is the estimation of the cardiac motion. The main aim is to offer objective and understandable measures to physicians for helping them in the diagnostic of cardiovascular diseases. We employ ultrafast ultrasound data and tools for imaging motion drawn from diverse areas such as low-rank analysis and variational deformation to perform a realistic cardiac motion estimation. The significance is that by taking low-rank data with carefully chosen penalization, synergies in this complex variational problem can be created. We demonstrate how our proposed solution deals with complex deformations through careful numerical experiments using realistic and simulated data. We then move from diagnostic to robotic surgeries where surgeons perform delicate procedures remotely through robotic manipulators without directly interacting with the patients. As a result, they lack force feedback, which is an important primary sense for increasing surgeon-patient transparency and avoiding injuries and high mental workload. To solve this problem, we follow the conservation principies of continuum mechanics in which it is clear that the change in shape of an elastic object is directly proportional to the force applied. Thus, we create a variational framework to acquire the deformation that the tissues undergo due to an applied force. Then, this information is used in a learning system to find the nonlinear relationship between the given data and the applied force. We carried out experiments with in-vivo and ex-vivo data and combined statistical, graphical and perceptual analyses to demonstrate the strength of our solution. Finally, we explore robotic cardiac surgery, which allows carrying out complex procedures including Off-Pump Coronary Artery Bypass Grafting (OPCABG). This procedure avoids the associated complications of using Cardiopulmonary Bypass (CPB) since the heart is not arrested while performing the surgery on a beating heart. Thus, surgeons have to deal with a dynamic target that compromisetheir dexterity and the surgery's precision. To compensate the heart motion, we propase a solution composed of three elements: an energy function to estimate the 3D heart motion, a specular highlight detection strategy and a prediction approach for increasing the robustness of the solution. We conduct evaluation of our solution using phantom and realistic datasets. We conclude the thesis by reporting our findings on these three applications and highlight the dependency between motion estimation and motion understanding at any dynamic event, particularly in clinical scenarios.L’estimació i comprensió del moviment dins d’una seqüència d’imatges és un tema central en la visió per ordinador, el que genera un gran interès perquè vivim en un entorn ple d’esdeveniments dinàmics. Per aquest motiu és considerat com un component natural i factor clau dins d’un ampli ventall d’aplicacions, el qual inclou el reconeixement d’objectes, la reconstrucció de formes tridimensionals, la navegació autònoma i el diagnòstic de malalties. En particular, ens situem en l’àmbit mèdic en el qual la comprensió del cos humà, amb finalitats clíniques, requereix l’obtenció de patrons complexos de moviment dels òrgans. Aquesta és, en general, una tasca difícil quan s’utilitzen només dades de tipus visual. En aquesta tesi afrontem el problema plantejant-nos la pregunta - Com es pot aconseguir una estimació realista del moviment amb l’objectiu d’oferir una millor comprensió clínica? La tesi se centra en la resposta mitjançant l’ús d’una formulació variacional com a base per entendre un dels moviments més complexos del cos humà, el del cor, a través de tres aplicacions: (i) estimació del moviment cardíac per al diagnòstic, (ii) estimació de forces i (iii) predicció del moviment, orientant-se les dues últimes en cirurgia robòtica. En primer lloc, ens centrem en un tema principal en la imatge cardíaca, que és l’estimació del moviment cardíac. L’objectiu principal és oferir als metges mesures objectives i comprensibles per ajudar-los en el diagnòstic de les malalties cardiovasculars. Fem servir dades d’ultrasons ultraràpids i eines per al moviment d’imatges procedents de diverses àrees, com ara l’anàlisi de baix rang i la deformació variacional, per fer una estimació realista del moviment cardíac. La importància rau en que, en prendre les dades de baix rang amb una penalització acurada, es poden crear sinergies en aquest problema variacional complex. Mitjançant acurats experiments numèrics, amb dades realístiques i simulades, hem demostrat com les nostres propostes solucionen deformacions complexes. Després passem del diagnòstic a la cirurgia robòtica, on els cirurgians realitzen procediments delicats remotament, a través de manipuladors robòtics, sense interactuar directament amb els pacients. Com a conseqüència, no tenen la percepció de la força com a resposta, que és un sentit primari important per augmentar la transparència entre el cirurgià i el pacient, per evitar lesions i per reduir la càrrega de treball mental. Resolem aquest problema seguint els principis de conservació de la mecànica del medi continu, en els quals està clar que el canvi en la forma d’un objecte elàstic és directament proporcional a la força aplicada. Per això hem creat un marc variacional que adquireix la deformació que pateixen els teixits per l’aplicació d’una força. Aquesta informació s’utilitza en un sistema d’aprenentatge, per trobar la relació no lineal entre les dades donades i la força aplicada. Hem dut a terme experiments amb dades in-vivo i ex-vivo i hem combinat l’anàlisi estadístic, gràfic i de percepció que demostren la robustesa de la nostra solució. Finalment, explorem la cirurgia cardíaca robòtica, la qual cosa permet realitzar procediments complexos, incloent la cirurgia coronària sense bomba (off-pump coronary artery bypass grafting o OPCAB). Aquest procediment evita les complicacions associades a l’ús de circulació extracorpòria (Cardiopulmonary Bypass o CPB), ja que el cor no s’atura mentre es realitza la cirurgia. Això comporta que els cirurgians han de tractar amb un objectiu dinàmic que compromet la seva destresa i la precisió de la cirurgia. Per compensar el moviment del cor, proposem una solució composta de tres elements: un funcional d’energia per estimar el moviment tridimensional del cor, una estratègia de detecció de les reflexions especulars i una aproximació basada en mètodes de predicció, per tal d’augmentar la robustesa de la solució. L’avaluació de la nostra solució s’ha dut a terme mitjançant conjunts de dades sintètiques i realistes. La tesi conclou informant dels nostres resultats en aquestes tres aplicacions i posant de relleu la dependència entre l’estimació i la comprensió del moviment en qualsevol esdeveniment dinàmic, especialment en escenaris clínics.Postprint (published version