Ανάπτυξη τεχνολογιών επαυξημένης πραγματικότητας στην ιατρική εκπαίδευση με προσομοιωτές

Στην παρούσα διδακτορική διατριβή παρουσιάζουμε ένα πρωτοπόρο σύστημα εκπαίδευσης και αξιολόγησης βασικών δεξιοτήτων λαπαροσκοπικής χειρουργικής σε περιβάλλον Επαυξημένης Πραγματικότητας (ΕΠ). Το προτεινόμενο σύστημα αποτελεί μια πλήρως λειτουργική πλατφόρμα εκπαίδευσης η οποία επιτρέπει σε χειρουργούς να εξασκηθούν χρησιμοποιώντας πραγματικά λαπαροσκοπικά εργαλεία και αλληλεπιδρώντας με ψηφιακά αντικείμενα εντός ενός πραγματικού περιβάλλοντος εκπαίδευσης. Το σύστημα αποτελείται από ένα τυπικό κουτί λαπαροσκοπικής εκπαίδευσης, πραγματικά χειρουργικά εργαλεία, κάμερα και συστοιχία αισθητήρων που επιτρέπουν την ανίχνευση και καταγραφή των κινήσεων του χειρουργού σε πραγματικό χρόνο. Χρησιμοποιώντας το προτεινόμενο σύστημα, σχεδιάσαμε και υλοποιήσαμε σενάρια εκπαίδευσης παρόμοια με τις ασκήσεις του προγράμματος FLS®, στοχεύοντας σε δεξιότητες όπως η αίσθηση βάθους, ο συντονισμός χεριού-ματιού, και η παράλληλη χρήση δύο χεριών. Επιπλέον των βασικών δεξιοτήτων, το προτεινόμενο σύστημα χρησιμοποιήθηκε για τον σχεδιασμό σεναρίου εξάσκησης διαδικαστικών δεξιοτήτων, οι οποίες περιλάμβανουν την εφαρμογή χειρουργικών clips καθώς και την απολίνωση εικονικής αρτηρίας, σε περιβάλλον ΕΠ. Τα αποτελέσματα συγκριτικών μελετών μεταξύ έμπειρων και αρχαρίων χειρουργών που πραγματοποιήθηκαν στα πλαίσια της παρούσας διατριβής υποδηλώνουν την εγκυρότητα του προτεινόμενου συστήματος. Επιπλέον, εξήχθησαν σημαντικά συμπεράσματα σχετικά με την πιθανή χρήση της ΕΑ στην λαπαροσκοπική προσομοίωση. Η συγκεκριμένη τεχνολογία προσφέρει αυξημένη αίσθηση οπτικού ρεαλισμού και ευελιξία στον σχεδιασμό εκπαιδευτικών σεναρίων, παρουσιάζοντας σημαντικά μικρότερες απαιτήσεις από πλευράς εξοπλισμού σε σύγκριση με τις υπάρχουσες εμπορικές πλατφόρμες. Βάσει των αποτελεσμάτων της παρούσας διατριβής μπορεί με ασφάλεια να εξαχθεί το συμπέρασμα πως η ΕΠ αποτελεί μια πολλά υποσχόμενη τεχνολογία που θα μπορούσε να χρησιμοποιηθεί για τον σχεδιασμό προσομοιωτών λαπαροσκοπικής χειρουργικής ως εναλλακτική των υπαρχόντων τεχνολογιών και συστημάτων.In this thesis we present what is, to the best of our knowledge, the first framework for training and assessment of fundamental psychomotor and procedural laparoscopic skills in an interactive Augmented Reality (AR) environment. The proposed system is a fully-featured laparoscopic training platform, allowing surgeons to practice by manipulating real instruments while interacting with virtual objects within a real environment. It consists of a standard laparoscopic box-trainer, real instruments, a camera and a set of sensory devices for real-time tracking of surgeons’ actions. The proposed framework has been used for the implementation of AR-based training scenarios similar to the drills of the FLS® program, focusing on fundamental laparoscopic skills such as depth-perception, hand-eye coordination and bimanual operation. Moreover, this framework allowed the implementation of a proof-of-concept procedural skills training scenario, which involved clipping and cutting of a virtual artery within an AR environment. Comparison studies conducted for the evaluation of the presented framework indicated high content and face validity. In addition, significant conclusions regarding the potentials of introducing AR in laparoscopic simulation training and assessment were drawn. This technology provides an advanced sense of visual realism combined with a great flexibility in training task prototyping, with minimum requirements in terms of hardware as compared to commercially available platforms. Thereby, it can be safely stated that AR is a promising technology which can indeed provide a valuable alternative to the training modalities currently used in MIS

Estimating and understanding motion : from diagnostic to robotic surgery

Estimating and understanding motion from an image sequence is a central topic in computer vision. The high interest in this topic is because we are living in a world where many events that occur in the environment are dynamic. This makes motion estimation and understanding a natural component and a key factor in a widespread of applications including object recognition , 3D shape reconstruction, autonomous navigation and medica! diagnosis. Particularly, we focus on the medical domain in which understanding the human body for clinical purposes requires retrieving the organs' complex motion patterns, which is in general a hard problem when using only image data. In this thesis, we cope with this problem by posing the question - How to achieve a realistic motion estimation to offer a better clinical understanding? We focus this thesis on answering this question by using a variational formulation as a basis to understand one of the most complex motions in the human's body, the heart motion, through three different applications: (i) cardiac motion estimation for diagnostic, (ii) force estimation and (iii) motion prediction, both for robotic surgery. Firstly, we focus on a central topic in cardiac imaging that is the estimation of the cardiac motion. The main aim is to offer objective and understandable measures to physicians for helping them in the diagnostic of cardiovascular diseases. We employ ultrafast ultrasound data and tools for imaging motion drawn from diverse areas such as low-rank analysis and variational deformation to perform a realistic cardiac motion estimation. The significance is that by taking low-rank data with carefully chosen penalization, synergies in this complex variational problem can be created. We demonstrate how our proposed solution deals with complex deformations through careful numerical experiments using realistic and simulated data. We then move from diagnostic to robotic surgeries where surgeons perform delicate procedures remotely through robotic manipulators without directly interacting with the patients. As a result, they lack force feedback, which is an important primary sense for increasing surgeon-patient transparency and avoiding injuries and high mental workload. To solve this problem, we follow the conservation principies of continuum mechanics in which it is clear that the change in shape of an elastic object is directly proportional to the force applied. Thus, we create a variational framework to acquire the deformation that the tissues undergo due to an applied force. Then, this information is used in a learning system to find the nonlinear relationship between the given data and the applied force. We carried out experiments with in-vivo and ex-vivo data and combined statistical, graphical and perceptual analyses to demonstrate the strength of our solution. Finally, we explore robotic cardiac surgery, which allows carrying out complex procedures including Off-Pump Coronary Artery Bypass Grafting (OPCABG). This procedure avoids the associated complications of using Cardiopulmonary Bypass (CPB) since the heart is not arrested while performing the surgery on a beating heart. Thus, surgeons have to deal with a dynamic target that compromisetheir dexterity and the surgery's precision. To compensate the heart motion, we propase a solution composed of three elements: an energy function to estimate the 3D heart motion, a specular highlight detection strategy and a prediction approach for increasing the robustness of the solution. We conduct evaluation of our solution using phantom and realistic datasets. We conclude the thesis by reporting our findings on these three applications and highlight the dependency between motion estimation and motion understanding at any dynamic event, particularly in clinical scenarios.L’estimació i comprensió del moviment dins d’una seqüència d’imatges és un tema central en la visió per ordinador, el que genera un gran interès perquè vivim en un entorn ple d’esdeveniments dinàmics. Per aquest motiu és considerat com un component natural i factor clau dins d’un ampli ventall d’aplicacions, el qual inclou el reconeixement d’objectes, la reconstrucció de formes tridimensionals, la navegació autònoma i el diagnòstic de malalties. En particular, ens situem en l’àmbit mèdic en el qual la comprensió del cos humà, amb finalitats clíniques, requereix l’obtenció de patrons complexos de moviment dels òrgans. Aquesta és, en general, una tasca difícil quan s’utilitzen només dades de tipus visual. En aquesta tesi afrontem el problema plantejant-nos la pregunta - Com es pot aconseguir una estimació realista del moviment amb l’objectiu d’oferir una millor comprensió clínica? La tesi se centra en la resposta mitjançant l’ús d’una formulació variacional com a base per entendre un dels moviments més complexos del cos humà, el del cor, a través de tres aplicacions: (i) estimació del moviment cardíac per al diagnòstic, (ii) estimació de forces i (iii) predicció del moviment, orientant-se les dues últimes en cirurgia robòtica. En primer lloc, ens centrem en un tema principal en la imatge cardíaca, que és l’estimació del moviment cardíac. L’objectiu principal és oferir als metges mesures objectives i comprensibles per ajudar-los en el diagnòstic de les malalties cardiovasculars. Fem servir dades d’ultrasons ultraràpids i eines per al moviment d’imatges procedents de diverses àrees, com ara l’anàlisi de baix rang i la deformació variacional, per fer una estimació realista del moviment cardíac. La importància rau en que, en prendre les dades de baix rang amb una penalització acurada, es poden crear sinergies en aquest problema variacional complex. Mitjançant acurats experiments numèrics, amb dades realístiques i simulades, hem demostrat com les nostres propostes solucionen deformacions complexes. Després passem del diagnòstic a la cirurgia robòtica, on els cirurgians realitzen procediments delicats remotament, a través de manipuladors robòtics, sense interactuar directament amb els pacients. Com a conseqüència, no tenen la percepció de la força com a resposta, que és un sentit primari important per augmentar la transparència entre el cirurgià i el pacient, per evitar lesions i per reduir la càrrega de treball mental. Resolem aquest problema seguint els principis de conservació de la mecànica del medi continu, en els quals està clar que el canvi en la forma d’un objecte elàstic és directament proporcional a la força aplicada. Per això hem creat un marc variacional que adquireix la deformació que pateixen els teixits per l’aplicació d’una força. Aquesta informació s’utilitza en un sistema d’aprenentatge, per trobar la relació no lineal entre les dades donades i la força aplicada. Hem dut a terme experiments amb dades in-vivo i ex-vivo i hem combinat l’anàlisi estadístic, gràfic i de percepció que demostren la robustesa de la nostra solució. Finalment, explorem la cirurgia cardíaca robòtica, la qual cosa permet realitzar procediments complexos, incloent la cirurgia coronària sense bomba (off-pump coronary artery bypass grafting o OPCAB). Aquest procediment evita les complicacions associades a l’ús de circulació extracorpòria (Cardiopulmonary Bypass o CPB), ja que el cor no s’atura mentre es realitza la cirurgia. Això comporta que els cirurgians han de tractar amb un objectiu dinàmic que compromet la seva destresa i la precisió de la cirurgia. Per compensar el moviment del cor, proposem una solució composta de tres elements: un funcional d’energia per estimar el moviment tridimensional del cor, una estratègia de detecció de les reflexions especulars i una aproximació basada en mètodes de predicció, per tal d’augmentar la robustesa de la solució. L’avaluació de la nostra solució s’ha dut a terme mitjançant conjunts de dades sintètiques i realistes. La tesi conclou informant dels nostres resultats en aquestes tres aplicacions i posant de relleu la dependència entre l’estimació i la comprensió del moviment en qualsevol esdeveniment dinàmic, especialment en escenaris clínics.Postprint (published version

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task

Learning efficient haptic shape exploration with a rigid tactile sensor array

Haptic exploration is a key skill for both robots and humans to discriminate and handle unknown objects or to recognize familiar objects. Its active nature is evident in humans who from early on reliably acquire sophisticated sensory-motor capabilities for active exploratory touch and directed manual exploration that associates surfaces and object properties with their spatial locations. This is in stark contrast to robotics. In this field, the relative lack of good real-world interaction models - along with very restricted sensors and a scarcity of suitable training data to leverage machine learning methods - has so far rendered haptic exploration a largely underdeveloped skill. In the present work, we connect recent advances in recurrent models of visual attention with previous insights about the organisation of human haptic search behavior, exploratory procedures and haptic glances for a novel architecture that learns a generative model of haptic exploration in a simulated three-dimensional environment. The proposed algorithm simultaneously optimizes main perception-action loop components: feature extraction, integration of features over time, and the control strategy, while continuously acquiring data online. We perform a multi-module neural network training, including a feature extractor and a recurrent neural network module aiding pose control for storing and combining sequential sensory data. The resulting haptic meta-controller for the rigid $16 \times 16$ tactile sensor array moving in a physics-driven simulation environment, called the Haptic Attention Model, performs a sequence of haptic glances, and outputs corresponding force measurements. The resulting method has been successfully tested with four different objects. It achieved results close to $100 \%$ while performing object contour exploration that has been optimized for its own sensor morphology

Computerized Evaluatution of Microsurgery Skills Training

The style of imparting medical training has evolved, over the years. The traditional methods of teaching and practicing basic surgical skills under apprenticeship model, no longer occupy the first place in modern technically demanding advanced surgical disciplines like neurosurgery. Furthermore, the legal and ethical concerns for patient safety as well as cost-effectiveness have forced neurosurgeons to master the necessary microsurgical techniques to accomplish desired results. This has lead to increased emphasis on assessment of clinical and surgical techniques of the neurosurgeons. However, the subjective assessment of microsurgical techniques like micro-suturing under the apprenticeship model cannot be completely unbiased. A few initiatives using computer-based techniques, have been made to introduce objective evaluation of surgical skills. This thesis presents a novel approach involving computerized evaluation of different components of micro-suturing techniques, to eliminate the bias of subjective assessment. The work involved acquisition of cine clips of micro-suturing activity on synthetic material. Image processing and computer vision based techniques were then applied to these videos to assess different characteristics of micro-suturing viz. speed, dexterity and effectualness. In parallel subjective grading on these was done by a senior neurosurgeon. Further correlation and comparative study of both the assessments was done to analyze the efficacy of objective and subjective evaluation

Embodied Interactions for Spatial Design Ideation: Symbolic, Geometric, and Tangible Approaches

Computer interfaces are evolving from mere aids for number crunching into active partners in creative processes such as art and design. This is, to a great extent, the result of mass availability of new interaction technology such as depth sensing, sensor integration in mobile devices, and increasing computational power. We are now witnessing the emergence of maker culture that can elevate art and design beyond the purview of enterprises and professionals such as trained engineers and artists. Materializing this transformation is not trivial; everyone has ideas but only a select few can bring them to reality. The challenge is the recognition and the subsequent interpretation of human actions into design intent

Virtual Reality Integration on Tomo-GPU System

With the ever-greater creation of data, new ways to extract information from it in faster ways is a subject of great interest to the scientific community in general and any entity that may benefit with the interpretation of data. Virtual reality, although not a recent discovery only now is becoming broadly available and driving new state of the art designs and implementations. Nonetheless, already existing results, provide positive feedback of virtual reality on some cases of data visualization. One of the scientific areas that may benefit from virtual reality technology visualization is the scientific field of material sciences. A current project of FCT is the Tomo-GPU system that was developed to aid the material scientists in processing and visualizing their data. This work focuses on the integration of a virtual reality visualization on the Tomo-GPU system to aid material scientist in interpreting their data more efficiently