183 research outputs found

    Teaching Unknown Objects by Leveraging Human Gaze and Augmented Reality in Human-Robot Interaction

    Get PDF
    Roboter finden aufgrund ihrer außergewöhnlichen Arbeitsleistung, PrĂ€zision, Effizienz und Skalierbarkeit immer mehr Verwendung in den verschiedensten Anwendungsbereichen. Diese Entwicklung wurde zusĂ€tzlich begĂŒnstigt durch Fortschritte in der KĂŒnstlichen Intelligenz (KI), insbesondere im Maschinellem Lernen (ML). Mit Hilfe moderner neuronaler Netze sind Roboter in der Lage, Objekte in ihrer Umgebung zu erkennen und mit ihnen zu interagieren. Ein erhebliches Manko besteht jedoch darin, dass das Training dieser Objekterkennungsmodelle, in aller Regel mit einer zugrundeliegenden AbhĂ€ngig von umfangreichen DatensĂ€tzen und der VerfĂŒgbarkeit großer Datenmengen einhergeht. Dies ist insbesondere dann problematisch, wenn der konkrete Einsatzort des Roboters und die Umgebung, einschließlich der darin befindlichen Objekte, nicht im Voraus bekannt sind. Die breite und stĂ€ndig wachsende Palette von Objekten macht es dabei praktisch unmöglich, das gesamte Spektrum an existierenden Objekten allein mit bereits zuvor erstellten DatensĂ€tzen vollstĂ€ndig abzudecken. Das Ziel dieser Dissertation war es, einem Roboter unbekannte Objekte mit Hilfe von Human-Robot Interaction (HRI) beizubringen, um ihn von seiner AbhĂ€ngigkeit von Daten sowie den EinschrĂ€nkungen durch vordefinierte Szenarien zu befreien. Die Synergie von Eye Tracking und Augmented Reality (AR) ermöglichte es dem als Lehrer fungierenden Menschen, mit dem Roboter zu kommunizieren und ihn mittels des menschlichen Blickes auf Objekte hinzuweisen. Dieser holistische Ansatz ermöglichte die Konzeption eines multimodalen HRI-Systems, durch das der Roboter Objekte identifizieren und dreidimensional segmentieren konnte, obwohl sie ihm zu diesem Zeitpunkt noch unbekannt waren, um sie anschließend aus unterschiedlichen Blickwinkeln eigenstĂ€ndig zu inspizieren. Anhand der Klasseninformationen, die ihm der Mensch mitteilte, war der Roboter daraufhin in der Lage, die entsprechenden Objekte zu erlernen und spĂ€ter wiederzuerkennen. Mit dem Wissen, das dem Roboter durch diesen auf HRI basierenden Lehrvorgang beigebracht worden war, war dessen FĂ€higkeit Objekte zu erkennen vergleichbar mit den FĂ€higkeiten modernster Objektdetektoren, die auf umfangreichen DatensĂ€tzen trainiert worden waren. Dabei war der Roboter jedoch nicht auf vordefinierte Klassen beschrĂ€nkt, was seine Vielseitigkeit und AnpassungsfĂ€higkeit unter Beweis stellte. Die im Rahmen dieser Dissertation durchgefĂŒhrte Forschung leistete bedeutende BeitrĂ€ge an der Schnittstelle von Machine Learning (ML), AR, Eye Tracking und Robotik. Diese Erkenntnisse tragen nicht nur zum besseren VerstĂ€ndnis der genannten Felder bei, sondern ebnen auch den Weg fĂŒr weitere interdisziplinĂ€re Forschung. Die in dieser Dissertation enthalten wissenschaftlichen Artikel wurden auf hochrangigen Konferenzen in den Bereichen Robotik, Eye Tracking und HRI veröffentlicht.Robots are becoming increasingly popular in a wide range of environments due to their exceptional work capacity, precision, efficiency, and scalability. This development has been further encouraged by advances in Artificial Intelligence (AI), particularly Machine Learning (ML). By employing sophisticated neural networks, robots are given the ability to detect and interact with objects in their vicinity. However, a significant drawback arises from the underlying dependency on extensive datasets and the availability of substantial amounts of training data for these object detection models. This issue becomes particularly problematic when the specific deployment location of the robot and the surroundings, including the objects within it, are not known in advance. The vast and ever-expanding array of objects makes it virtually impossible to comprehensively cover the entire spectrum of existing objects using preexisting datasets alone. The goal of this dissertation was to teach a robot unknown objects in the context of Human-Robot Interaction (HRI) in order to liberate it from its data dependency, unleashing it from predefined scenarios. In this context, the combination of eye tracking and Augmented Reality (AR) created a powerful synergy that empowered the human teacher to seamlessly communicate with the robot and effortlessly point out objects by means of human gaze. This holistic approach led to the development of a multimodal HRI system that enabled the robot to identify and visually segment the Objects of Interest (OOIs) in three-dimensional space, even though they were initially unknown to it, and then examine them autonomously from different angles. Through the class information provided by the human, the robot was able to learn the objects and redetect them at a later stage. Due to the knowledge gained from this HRI based teaching process, the robot’s object detection capabilities exhibited comparable performance to state-of-the-art object detectors trained on extensive datasets, without being restricted to predefined classes, showcasing its versatility and adaptability. The research conducted within the scope of this dissertation made significant contributions at the intersection of ML, AR, eye tracking, and robotics. These findings not only enhance the understanding of these fields, but also pave the way for further interdisciplinary research. The scientific articles included in this dissertation have been published at high-impact conferences in the fields of robotics, eye tracking, and HRI

    The role of semaphorins in response to injury in C. elegans neurons

    Get PDF
    When neural tissue is injured by trauma, delicate neuronal processes such as axons and dendrites are prone to lesion damage and often disconnect. The molecular, cellular, and circuit mechanisms that underlie the regrowth and reconnection of these processes and the recovery of behavior are major challenges in the fields of neuroscience, regeneration, and resilience. At the molecular and cellular levels, signaling pathways that mediate neuronal growth cone guidance during development can play a role in neuronal regeneration and recovery from injury. One family of signaling proteins involved in this process comprises the highly conserved semaphorins and their receptors, the plexins. Across various species, from C. elegans to humans, semaphorins and plexins are crucial for axon pathfinding and synapse formation during development. In the mammalian nervous system, the semaphorin signaling system is comprised of more than 20 semaphorins and 9 plexins, whereas the C. elegans genome only encodes 3 semaphorins and 2 plexin receptors. Among them, the transmembrane semaphorins, SMP-1 and SMP-2, signal through their receptor PLX-1, while the secreted semaphorin MAB-20, signals through PLX-2. This dissertation explores the role of semaphorin signaling in neuroregeneration in vivo, by making use of the experimental advantages of Caenorhabditis elegans. Importantly, this versatile model animal has the natural ability to regenerate neuronal processes after injury and optic methods were developed to precisely disconnect single neurites in otherwise intact animals using laser microsurgery. Moreover, the semaphorin system is relatively simple and genetically amenable, and transgenic, microscopy and behavior analysis methods are well established. The development and assessment of a new laser microsurgery system as part of this thesis allowed reliable and accurate disconnection of identifiable axons and dendrites. The elucidated expression patterns and involvement of C. elegans semaphorins in neural regeneration have shed significant light with regard to the role this pathway plays in C. elegans regeneration and added to the field of knowledge of neural regeneration research. The findings reveal that regrowth and reconnection are more prevalent in the absence of both plexin receptors and the secreted semaphorin MAB-20. This suggests that the semaphorin signaling in this system restricts neural growth, possibly to prevent aberrant reconnection. The membrane-bound SMP-1 and SMP-2 might have a redundant role, signaling through PLX-1. These results align with the inhibitory effects of semaphorin signaling on axonal growth and guidance during development in the mammalian system. Therefore, secreted and membrane-bound semaphorin signaling pathways restrict regeneration using distinct processes, likely involving spatial specificity and recurrent signals. Findings such as the ones presented in this thesis delve deeper into the mechanisms and factors involved in promoting regeneration and aid to uncover valuable insights that could assist in overcoming the challenges faced by regenerative medicine in treating central nervous system injuries and disorders

    Vision-and-Force-Based Compliance Control for a Posterior Segment Ophthalmic Surgical Robot

    Get PDF
    In ophthalmic surgery, particularly in procedures involving the posterior segment, clinicians face significant challenges in maintaining precise control of hand-held instruments without damaging the fundus tissue. Typical targets of this type of surgery are the internal limiting membrane (ILM) and the epiretinal membrane (ERM) which have an average thickness of only 60 ÎŒm and 2 ÎŒm , respectively, making it challenging, even for experienced clinicians utilising dedicated ophthalmic surgical robots, to peel these delicate membranes successfully without damaging the healthy tissue. Minimal intra-operative motion errors when driving both hand-held and robotic-assisted surgical tools may result in significant stress on the delicate tissue of the fundus, potentially causing irreversible damage to the eye. To address these issues, this work proposes an intra-operative vision-and-force-based compliance control method for a posterior segment ophthalmic surgical robot. This method aims to achieve compliance control of the surgical instrument in contact with the tissue to minimise the risk of tissue damage. In this work we demonstrate that we can achieve a maximum motion error for the end effector (EE) of our ophthalmic robot of just 8 ÎŒm , resulting in a 64 % increase in motion accuracy compared to our previous work where the system was firstly introduced. The results of the proposed compliance control demonstrate consistent performance in the force range of 40 mN during membrane tearing

    OPTICAL COHERENCE TOMOGRAPHY OPHTHALMIC SURGICAL GUIDANCE

    Get PDF
    Optical coherence tomography (OCT) performs high-resolution cross-sectional and volumetric tissue imaging in situ through the combination of confocal gating, coherence gating, and polarization gating. Because it is noninvasive, OCT has been used in multiple clinical applications such as tissue pathology assessment and interventional procedure guidance. Moreover, OCT can perform functional measurements such as phase-sensitive measurement of blood flow and polarization-sensitive measurement of tissue birefringence. These features made OCT one of the most widely used imaging systems in ophthalmology. In this thesis, we present several novel OCT methods developed for microsurgery guidance and OCT image analysis. The thesis mainly consists of five parts, which are shown as follows. First, we present a BC-mode OCT image visualization method for microsurgery guidance, where multiple sparsely sampled B-scans are combined to generate a single cross-sectional image with an enhanced instrument and tissue layer visibility and reduced shadowing artifacts. The performance of the proposed method is demonstrated by guiding a 30-gauge needle into an ex-vivo human cornea. Second, we present a microscope-integrated OCT guided robotic subretinal injection method. A workflow is designed for accurate and stable robotic needle navigation. The performance of the proposed method is demonstrated on ex-vivo porcine eye subretinal injection. Third, we present optical flow OCT technique that quantifies accurate velocity fields. The accuracy of the proposed method is verified through phantom flow experiments by using a diluted milk powder solution as the scattering medium, in both cases of advective flow and turbulent flow. Fourth, we present a wrapped Gaussian mixture model to stabilize the phase of swept source OCT systems. A closed-form iteration solution is derived using the expectation-maximization algorithm. The performance of the proposed method is demonstrated through ex-vivo, in-vivo, and flow phantom experiments. The results show its robustness in different application scenarios. Fifth, we present a numerical landmark localization algorithm based on a convolutional neural network and a conditional random field. The robustness of the proposed method is demonstrated through ex-vivo porcine intestine landmark localization experiments

    Hierarchical, informed and robust machine learning for surgical tool management

    Get PDF
    This thesis focuses on the development of a computer vision and deep learning based system for the intelligent management of surgical tools. The work accomplished included the development of a new dataset, creation of state of the art techniques to cope with volume, variety and vision problems, and designing or adapting algorithms to address specific surgical tool recognition issues. The system was trained to cope with a wide variety of tools, with very subtle differences in shapes, and was designed to work with high volumes, as well as varying illuminations and backgrounds. Methodology that was adopted in this thesis included the creation of a surgical tool image dataset and development of a surgical tool attribute matrix or knowledge-base. This was significant because there are no large scale publicly available surgical tool datasets, nor are there established annotations or datasets of textual descriptions of surgical tools that can be used for machine learning. The work resulted in the development of a new hierarchical architecture for multi-level predictions at surgical speciality, pack, set and tool level. Additional work evaluated the use of synthetic data to improve robustness of the CNN, and the infusion of knowledge to improve predictive performance

    Augmented reality (AR) for surgical robotic and autonomous systems: State of the art, challenges, and solutions

    Get PDF
    Despite the substantial progress achieved in the development and integration of augmented reality (AR) in surgical robotic and autonomous systems (RAS), the center of focus in most devices remains on improving end-effector dexterity and precision, as well as improved access to minimally invasive surgeries. This paper aims to provide a systematic review of different types of state-of-the-art surgical robotic platforms while identifying areas for technological improvement. We associate specific control features, such as haptic feedback, sensory stimuli, and human-robot collaboration, with AR technology to perform complex surgical interventions for increased user perception of the augmented world. Current researchers in the field have, for long, faced innumerable issues with low accuracy in tool placement around complex trajectories, pose estimation, and difficulty in depth perception during two-dimensional medical imaging. A number of robots described in this review, such as Novarad and SpineAssist, are analyzed in terms of their hardware features, computer vision systems (such as deep learning algorithms), and the clinical relevance of the literature. We attempt to outline the shortcomings in current optimization algorithms for surgical robots (such as YOLO and LTSM) whilst providing mitigating solutions to internal tool-to-organ collision detection and image reconstruction. The accuracy of results in robot end-effector collisions and reduced occlusion remain promising within the scope of our research, validating the propositions made for the surgical clearance of ever-expanding AR technology in the future

    Examination tools for the endoscopic evaluation of the laryngeal adductor reflex

    Get PDF
    Der gesunde, menschliche Kehlkopf schĂŒtzt die tieferen Atemwege durch reflexhafte Mechanismen vor dem Eindringen von Partikeln, der sogenannten Aspiration. Einer dieser Mechanismen ist der laryngeale Adduktionsreflex (LAR), der eine rasche ZusammenfĂŒhrung der Stimmlippen bewirkt. Störungen des LAR können zu einer erhöhten Aspirationswahrscheinlichkeit fĂŒhren – ein Risikofaktor fĂŒr eine potentiell lebensbedrohliche LungenentzĂŒndung. Ein Routinescreening des LAR bei Verdacht auf einen pathologischen Reflexablauf ist daher medizinisch sinnvoll. Bisherige LAR-Evaluationsverfahren beruhen jedoch auf invasiven, nutzerabhĂ€ngigen und/oder ungezielten Methoden. Die Reflexperformance wird bislang zudem hauptsĂ€chlich qualitativ bewertet. Zur Reduktion der genannten Nachteile wurde an der Medizinischen Hochschule Hannover ein alternatives Verfahren entwickelt und initial erprobt. Dieser Microdroplet Impulse Testing of the LAR (MIT-LAR) genannte Ansatz beruht auf dem Beschuss der Larynxschleimhaut mit einem Tröpfchen. Durch Nutzung eines Hochgeschwindigkeitslaryngoskopsystems und manuelle Auswertung der gewonnenen Bildsequenzen konnte die LAR-Latenz bei Testpersonen mit hoher zeitlicher Auflösung gemessen werden. Obgleich dieses MIT-LAR-System einen Fortschritt gegenĂŒber vorherigen Verfahren darstellt, weist es hinsichtlich der Reproduzierbarkeit der LAR-Auslösung sowie hinsichtlich der ObjektivitĂ€t der optischen LAR-Analyse weiteres Optimierungspotential auf. Sowohl die tropfenvermittelte Stimulation als auch die optische Analyse des LAR werden in der vorliegenden, interdisziplinĂ€ren Arbeit adressiert: Ein neuartiger Tropfenapplikator ermöglicht die Bildung eines stabilen Stimulationströpfchens mit variabler MĂŒndungsenergie. Eine histologische Analyse des LĂ€sionspotentials an Schweinekehlköpfen ergibt keinen Hinweis auf GewebeschĂ€den. Zwei stereoskopische Hochgeschwindigkeitslaryngoskope werden konzipiert und aufgebaut. In Kombination mit dem Tropfenapplikator und einem Algorithmus zur Approximation der Tropfenflugbahn ermöglichen diese die Vorhersage des Tropfenaufprallortes. Bei Verwendung eines stablinsen- bzw. bildleiterbasierten Systems werden im Labor Vorhersagefehler von (0,9 ± 0,6) mm bzw. (1,3 ± 0,8) mm gemessen. Abschließend wird ein Verfahren zur automatisierten Analyse von MIT-LAR-Sequenzen entwickelt und an einem Datensatz erprobt. Dies fĂŒhrt zur erstmaligen, computergestĂŒtzten Messung der Stimmlippen-Winkelgeschwindigkeit wĂ€hrend der Adduktionsphase des menschlichen LAR. Im Fall einer vollstĂ€ndigen bzw.~unvollstĂ€ndigen Adduktion werden Werte von (891 ± 516) °/s bzw. (421 ± 221) °/s erhalten. Dies stellt eine Erweiterung des medizinischen Wissensstandes dar.Several reflexive mechanisms in the human larynx protect the deeper respiratory tract from the intrusion of foreign particles, the so-called aspiration. The laryngeal adductor reflex (LAR), which leads to a rapid closure of the glottis, is one of these mechanisms. In consequence, disturbances of the LAR can lead to aspiration – a risk factor for potentially fatal pneumonia. Therefore, a routine screening of the LAR is highly beneficial in cases where a pathological reflex phenotype is suspected. Current LAR evaluation approaches rely on invasive, user-dependent, and/or untargeted methods. Moreover, the reflex performance is currently mainly being assessed qualitatively. To mitigate these disadvantages, an alternative method has recently been developed and initially tested at Hannover Medical School. This method, referred to as Microdroplet Impulse Testing of the LAR (MIT-LAR), is based on impacting the laryngeal mucosa with a droplet. By using a high-speed laryngoscope, combined with a manual analysis of the recorded high-speed sequence showing the reflexive response, the LAR onset latency could be measured at a high temporal resolution. Although the MIT-LAR system represents a technological progress with respect to prior methods, it still offers further potential for development regarding the reproducibility of LAR stimulation and the objectivity of LAR evaluation. Both droplet-based LAR stimulation and optical LAR analysis are in the focus of the present, interdisciplinary work: A novel droplet applicator module enables stabilization of droplet formation and droplet muzzle energy control. A histological analysis of the droplet’s lesion potential on porcine larynges does not yield any sign of tissue damage. Two stereoscopic high-speed laryngoscopes are designed and set up. In combination with the droplet applicator and an algorithm for the approximation of the droplet trajectory, this enables the prediction of the droplet impact site. The prediction error of both laryngoscopic systems is evaluated in a laboratory setting. A value of (0.9 0.6)mm is measured using a rod lens-based system; a fiber-based optics yields a value of (1.3 0.8)mm. Finally, a method for the automatic analysis of MIT-LAR sequences is developed and tested on a data set. This leads to the first computer-assisted measurement of the angular velocity of the vocal folds during the adduction phase of the human LAR. When complete/incomplete adduction is achieved, values of (891 516) ° s−1 and (421 221) ° s−1 are obtained, respectively. This constitutes an expansion of the state of medical knowledg

    Imaging fascicular organisation in mammalian vagus nerve for selective VNS

    Get PDF
    Nerves contain a large number of nerve fibres, or axons, organised into bundles known as fascicles. Despite the somatic nervous system being well understood, the organisation of the fascicles within the nerves of the autonomic nervous system remains almost completely unknown. The new field of bioelectronics medicine, Electroceuticals, involves the electrical stimulation of nerves to treat diseases instead of administering drugs or performing complex surgical procedures. Of particular interest is the vagus nerve, a prime target for intervention due to its afferent and efferent innervation to the heart, lungs and majority of the visceral organs. Vagus nerve stimulation (VNS) is a promising therapy for treatment of various conditions resistant to standard therapeutics. However, due to the unknown anatomy, the whole nerve is stimulated which leads to unwanted off-target effects. Electrical Impedance Tomography (EIT) is a non-invasive medical imaging technique in which the impedance of a part of the body is inferred from electrode measurements and used to form a tomographic image of that part. Micro-computed tomography (microCT) is an ex vivo method that has the potential to allow for imaging and tracing of fascicles within experimental models and facilitate the development of a fascicular map. Additionally, it could validate the in vivo technique of EIT. The aim of this thesis was to develop and optimise the microCT imaging method for imaging the fascicles within the nerve and to determine the fascicular organisation of the vagus nerve, ultimately allowing for selective VNS. Understanding and imaging the fascicular anatomy of nerves will not only allow for selective VNS and the improvement of its therapeutic efficacy but could also be integrated into the study on all peripheral nerves for peripheral nerve repair, microsurgery and improving the implementation of nerve guidance conduits. Chapter 1 provides an introduction to vagus nerve anatomy and the principles of microCT, neuronal tracing and EIT. Chapter 2 describes the optimisation of microCT for imaging the fascicular anatomy of peripheral nerves in the experimental rat sciatic and pig vagus nerve models, including the development of pre-processing methods and scanning parameters. Cross-validation of this optimised microCT method, neuronal tracing and EIT in the rat sciatic nerve was detailed in Chapter 3. Chapter 4 describes the study with microCT with tracing, EIT and selective stimulation in pigs, a model for human nerves. The microCT tracing approach was then extended into the subdiaphragmatic branches of the vagus nerves, detailed in Chapter 5. The ultimate goal of human vagus nerve tracing was preliminarily performed and described in Chapter 6. Chapter 7 concludes the work and describes future work. Lastly, Appendix 1 (Chapter 8) is a mini review on the application of selective vagus nerve stimulation to treat acute respiratory distress syndrome and Appendix 2 is morphological data corresponding to Chapter 4

    Tracking and Mapping in Medical Computer Vision: A Review

    Full text link
    As computer vision algorithms are becoming more capable, their applications in clinical systems will become more pervasive. These applications include diagnostics such as colonoscopy and bronchoscopy, guiding biopsies and minimally invasive interventions and surgery, automating instrument motion and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing and applying algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. We then review datasets provided in the field and the clinical needs therein. Then, we delve in depth into the algorithmic side, and summarize recent developments, which should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. Finally, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications in the field. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.Comment: 31 pages, 17 figure

    Bionic Lid Implant for Natural Closure (BLINC)

    Get PDF
    Facial nerve palsy (FNP) leads to an inability to blink. The exposed eye is at risk of developing corneal keratopathy and currently there is a lack of solution to active eye closure that is immediate and reliable. Bionic Lid Implant for Natural Closure (BLINC) proposes the use of an implantable actuator combined with the effects of an eyelid sling for dynamic eye closure. The aims of this thesis are to 1) explore the clinical need for BLINC, 2) describe the BLINC technology, and 3) present the results of its application in cadaveric and live models. Methods The aims of this project are addressed in three parts. In part one, the current therapies addressing key clinical end points in FNP from an ocular perspective and the setting where BLINC may first be used are explored. In part two the science behind BLINC is outlined. Finally in part three application of BLINC in cadaveric and live models are studied followed by a discussion on future steps preceding a pilot study in humans. Results Patients with FNP consistently identify issues related to the eye a primary concern. Current reanimation strategies offer the possibility of dynamic eye closure but the results are delayed and often unpredictable. BLINC reliably achieves active eye closure in cadaveric models by means of a wireless-powered, implantable electromagnetic actuator in conjunction with an eyelid sling. BLINC closes the eye in a similar fashion to natural closure for a symmetrical blink in FNP. Successful application of an inactive device in its complete form is achieved in a live animal without significant morbidity. Conclusion BLINC offers the possibility of restoring active eye closure with use of an implantable actuator. The concept has been successfully demonstrated in cadaveric models with successful device implantation in a live model. Future live trials are needed to address the remaining biocompatibility issues in preparation for human application
    • 

    corecore