3D landmark detection for augmented reality based otologic procedures

International audienceEar consists of the smallest bones in the human body and does not contain significant amount of distinct landmark points that may be used to register a preoperative CT-scan with the surgical video in an augmented reality framework. Learning based algorithms may be used to help the surgeons to identify landmark points. This paper presents a convolutional neural network approach to landmark detection in preoperative ear CT images and then discusses an augmented reality system that can be used to visualize the cochlear axis on an otologic surgical video

Effect of Rotating Auditory Scene on Postural Control in Normal Subjects, Patients With Bilateral Vestibulopathy, Unilateral, or Bilateral Cochlear Implants

Objective: The aim of this study was to investigate the impact of a rotating sound stimulation on the postural performances in normal subjects, patients with bilateral vestibulopathy (BVP), unilateral (UCI), and bilateral (BCI) cochlear implantees.Materials and Methods: Sixty-nine adults were included (32 women and 37 men) in a multicenter prospective study. The group included 37 healthy subjects, 10 BVP, 15 UCI, and 7 BCI patients. The average of age was 47 ± 2.0 (range: 23–82). In addition to a complete audiovestibular work up, a dynamic posturography (Multitest Framiral, Grasse) was conducted in silence and with a rotating cocktail party sound delivered by headphone. The center of pressure excursion surface (COPS), sensory preferences, as well as fractal, diffusion, and wavelet analysis of stabilometry were collected.Results: The rotating sound seemed to influenced balance in all subgroups except in controls. COPS increased with sound in the BVP and BCI groups in closed eyes and sway-referenced condition indicating a destabilizing effect while it decreased in UCI in the same condition suggesting stabilization (p < 0.05, linear mixed model corrected for age, n = 69). BVP had higher proprioceptive preferences, BCI had higher vestibular and visual preferences, and UCI had only higher vestibular preferences than controls. Sensory preferences were not altered by rotating sound.Conclusions: The rotating sound destabilized BVP and BCI patients with binaural hearing while it stabilized UCI patients with monaural hearing and no sound rotation effect. This difference suggests that binaural auditory cues are exploited in BCI patients for their balance

La réalité augmentée dans la chirurgie otologique mini-invasive et l’implantation cochléaire transmodiolaire

Optimal exposure is necessary for minimally invasive otologic surgery. Augmented reality allows to enrich the real environment by adding data without replacing it by a virtual environment and to view this information on a single screen by combining the vision of the operating field and the imaging. This technique is very little described in otology in the literature. This project aims to develop augmented reality solutions for otological surgeries and transmodiolar implantation. To prepare this augmented reality project, we studied the visualization of the middle ear by a virtual endoscope based on preoperative high resolution CT-scans. Then, we worked on the application of augmented reality to transtympanic otologic surgery in order to design a minimally invasive system based solely on vision algorithms thanks to the superposition of images from pre-operative CT-scans with those recorded in real time corresponding to the surgeon's intraoperative vision without the need for traditional tracking systems (magnetic and/or optical). This allowed us to visualize the anatomical structures of the tympanic cavity and to mobilize a micro-instrument in the tympanic cavity without lifting the tympano-meatal flap. At the same time, we developed the augmented reality system applied to transmodiolar implantation that allowed visualization of the entry point and axis of the modiolus on the surgical video which are not visible directly by vision of the otologic surgeon. During this work, we were able to show that augmented reality has an interest in the development of minimally invasive otologic surgery, both in the approach and in the surgical technique, and opens perspectives for future work.Une exposition optimale est le prérequis pour la chirurgie otologique mini-invasive. La réalité augmentée permet d’enrichir l’environnement réel en y ajoutant des données sans le remplacer par un environnement virtuel et de visionner ces informations sur un seul écran en combinant la vision du champ opératoire et de l’imagerie. Cette technique est très peu développée en otologie. Ce projet vise à développer des solutions de réalité augmentée pour les chirurgies otologiques et l’implantation transmodiolaire. Pour préparer ce projet de réalité augmentée , nous avons étudié la visualisation de l’oreille moyenne par un endoscope virtuel basé sur des scanners de haute résolution. Puis, nous avons travaillé sur l’application de la réalité augmentée à la chirurgie otologique trans-tympanique afin de concevoir un système mini-invasif uniquement basé sur des algorithmes de vision grâce à la superposition des images des scanners pré-opératoires à celles enregistrées en temps réel correspondant à la vision per-opératoire du chirurgien sans nécessité de systèmes traditionnels de suivi (magnétique et/ou optique). Cela permet de visualiser les structures anatomiques de la caisse tympanique et de mobiliser un micro-instrument dans celle-ci sans lever le lambeau tympano-méatal. Parallèlement à cela, nous avons développé un système de réalité augmentée appliqué à l’implantation transmodiolaire qui permettait de visualiser le point d’entrée et l’axe du modiolus sur la vidéo chirurgicale. Au cours de ces travaux, nous avons pu montrer que la réalité augmentée avait un intérêt dans le développement de la chirurgie otologique mini-invasive que ce soit lors de la voie d’abord ou de la technique chirurgicale et ouvre des perspectives pour des travaux futurs

Augmented Reality in minimally invasive otologic surgery and transmodiolar cochlear implantation

Une exposition optimale est le prérequis pour la chirurgie otologique mini-invasive. La réalité augmentée permet d’enrichir l’environnement réel en y ajoutant des données sans le remplacer par un environnement virtuel et de visionner ces informations sur un seul écran en combinant la vision du champ opératoire et de l’imagerie. Cette technique est très peu développée en otologie. Ce projet vise à développer des solutions de réalité augmentée pour les chirurgies otologiques et l’implantation transmodiolaire. Pour préparer ce projet de réalité augmentée , nous avons étudié la visualisation de l’oreille moyenne par un endoscope virtuel basé sur des scanners de haute résolution. Puis, nous avons travaillé sur l’application de la réalité augmentée à la chirurgie otologique trans-tympanique afin de concevoir un système mini-invasif uniquement basé sur des algorithmes de vision grâce à la superposition des images des scanners pré-opératoires à celles enregistrées en temps réel correspondant à la vision per-opératoire du chirurgien sans nécessité de systèmes traditionnels de suivi (magnétique et/ou optique). Cela permet de visualiser les structures anatomiques de la caisse tympanique et de mobiliser un micro-instrument dans celle-ci sans lever le lambeau tympano-méatal. Parallèlement à cela, nous avons développé un système de réalité augmentée appliqué à l’implantation transmodiolaire qui permettait de visualiser le point d’entrée et l’axe du modiolus sur la vidéo chirurgicale. Au cours de ces travaux, nous avons pu montrer que la réalité augmentée avait un intérêt dans le développement de la chirurgie otologique mini-invasive que ce soit lors de la voie d’abord ou de la technique chirurgicale et ouvre des perspectives pour des travaux futurs.Optimal exposure is necessary for minimally invasive otologic surgery. Augmented reality allows to enrich the real environment by adding data without replacing it by a virtual environment and to view this information on a single screen by combining the vision of the operating field and the imaging. This technique is very little described in otology in the literature. This project aims to develop augmented reality solutions for otological surgeries and transmodiolar implantation. To prepare this augmented reality project, we studied the visualization of the middle ear by a virtual endoscope based on preoperative high resolution CT-scans. Then, we worked on the application of augmented reality to transtympanic otologic surgery in order to design a minimally invasive system based solely on vision algorithms thanks to the superposition of images from pre-operative CT-scans with those recorded in real time corresponding to the surgeon's intraoperative vision without the need for traditional tracking systems (magnetic and/or optical). This allowed us to visualize the anatomical structures of the tympanic cavity and to mobilize a micro-instrument in the tympanic cavity without lifting the tympano-meatal flap. At the same time, we developed the augmented reality system applied to transmodiolar implantation that allowed visualization of the entry point and axis of the modiolus on the surgical video which are not visible directly by vision of the otologic surgeon. During this work, we were able to show that augmented reality has an interest in the development of minimally invasive otologic surgery, both in the approach and in the surgical technique, and opens perspectives for future work

La réalité augmentée dans la chirurgie otologique mini-invasive et l’implantation cochléaire transmodiolaire

Optimal exposure is necessary for minimally invasive otologic surgery. Augmented reality allows to enrich the real environment by adding data without replacing it by a virtual environment and to view this information on a single screen by combining the vision of the operating field and the imaging. This technique is very little described in otology in the literature. This project aims to develop augmented reality solutions for otological surgeries and transmodiolar implantation. To prepare this augmented reality project, we studied the visualization of the middle ear by a virtual endoscope based on preoperative high resolution CT-scans. Then, we worked on the application of augmented reality to transtympanic otologic surgery in order to design a minimally invasive system based solely on vision algorithms thanks to the superposition of images from pre-operative CT-scans with those recorded in real time corresponding to the surgeon's intraoperative vision without the need for traditional tracking systems (magnetic and/or optical). This allowed us to visualize the anatomical structures of the tympanic cavity and to mobilize a micro-instrument in the tympanic cavity without lifting the tympano-meatal flap. At the same time, we developed the augmented reality system applied to transmodiolar implantation that allowed visualization of the entry point and axis of the modiolus on the surgical video which are not visible directly by vision of the otologic surgeon. During this work, we were able to show that augmented reality has an interest in the development of minimally invasive otologic surgery, both in the approach and in the surgical technique, and opens perspectives for future work.Une exposition optimale est le prérequis pour la chirurgie otologique mini-invasive. La réalité augmentée permet d’enrichir l’environnement réel en y ajoutant des données sans le remplacer par un environnement virtuel et de visionner ces informations sur un seul écran en combinant la vision du champ opératoire et de l’imagerie. Cette technique est très peu développée en otologie. Ce projet vise à développer des solutions de réalité augmentée pour les chirurgies otologiques et l’implantation transmodiolaire. Pour préparer ce projet de réalité augmentée , nous avons étudié la visualisation de l’oreille moyenne par un endoscope virtuel basé sur des scanners de haute résolution. Puis, nous avons travaillé sur l’application de la réalité augmentée à la chirurgie otologique trans-tympanique afin de concevoir un système mini-invasif uniquement basé sur des algorithmes de vision grâce à la superposition des images des scanners pré-opératoires à celles enregistrées en temps réel correspondant à la vision per-opératoire du chirurgien sans nécessité de systèmes traditionnels de suivi (magnétique et/ou optique). Cela permet de visualiser les structures anatomiques de la caisse tympanique et de mobiliser un micro-instrument dans celle-ci sans lever le lambeau tympano-méatal. Parallèlement à cela, nous avons développé un système de réalité augmentée appliqué à l’implantation transmodiolaire qui permettait de visualiser le point d’entrée et l’axe du modiolus sur la vidéo chirurgicale. Au cours de ces travaux, nous avons pu montrer que la réalité augmentée avait un intérêt dans le développement de la chirurgie otologique mini-invasive que ce soit lors de la voie d’abord ou de la technique chirurgicale et ouvre des perspectives pour des travaux futurs

Vision-Based Augmented Reality System for Middle Ear Surgery: Evaluation in Operating Room Environment

International audienc

Augmented Reality Based Transmodiolar Cochlear Implantation

International audienc

Minimally Invasive Cochlear Implantation Assisted by Intraoperative CT Scan Combined to Neuronavigation

International audienc

Contribution of Augmented Reality to Minimally Invasive Computer-Assisted Cranial Base Surgery

International audienceCranial base procedures involve manipulation of small, delicate and complex structures in the fields of otology, rhinology, neurosurgery and maxillofacial surgery. Critical nerves and blood vessels are in close proximity of these structures. Augmented reality is an emerging technology that can revolutionize the cranial base procedures by providing supplementary anatomical and navigational information unified on a single display. However, the awareness and acceptance of possibilities of augmented reality systems in cranial base domain is fairly low. This article aims at evaluating the usefulness of augmented reality systems in cranial base surgeries and highlights the challenges that current technology faces and their potential solutions. A technical perspective about different strategies employed in development of an augmented realty system is also presented. The current trend suggests an increase in interest towards augmented reality systems that may lead to safer and cost-effective procedures. However, several issues need to be addressed before it can be widely integrated into routine practice

Augmented Reality of the Middle Ear Combining Otoendoscopy and Temporal Bone Computed Tomography

International audienceHYPOTHESIS:Augmented reality (AR) may enhance otologic procedures by providing sub-millimetric accuracy and allowing the unification of information in a single screen.BACKGROUND:Several issues related to otologic procedures can be addressed through an AR system by providing sub-millimetric precision, supplying a global view of the middle ear cleft, and advantageously unifying the information in a single screen. The AR system is obtained by combining otoendoscopy with temporal bone computer tomography (CT).METHODS:Four human temporal bone specimens were explored by high-resolution CT-scan and dynamic otoendoscopy with video recordings. The initialization of the system consisted of a semi-automatic registration between the otoendoscopic video and the 3D CT-scan reconstruction of the middle ear. Endoscope movements were estimated by several computer vision techniques (feature detectors/descriptors and optical flow) and used to warp the CT-scan to keep the correspondence with the otoendoscopic video.RESULTS:The system maintained synchronization between the CT-scan image and the otoendoscopic video in all experiments during slow and rapid (5-10 mm/s) endoscope movements. Among tested algorithms, two feature-based methods, scale-invariant feature transform (SIFT); and speeded up robust features (SURF), provided sub-millimeter mean tracking errors (0.38 ± 0.53 mm and 0.20 ± 0.16 mm, respectively) and an adequate image refresh rate (11 and 17 frames per second, respectively) after 2 minutes of procedure with continuous endoscope movements.CONCLUSION:A precise augmented reality combining video and 3D CT-scan data can be applied to otoendoscopy without the use of conventional neuronavigation tracking thanks to computer vision algorithms