Retrieval and Registration of Long-Range Overlapping Frames for Scalable Mosaicking of In Vivo Fetoscopy

Purpose: The standard clinical treatment of Twin-to-Twin Transfusion Syndrome consists in the photo-coagulation of undesired anastomoses located on the placenta which are responsible to a blood transfer between the two twins. While being the standard of care procedure, fetoscopy suffers from a limited field-of-view of the placenta resulting in missed anastomoses. To facilitate the task of the clinician, building a global map of the placenta providing a larger overview of the vascular network is highly desired. Methods: To overcome the challenging visual conditions inherent to in vivo sequences (low contrast, obstructions or presence of artifacts, among others), we propose the following contributions: (i) robust pairwise registration is achieved by aligning the orientation of the image gradients, and (ii) difficulties regarding long-range consistency (e.g. due to the presence of outliers) is tackled via a bag-of-word strategy, which identifies overlapping frames of the sequence to be registered regardless of their respective location in time. Results: In addition to visual difficulties, in vivo sequences are characterised by the intrinsic absence of gold standard. We present mosaics motivating qualitatively our methodological choices and demonstrating their promising aspect. We also demonstrate semi-quantitatively, via visual inspection of registration results, the efficacy of our registration approach in comparison to two standard baselines. Conclusion: This paper proposes the first approach for the construction of mosaics of placenta in in vivo fetoscopy sequences. Robustness to visual challenges during registration and long-range temporal consistency are proposed, offering first positive results on in vivo data for which standard mosaicking techniques are not applicable.Comment: Accepted for publication in International Journal of Computer Assisted Radiology and Surgery (IJCARS

Novel methods for subcellular in vivo imaging of the cornea with the Rostock Cornea Module 2.0

The Rostock Cornea Module transforms a confocal laser scanning ophthalmoscope into a corneal confocal laser scanning microscope. In this thesis, an improved version, the Rostock Cornea Module 2.0, and its achieved results were demonstrated. These include a concave contact cap design to attenuate eye movements to improve 3D volume reconstruction, an oscillating focal plane to improve mosaicking of the subbasal nerve plexus, the integration of simultaneous optical coherence tomography, multiwavelength corneal imaging, the clinical usage, and the automated morphological characterization

Interlaced spectrally encoded confocal scanning laser ophthalmoscopy and spectral domain optical coherence tomography

Scanning laser ophthalmoscopy (SLO) and spectral domain optical coherence tomography (SDOCT) have become essential clinical diagnostic tools in ophthalmology by allowing for video-rate noninvasive en face and depth-resolved visualization of retinal structure. Current generation multimodal imaging systems that combine both SLO and OCT as a means of image tracking remain complex in their hardware implementations. Here, we combine a spectrally encoded confocal scanning laser ophthalmoscope (SECSLO) with an ophthalmic SDOCT system. This novel implementation of an interlaced SECSLO-SDOCT system allows for video-rate SLO fundus images to be acquired alternately with high-resolution SDOCT B-scans as a means of image aiming, guidance, and registration as well as motion tracking. The system shares the illumination source, detection system, and scanning optics between both SLO and OCT as a method of providing a simple multimodal ophthalmic imaging system that can readily be implemented as a table-top or hand-held device

Robust endoscopic image mosaicking via fusion of multimodal estimation

We propose an endoscopic image mosaicking algorithm that is robust to light conditioning changes, specular reflections, and feature-less scenes. These conditions are especially common in minimally invasive surgery where the light source moves with the camera to dynamically illuminate close range scenes. This makes it difficult for a single image registration method to robustly track camera motion and then generate consistent mosaics of the expanded surgical scene across different and heterogeneous environments. Instead of relying on one specialised feature extractor or image registration method, we propose to fuse different image registration algorithms according to their uncertainties, formulating the problem as affine pose graph optimisation. This allows to combine landmarks, dense intensity registration, and learning-based approaches in a single framework. To demonstrate our application we consider deep learning-based optical flow, hand-crafted features, and intensity-based registration, however, the framework is general and could take as input other sources of motion estimation, including other sensor modalities. We validate the performance of our approach on three datasets with very different characteristics to highlighting its generalisability, demonstrating the advantages of our proposed fusion framework. While each individual registration algorithm eventually fails drastically on certain surgical scenes, the fusion approach flexibly determines which algorithms to use and in which proportion to more robustly obtain consistent mosaics

Cable-driven parallel robot for transoral laser phonosurgery

Transoral laser phonosurgery (TLP) is a common surgical procedure in otolaryngology. Currently, two techniques are commonly used: free beam and fibre delivery. For free beam delivery, in combination with laser scanning techniques, accurate laser pattern scanning can be achieved. However, a line-of-sight to the target is required. A suspension laryngoscope is adopted to create a straight working channel for the scanning laser beam, which could introduce lesions to the patient, and the manipulability and ergonomics are poor. For the fibre delivery approach, a flexible fibre is used to transmit the laser beam, and the distal tip of the laser fibre can be manipulated by a flexible robotic tool. The issues related to the limitation of the line-of-sight can be avoided. However, the laser scanning function is currently lost in this approach, and the performance is inferior to that of the laser scanning technique in the free beam approach. A novel cable-driven parallel robot (CDPR), LaryngoTORS, has been developed for TLP. By using a curved laryngeal blade, a straight suspension laryngoscope will not be necessary to use, which is expected to be less traumatic to the patient. Semi-autonomous free path scanning can be executed, and high precision and high repeatability of the free path can be achieved. The performance has been verified in various bench and ex vivo tests. The technical feasibility of the LaryngoTORS robot for TLP was considered and evaluated in this thesis. The LaryngoTORS robot has demonstrated the potential to offer an acceptable and feasible solution to be used in real-world clinical applications of TLP. Furthermore, the LaryngoTORS robot can combine with fibre-based optical biopsy techniques. Experiments of probe-based confocal laser endomicroscopy (pCLE) and hyperspectral fibre-optic sensing were performed. The LaryngoTORS robot demonstrates the potential to be utilised to apply the fibre-based optical biopsy of the larynx.Open Acces

A deep learning model to assess and enhance eye fundus image quality

Engineering aims to design, build, and implement solutions that will increase and/or improve the life quality of human beings. Likewise, from medicine, solutions are generated for the same purposes, enabling these two knowledge areas to converge for a common goal. With the thesis work “A Deep Learning Model to Assess and Enhance Eye Fundus Image Quality", a model was proposed and implement a model that allows us to evaluate and enhance the quality of fundus images, which contributes to improving the efficiency and effectiveness of a subsequent diagnosis based on these images. On the one hand, for the evaluation of these images, a model based on a lightweight convolutional neural network architecture was developed, termed as Mobile Fundus Quality Network (MFQ-Net). This model has approximately 90% fewer parameters than those of the latest generation. For its evaluation, the Kaggle public data set was used with two sets of quality annotations, binary (good and bad) and three classes (good, usable and bad) obtaining an accuracy of 0.911 and 0.856 in the binary mode and three classes respectively in the classification of the fundus image quality. On the other hand, a method was developed for eye fundus quality enhancement termed as Pix2Pix Fundus Oculi Quality Enhancement (P2P-FOQE). This method is based on three stages which are; pre-enhancement: for color adjustment, enhancement: with a Pix2Pix network (which is a Conditional Generative Adversarial Network) as the core of the method and post-enhancement: which is a CLAHE adjustment for contrast and detail enhancement. This method was evaluated on a subset of quality annotations for the Kaggle public database which was re-classified for three categories (good, usable, and poor) by a specialist from the Fundación Oftalmolóica Nacional. With this method, the quality of these images for the good class was improved by 72.33%. Likewise, the image quality improved from the bad class to the usable class, and from the bad class to the good class by 56.21% and 29.49% respectively.La ingeniería busca diseñar, construir e implementar soluciones que permitan aumentar y/o mejorar la calidad de vida de los seres humanos. Igualmente, desde la medicina son generadas soluciones con los mismos fines, posibilitando que estas dos áreas del conocimiento convergan por un bien común. Con el trabajo de tesis “A Deep Learning Model to Assess and Enhance Eye Fundus Image Quality”, se propuso e implementó un modelo que permite evaluar y mejorar la calidad de las imágenes de fondo de ojo, lo cual contribuye a mejorar la eficiencia y eficacia de un posterior diagnóstico basado en estas imágenes. Para la evaluación de estás imágenes, se desarrolló un modelo basado en una arquitectura de red neuronal convolucional ligera, la cual fue llamada Mobile Fundus Quality Network (MFQ-Net). Este modelo posee aproximadamente 90% menos parámetros que aquellos de última generación. Para su evaluación se utilizó el conjunto de datos públicos de Kaggle con dos sets de anotaciones de calidad, binario (buena y mala) y tres clases (buena, usable y mala) obteniendo en la tareas de clasificación de la calidad de la imagen de fondo de ojo una exactitud de 0.911 y 0.856 en la modalidad binaria y tres clases respectivamente. Por otra parte, se desarrolló un método el cual realiza una mejora de la calidad de imágenes de fondo de ojo llamado Pix2Pix Fundus Oculi Quality Enhacement (P2P-FOQE). Este método está basado en tres etapas las cuales son; premejora: para ajuste de color, mejora: con una red Pix2Pix (la cual es una Conditional Generative Adversarial Network) como núcleo del método y postmejora: la cual es un ajuste CLAHE para contraste y realce de detalles. Este método fue evaluado en un subconjunto de anotaciones de calidad para la base de datos pública de Kaggle el cual fue re clasificado por un especialista de la Fundación Oftalmológica Nacional para tres categorías (buena, usable y mala). Con este método fue mejorada la calidad de estas imágenes para la clase buena en un 72,33%. Así mismo, la calidad de imagen mejoró de la clase mala a la clase utilizable, y de la clase mala a clase buena en 56.21% y 29.49% respectivamente.Línea de investigación: Visión por computadora para análisis de imágenes médicasMaestrí

Psychiatric Case Record

Bipolar Disorder-Mania: Patient was apparently normal one-month back, Then all of a sudden he developed sleep disturbances –mainly difficult in initiation of sleep. He also started abusing his family members for unwanted things. Subsequently, he started talking excessively and irritable. Sometimes he sings film songs and dances. He used to say that God Supreme exists in himself and so he has all the powers of Almighty. With that superior power he says that he can solve all the problems in this world. He also says that he has invented herbs to keep people young. For the past one week, he talks excessively without having an hour of sleep & wanders here and there & found excessively smoking. He becomes excessively spiritual and goes to near by villages for offering prayers to God. He takes only a little food everyday and he is very much keen in personal cleanliness. Paranoid Schizophrenia: She was apparently normal 8 months back, then she developed sleep disturbances in the form of difficult in falling asleep. She was found talking & smiling to herself at night & day with mirror gazing. She started saying that her neighbour & relatives are planning to kill herself by poisoning. In this context she had frequent quarrels with them and she refused to take food prepared by her mother in law. She left the home at night without informing any one and started wandering in the road side near her home. She was complaining that she hears voices as if her neighbour & relatives were talking about her among themselves She was not doing house hold activities for past 6 months and she was not taking care of her child. Her personal hygiene was very much deteriorated slowly as she used to take bath & brush, only if she was asked to do so. She started abusing & assaulting the strangers and family members. Generalised Anxiety Disorder: Six months back he was apparently normal. He is working as a system analyst in a private bank . He had once, made a mistake in his bank work for which he was given charges by his employer, followed this event he becomes very tense and afraid whenever his boss called him. He is very cautious that he should not commit any mistakes. Even though he is not doing so, he fears that he may commit some mistake in his work. At that moment he develops palpitation, giddiness, breathlessness, excessive sweating over palms and soles. Slowly these symptoms present through out the day even when he was not in his office, and he could not control his fearfulness. For the past 6 months he didn’t sleep well. His sleep is disturbed by bad dreams. Recurrent Depressive Disorder: Patient was apparently alright 2 months back. Then she developed sleep disturbances particularly early morning awakening, she use to wake up by 3.00 am and use to brood about herself and started crying. She was not doing her domestic work as before, as she felt excess tiredness and use to take frequent rests. She developed poor communication. She had lost her interest in pleasurable activities and was not interested in watching TV, and attending family gatherings. She stayed aloof most of the time & calm, quiet and withdrawn. She was expressing her helplessness and hopelessness about the future. She started to have decline in maintaining self care. 15 days back, she frequently expressed suicidal ideas and she had attempted suicide by hanging herself and was rescued by neighbours. 5 days back, she started talking in an irrelevant manner. She was smiling to self. She was assaulting her family members. She was suspicious that her neighbour had done black magic on her and also saying that people are talking about her. She reported hearing the voice of her neighbour scolding and threatening her. Organic Brain Syndrome – Dementia: Ten months back he was apparently alright. Then his relatives noticed himself frequently misplaces things inside his home. Then he started behaving aggressively. He was beating his wife without reason. He was roaming here and there, running out of home and wandering aimlessly. He was not able to come back home when he goes out. He was brought back to home by his relatives. Slowly he developed fearfulness and tremulousness while he was staying alone. He also started saying that his family members & neighbours were talking about himself, in this context he would make frequent quarrels with them. He also started hearing voices of known male voices abusing himself in third person. He sleeps for few hour only. He is passing urine and motion inside the house. He is asking about his brother and mother-in-law who were expired long back. He behaves abnormally such as pouring water in the plate while eating. And his relatives found the symptoms were worsened by evening. All these symptoms started insidiously, increased in severity through time and attained the present state. No history of loss of appetite / crying spells / suicidal tendencies / convulsions / fever / head injury

Cellular in vivo 3D imaging of the cornea by confocal laser scanning microscopy

We present an in vivo confocal laser scanning microscopy based method for large 3D reconstruction of the cornea on a cellular level with cropped volume sizes up to 266 x 286 x 396 µm3. The microscope objective used is equipped with a piezo actuator for automated, fast and precise closed-loop focal plane control. Furthermore, we present a novel concave surface contact cap, which significantly reduces eye movements by up to 87%, hence increasing the overlapping image area of the whole stack. This increases the cuboid volume of the generated 3D reconstruction significantly. The possibility to generate oblique sections using isotropic volume stacks opens the window to slit lamp microscopy on a cellular level

Medical Image Analysis using Deep Relational Learning

In the past ten years, with the help of deep learning, especially the rapid development of deep neural networks, medical image analysis has made remarkable progress. However, how to effectively use the relational information between various tissues or organs in medical images is still a very challenging problem, and it has not been fully studied. In this thesis, we propose two novel solutions to this problem based on deep relational learning. First, we propose a context-aware fully convolutional network that effectively models implicit relation information between features to perform medical image segmentation. The network achieves the state-of-the-art segmentation results on the Multi Modal Brain Tumor Segmentation 2017 (BraTS2017) and Multi Modal Brain Tumor Segmentation 2018 (BraTS2018) data sets. Subsequently, we propose a new hierarchical homography estimation network to achieve accurate medical image mosaicing by learning the explicit spatial relationship between adjacent frames. We use the UCL Fetoscopy Placenta dataset to conduct experiments and our hierarchical homography estimation network outperforms the other state-of-the-art mosaicing methods while generating robust and meaningful mosaicing result on unseen frames.Comment: arXiv admin note: substantial text overlap with arXiv:2007.0778