Fast widefield techniques for fluorescence and phase endomicroscopy

Thesis (Ph.D.)--Boston UniversityEndomicroscopy is a recent development in biomedical optics which gives researchers and physicians microscope-resolution views of intact tissue to complement macroscopic visualization during endoscopy screening. This thesis presents HiLo endomicroscopy and oblique back-illumination endomicroscopy, fast widefield imaging techniques with fluorescence and phase contrast, respectively. Fluorescence imaging in thick tissue is often hampered by strong out-of-focus background signal. Laser scanning confocal endomicroscopy has been developed for optically-sectioned imaging free from background, but reliance on mechanical scanning fundamentally limits the frame rate and represents significant complexity and expense. HiLo is a fast, simple, widefield fluorescence imaging technique which rejects out-of-focus background signal without the need for scanning. It works by acquiring two images of the sample under uniform and structured illumination and synthesizing an optically sectioned result with real-time image processing. Oblique back-illumination microscopy (OBM) is a label-free technique which allows, for the first time, phase gradient imaging of sub-surface morphology in thick scattering tissue with a reflection geometry. OBM works by back-illuminating the sample with the oblique diffuse reflectance from light delivered via off-axis optical fibers. The use of two diametrically opposed illumination fibers allows simultaneous and independent measurement of phase gradients and absorption contrast. Video-rate single-exposure operation using wavelength multiplexing is demonstrated

Computational Optical Sectioning in Fibre Bundle Endomicroscopy

The field of fibre bundle endomicroscopy has emerged to enable real-time imaging of cellular level features in-vivo. The gold standard is confocal laserscanning, enabling optical sectioning. Point-scan confocal suffers from lower speeds, a need for complex alignment, and the added cost of a laser. This thesis presents three developments in computational optical sectioning for fibre bundle endomicroscopy.The first development is in structured illumination (SIM) endomicroscopy. Lower-cost, simplified endomicroscopes have been developed which use widefield incoherent illumination. Optical sectioning can be introduced to these systems using SIM. SIM improves imaging using spatial modulation of the focal plane and capturing a three-frame sequence. The acquired images are then numerically processed to reject out-of-focus light. This thesis reports and characterises the first high-speed SIM endomicroscope built using a miniature array ofmirrors, a digital micromirror device. The second development is automated motion compensation in SIM endomicroscopy. As a multi frame process, SIM is susceptible to motion artefacts, making the technique difficult to use in vivo and preventing the use of mosaicking to synthesise a larger effective field of view. I report and validate an automatic motion compensation technique to overcome motion artefacts and report the firstmosaics in SIM endomicroscopy.The third development is improvements in subtraction-based enhanced line scanning (ELS) endomicroscopy. The 2D scanning of a point scan confocal endomicroscope can be replaced by a scanning line which is synchronised to the sequential readout of a rolling shutter camera. While this leads to high-speed sectioning, as with all line scanning systems, far-from-focus light degrades images. It is possible to remove this by subtracting a second image taken with an offset detection slit. This has previously required two-cameras or two sequentialframes. The latter introduces motion artefacts. This thesis presents a novel approach to ELS using single frame acquisition with real-time mosaicking at 240frames/s

Complementary Situational Awareness for an Intelligent Telerobotic Surgical Assistant System

Robotic surgical systems have contributed greatly to the advancement of Minimally Invasive Surgeries (MIS). More specifically, telesurgical robots have provided enhanced dexterity to surgeons performing MIS procedures. However, current robotic teleoperated systems have only limited situational awareness of the patient anatomy and surgical environment that would typically be available to a surgeon in an open surgery. Although the endoscopic view enhances the visualization of the anatomy, perceptual understanding of the environment and anatomy is still lacking due to the absence of sensory feedback. In this work, these limitations are addressed by developing a computational framework to provide Complementary Situational Awareness (CSA) in a surgical assistant. This framework aims at improving the human-robot relationship by providing elaborate guidance and sensory feedback capabilities for the surgeon in complex MIS procedures. Unlike traditional teleoperation, this framework enables the user to telemanipulate the situational model in a virtual environment and uses that information to command the slave robot with appropriate admittance gains and environmental constraints. Simultaneously, the situational model is updated based on interaction of the slave robot with the task space environment. However, developing such a system to provide real-time situational awareness requires that many technical challenges be met. To estimate intraoperative organ information continuous palpation primitives are required. Intraoperative surface information needs to be estimated in real-time while the organ is being palpated/scanned. The model of the task environment needs to be updated in near real-time using the estimated organ geometry so that the force-feedback applied on the surgeon's hand would correspond to the actual location of the model. This work presents a real-time framework that meets these requirements/challenges to provide situational awareness of the environment in the task space. Further, visual feedback is also provided for the surgeon/developer to view the near video frame rate updates of the task model. All these functions are executed in parallel and need to have a synchronized data exchange. The system is very portable and can be incorporated to any existing telerobotic platforms with minimal overhead

Geometry-Aware Latent Representation Learning for Modeling Disease Progression of Barrett's Esophagus

Barrett's Esophagus (BE) is the only precursor known to Esophageal Adenocarcinoma (EAC), a type of esophageal cancer with poor prognosis upon diagnosis. Therefore, diagnosing BE is crucial in preventing and treating esophageal cancer. While supervised machine learning supports BE diagnosis, high interobserver variability in histopathological training data limits these methods. Unsupervised representation learning via Variational Autoencoders (VAEs) shows promise, as they map input data to a lower-dimensional manifold with only useful features, characterizing BE progression for improved downstream tasks and insights. However, the VAE's Euclidean latent space distorts point relationships, hindering disease progression modeling. Geometric VAEs provide additional geometric structure to the latent space, with RHVAE assuming a Riemannian manifold and $\mathcal{S}$-VAE a hyperspherical manifold. Our study shows that $\mathcal{S}$-VAE outperforms vanilla VAE with better reconstruction losses, representation classification accuracies, and higher-quality generated images and interpolations in lower-dimensional settings. By disentangling rotation information from the latent space, we improve results further using a group-based architecture. Additionally, we take initial steps towards $\mathcal{S}$-AE, a novel autoencoder model generating qualitative images without a variational framework, but retaining benefits of autoencoders such as stability and reconstruction quality

Klinikai és kísérletes, molekuláris és multimodalitású képalkotó vizsgálatok B sejtes malignus lymphomában

Doktori munkám során olyan képalkotó módszereket alkalmaztam, amelyek igazoltan hasznosak limfómás betegeknél a diagnózis felállításában és a terápiára adott válasz értékelésében. A disszertáció egy klinikai és egy preklinikai részre osztható. A szakirodalom alapján a PET-képekből származó radiomikai adatok hozzájárulhatnak a tumor részletgazdagabb in vivo jellemzéséhez, és így segíthetnek az egyénre szabott tumorkezelésben. A klinikai részben kétéves eseménymentes túlélést előrejelző modellt építettünk fel DLBCL-ben szenvedő betegek kezelés előtti FDG-PET/CT vizsgálatából származó radiomikai adatok segítségével. Az első preklinikai vizsgálatban egy új és pontos egérmodell (spontán limfóma) segítségével vizsgáltam egy képalkotás-által szigorúan irányított mintavételi módszert. PET/SPECT és CLI képalkotást kombináló új módszert fejlesztettünk ki egérmodellben, amely lehetővé teszi a legfontosabb érintett nyirokcsomó és/vagy az érintett nyirokcsomó legfontosabb részének irányított kivételét és feldolgozását. A limfómasejtek terjedési mechanizmusai és útvonalai még mindig tisztázatlanok. Célom volt a daganat terjedésének és heterogenitásának a jellemzése mind a preklinikai vizsgálatok első, mind a második részében. Dolgozatom második preklinikai részében a lymphoma peritoneális terjedésének korai kimutatására megvizsgáltuk különböző nukleáris medicina módszerek alkalmazhatóságát és kiválasztottuk ezek közül a legjobban használhatót egérmodell segítségével. Védésem során a klinikai és preklinikai kutatásaink során felmerülő új diagnosztikai lehetőségeket mutatom be a publikált közleményeink alapján

Advancing fluorescent contrast agent recovery methods for surgical guidance applications

Fluorescence-guided surgery (FGS) utilizes fluorescent contrast agents and specialized optical instruments to assist surgeons in intraoperatively identifying tissue-specific characteristics, such as perfusion, malignancy, and molecular function. In doing so, FGS represents a powerful surgical navigation tool for solving clinical challenges not easily addressed by other conventional imaging methods. With growing translational efforts, major hurdles within the FGS field include: insufficient tools for understanding contrast agent uptake behaviors, the inability to image tissue beyond a couple millimeters, and lastly, performance limitations of currently-approved contrast agents in accurately and rapidly labeling disease. The developments presented within this thesis aim to address such shortcomings. Current preclinical fluorescence imaging tools often sacrifice either 3D scale or spatial resolution. To address this gap in high-resolution, whole-body preclinical imaging tools available, the crux of this work lays on the development of a hyperspectral cryo-imaging system and image-processing techniques to accurately recapitulate high-resolution, 3D biodistributions in whole-animal experiments. Specifically, the goal is to correct each cryo-imaging dataset such that it becomes a useful reporter for whole-body biodistributions in relevant disease models. To investigate potential benefits of seeing deeper during FGS, we investigated short-wave infrared imaging (SWIR) for recovering fluorescence beyond the conventional top few millimeters. Through phantom, preclinical, and clinical SWIR imaging, we were able to 1) validate the capability of SWIR imaging with conventional NIR-I fluorophores, 2) demonstrate the translational benefits of SWIR-ICG angiography in a large animal model, and 3) detect micro-dose levels of an EGFR-targeted NIR-I probe during a Phase 0 clinical trial. Lastly, we evaluated contrast agent performances for FGS glioma resection and breast cancer margin assessment. To evaluate glioma-labeling performance of untargeted contrast agents, 3D agent biodistributions were compared voxel-by-voxel to gold-standard Gd-MRI and pathology slides. Finally, building on expertise in dual-probe ratiometric imaging at Dartmouth, a 10-pt clinical pilot study was carried out to assess the technique’s efficacy for rapid margin assessment. In summary, this thesis serves to advance FGS by introducing novel fluorescence imaging devices, techniques, and agents which overcome challenges in understanding whole-body agent biodistributions, recovering agent distributions at greater depths, and verifying agents’ performance for specific FGS applications

Development of a single-mode interstitial rotary probe for In Vivo deep brain fluorescence imaging

Ce mémoire rend compte de l'expertise développée par l'auteur au Centre de recherchede l'Institut universitaire en santé mentale de Québec (CRIUSMQ) en endoscopie fibrée. Il décrit la construction d'un nouveau type de microscope optique, le MicroscopeInterstitiel Panoramique (PIM). Par la juxtaposition d'un court morceau de fibre à gradientd'indice et d'un prisme à l'extrémité d'une fibre monomode, la lumière laser estfocalisée sur le côté de la sonde. Pour former une image, cette dernière est rapidementtournée autour de son axe pendant qu'elle est tirée verticalement par un actuateurpiézo-électrique. Ce design de système rotatif d'imagerie interstitielle peu invasif est uneffort pour limiter les dégâts causés par la sonde tout en imageant la plus grande régionpossible en imagerie optique cérébrale profonde.This thesis documents the expertise developed by the author at the Centre de recherchede l'Institut universitaire en santé mentale de Québec (CRIUSMQ) in fibered endoscopy, particularly the design and construction of a new kind of optical microscope: ThePanoramic Interstitial Microscope (PIM). Through the juxtaposition of a short piece ofGraded-Index fibre and a prism at the end of a single-mode fibre, laser light is focussedon the side of the probe. To form an image, the latter is quickly spun around its axiswhile it is being pulled vertically by a piezoelectric actuator. This minimally invasivefluorescence rotary interstitial imaging system is an endeavor to limit the damage causedby the probe while imaging enough tissue to provide good context to the user in deep brain optical imaging

DATA-DRIVEN APPROACH TO IMAGE CLASSIFICATION

Image classification has been a core topic in the computer vision community. Its recent success with convolutional neural network (CNN) algorithm has led to various real world applications such as large scale management of photos/videos on cloud/social-media, image based search for online retailers, self-driving cars, building robots and healthcare. Image classification can be broadly categorized into binary, multi-class and multi-label classification problems. Binary classification involves assigning one of the two class labels to an instance. In multi-class classification problem, an instance should be categorized into one of more than two classes. Multi-label classification is a generalized version of the multi-class classification problem where each image is assigned multiple labels as opposed to a single label. In this work, we first present various methods that take advantage of deep representations (fully connected layer of pre-trained CNN on the ImageNet dataset) and yield better performance on multi-label classification when compared to methods that use over a dozen conventional visual features. Following the success of deep representations, we intend to build a generic end-to-end deep learning framework to address all three problem categories of image classification. However, there are still no well established guidelines (in terms of choosing the number of layers to go deeper, the number of kernels and the size, the type of regularizer, the choice of non-linear function, etc.) to build an efficient deep neural network and often network architecture design is specific to a problem/dataset. Hence, we present some initial efforts in building a computational framework called Deep Decision Network (DDN) which is completely data-driven. DDN is a tree-like structured built stage-wise. During the learning phase, starting from the root network node, DDN automatically builds a network that splits the data into disjoint clusters of classes which would be handled by the subsequent expert networks. This results in a tree-like structured network driven by the data. The proposed approach provides an insight into the data by identifying the group of classes that are hard to classify and require more attention when compared to others. This feature is crucial for people trying to solve the problem with little or no domain knowledge, especially for applications in medical domain. Initially, we evaluate DDN on a binary classification problem and later extend it to more challenging multi-class and multi-label classification problems. The extension of DDN to multi-class and multi-label involves some changes but they still operate under the same underlying principle. In all the three cases, the proposed approach is tested for its recognition performance and scalability on publicly available datasets providing comparison to other methods

Development of a freehand three-dimensional radial endoscopic ultrasonography system

Oesophageal cancer is an aggressive malignancy with an overall five-year survival of 5-10% and two-thirds of patients have irresectable disease at diagnosis. Accurate staging of oesophageal cancer is important as survival closely correlates with the stage of the tumour, nodal involvement and presence of metastases (TNM staging). Endoscopic ultrasonography (EUS) is currently the most reliable modality for providing accurate T and N staging. Depending on findings of the staging, various treatment options including endoscopic, oncological, and surgical treatments may be performed. It was theorised that the development of three-dimensional radial endoscopic ultrasonography would reduce the operator dependence of EUS and provide accurate dimensional and volume measurements to aid planning and monitoring of treatment. This thesis investigates the development of a three dimensional endoscopic ultrasound technique that can be used with the radial echoendoscopes. Various agar-based tissue mimicking material (TMM) recipes were characterised using a scanning acoustic macroscope to obtain the acoustic properties of attenuation, backscatter and speed of sound. Using these results, a number of endoscopic ultrasound phantoms were developed for the in-vitro investigation and evaluation of 3D-EUS techniques. To increase my understanding of EUS equipment, the imaging and acoustic properties of the EUS endoscopes were characterised using a pipe phantom and a hydrophone. The dual ‘single element’ mechanical and ‘multi-element’ electronic echoendoscopes were investigated. Measured imaging properties included dead space, low contrast penetration, and pipe length. The measured acoustic properties included transmitted beam plots, active working frequency and peak pressures. Three-dimensional ultrasound techniques were developed for specific application to EUS. This included the study of positional monitoring systems, reconstruction algorithms and measurement techniques. A 3D-EUS system was developed using a Microscribe positional arm and frame grabber card, to acquire the 3D dataset. A Matlab 3D-EUS toolbox was written to reconstruct and analyse the volumes. The 3D-EUS systems were evaluated on the EUS phantom and in clinical cases. The usefulness of the 3D-EUS systems was evaluated in a cohort of patients, who were routinely investigated by conventional EUS for a variety of upper gastrointestinal pathology. 3D-EUS accurately staged early tumours and provided the necessary anatomical information to facilitate treatment. With regards to more advanced tumours, 3D-EUS was more accurate than EUS in T and N staging. 3D-EUS gave useful anatomical details in a variety of benign conditions such as varicies and GISTs