Motion compensation in structured illumination fluorescence endomicroscopy

Endomicroscopy is a technique for obtaining real-time images in vivo, eliminating the need to biopsy a tissue sample. A simple fluorescence endomicroscope can be constructed using a fiber bundle, camera, LED and filters, and individual images can be mosaicked as the probe is moved across the tissue to increase the image size. However, to improve image contrast optical sectioning is required for the removal of returning out-of-focus light. Commonly, this is done using the confocal technique, requiring more expensive laser sources and mechanical scanning mirrors which limits the frame rate. Structured illumination microscopy (SIM) instead uses line patterns projected onto the sample to allow for computational optical sectioning. This eliminates the need for point scanning and allows an incoherent light source, such as an LED, to be used, at the cost of some loss of signal-to-noise ratio. However, as SIM requires multiple images to be combined, motion of the probe results in severe image artefacts, preventing the use of mosaicking techniques. We report a SIM endomicroscope using a digital micro-mirror device (DMD) to generate line patterns at high speed, and with the ability to change the patterns on the fly. Combined with a high-speed camera, this reduces motion artefacts significantly, but not sufficiently to allow for video mosaicking techniques. We therefore demonstrate further reduction of artefacts by orienting the illumination patterns parallel to the direction of motion and performing inter-frame registration and correction. This offers potential for low cost, versatile, optically-sectioned endomicroscopy

Automatic motion compensation for structured illumination endomicroscopy using a flexible fiber bundle

Significance: Confocal laser scanning enables optical sectioning in clinical fiber bundle endomicroscopes, but lower-cost, simplified endomicroscopes use widefield incoherent illumination instead. Optical sectioning can be introduced in these simple systems using structured illumination microscopy (SIM), a multiframe digital subtraction process. However, SIM results in artifacts when the probe is in motion, making the technique difficult to use in vivo and preventing the use of mosaicking to synthesize a larger effective field of view (FOV). Aim: We report and validate an automatic motion compensation technique to overcome motion artifacts and allow generation of mosaics in SIM endomicroscopy. Approach: Motion compensation is achieved using image registration and real-time pattern orientation correction via a digital micromirror device. We quantify the similarity of moving probe reconstructions to those acquired with a stationary probe using the relative mean of the absolute differences (MAD). We further demonstrate mosaicking with a moving probe in mechanical and freehand operation. Results: Reconstructed SIM images show an improvement in the MAD from 0.85 to 0.13 for lens paper and from 0.27 to 0.12 for bovine tissue. Mosaics also show vastly reduced artifacts. Conclusion: The reduction in motion artifacts in individual SIM reconstructions leads to mosaics that more faithfully represent the morphology of tissue, giving clinicians a larger effective FOV than the probe itself can provide

Computational Optical Sectioning in Fibre Bundle Endomicroscopy

The field of fibre bundle endomicroscopy has emerged to enable real-time imaging of cellular level features in-vivo. The gold standard is confocal laserscanning, enabling optical sectioning. Point-scan confocal suffers from lower speeds, a need for complex alignment, and the added cost of a laser. This thesis presents three developments in computational optical sectioning for fibre bundle endomicroscopy.The first development is in structured illumination (SIM) endomicroscopy. Lower-cost, simplified endomicroscopes have been developed which use widefield incoherent illumination. Optical sectioning can be introduced to these systems using SIM. SIM improves imaging using spatial modulation of the focal plane and capturing a three-frame sequence. The acquired images are then numerically processed to reject out-of-focus light. This thesis reports and characterises the first high-speed SIM endomicroscope built using a miniature array ofmirrors, a digital micromirror device. The second development is automated motion compensation in SIM endomicroscopy. As a multi frame process, SIM is susceptible to motion artefacts, making the technique difficult to use in vivo and preventing the use of mosaicking to synthesise a larger effective field of view. I report and validate an automatic motion compensation technique to overcome motion artefacts and report the firstmosaics in SIM endomicroscopy.The third development is improvements in subtraction-based enhanced line scanning (ELS) endomicroscopy. The 2D scanning of a point scan confocal endomicroscope can be replaced by a scanning line which is synchronised to the sequential readout of a rolling shutter camera. While this leads to high-speed sectioning, as with all line scanning systems, far-from-focus light degrades images. It is possible to remove this by subtracting a second image taken with an offset detection slit. This has previously required two-cameras or two sequentialframes. The latter introduces motion artefacts. This thesis presents a novel approach to ELS using single frame acquisition with real-time mosaicking at 240frames/s

High-resolution fluorescence endomicroscopy for rapid evaluation of breast cancer margins

Breast cancer is a major public health problem world-wide and the second leading cause of cancer-related female deaths. Breast conserving surgery (BCS), in the form of wide local excision (WLE), allows complete tumour resection while maintaining acceptable cosmesis. It is the recommended treatment for a large number of patients with early stage disease or, in more advanced cases, following neoadjuvant chemotherapy. About 30% of patients undergoing BCS require one or more re-operative interventions, mainly due to the presence of positive margins. The standard of care for surgical margin assessment is post-operative examination of histopathological tissue sections. However, this process is invasive, introduces sampling errors and does not provide real-time assessment of the tumour status of radial margins. The objective of this thesis is to improve intra-operative assessment of margin status by performing optical biopsy in breast tissue. This thesis presents several technical and clinical developments related to confocal fluorescence endomicroscopy systems for real-time characterisation of different breast morphologies. The imaging systems discussed employ flexible fibre-bundle based imaging probes coupled to high-speed line-scan confocal microscope set-up. A preliminary study on 43 unfixed breast specimens describes the development and testing of line-scan confocal laser endomicroscope (LS-CLE) to image and classify different breast pathologies. LS-CLE is also demonstrated to assess the intra-operative tumour status of whole WLE specimens and surgical excisions with high diagnostic accuracy. A third study demonstrates the development and testing of a bespoke LS-CLE system with methylene blue (MB), an US Food and Drug Administration (FDA) approved fluorescent agent, and integration with robotic scanner to enable large-area in vivo imaging of breast cancer. The work also addresses three technical issues which limit existing fibre-bundle based fluorescence endomicroscopy systems: i) Restriction to use single fluorescence agent due to low-speed, single excitation and single fluorescence spectral band imaging systems; ii) Limited Field of view (FOV) of fibre-bundle endomicroscopes due to small size of the fibre tip and iii) Limited spatial resolution of fibre-bundle endomicroscopes due to the spacing between the individual fibres leading to fibre-pixelation effects. Details of design and development of a high-speed dual-wavelength LS-CLE system suitable for high-resolution multiplexed imaging are presented. Dual-wavelength imaging is achieved by sequentially switching between 488 nm and 660 nm laser sources for alternate frames, avoiding spectral bleed-through, and providing an effective frame rate of 60 Hz. A combination of hand-held or robotic scanning with real-time video mosaicking, is demonstrated to enable large-area imaging while still maintaining microscopic resolution. Finally, a miniaturised piezoelectric transducer-based fibre-shifting endomicroscope is developed to enhance the resolution over conventional fibre-bundle based imaging systems. The fibre-shifting endomicroscope provides a two-fold improvement in resolution and coupled to a high-speed LS-CLE scanning system, provides real-time imaging of biological samples at 30 fps. These investigations furthered the utility and applications of the fibre-bundle based fluorescence systems for rapid imaging and diagnosis of cancer margins.Open Acces

Developing endoscopic instrumentation and techniques for in vivo fluorescence lifetime imaging and spectroscopy

Confocal fluorescence endomicroscopes employ fibre optics along with miniaturised scanning and focussing mechanisms to allow microscopic investigation of remote tissue samples with sub-cellular resolution. For this reason they are widely used in biomedical research, both in clinical studies and in small animal imaging experiments. Fluorescence lifetime imaging microscopy (FLIM) has been shown to provide contrast between normal and unhealthy tissue in several diseases including gastro-intestinal (GI) cancer. As such, there is significant interest in developing instrumentation that will allow endoscopic confocal FLIM as this would permit the in vivo investigation of human GI tissue. This thesis describes the development and use of several instruments and techniques aimed at clinically viable in vivo fluorescence lifetime spectroscopy and confocal endomicroscopy. This research has consisted of two broad branches: the study of the fluorescence signature of healthy and diseased tissue both ex vivo and in vivo; and the development of a novel method for achieving beam scanning in confocal endomicroscopy. Firstly the tissue studies are discussed. This begins with the application of a compact steady-state diffuse reflectance/fluorescence spectrometer and a fibre-optic-coupled time-resolved spectrofluorometer to an in vivo investigation of the spectral signatures of skin cancer. This study – which involved the interrogation of 27 clinically diagnosed lesions – was carried out in collaboration with researchers at Lund University in Sweden and revealed significant differences between healthy and diseased tissue both in terms of fluorescence lifetime and steady state reflectance and fluorescence spectra. Further to this study, work is presented charting the development of a clinically viable spectrometer, which measures time-resolved fluorescence spectra with two excitation wavelengths (375 nm and 435 nm) as well as diffuse reflectance spectra. The entire system is contained within a compact trolley (120 x 70 x 55 cm) for easy transportation and safe use in a clinic. It utilises a fibre optic probe to deliver/collect light that can be inserted into the working channel of a medical endoscope meaning that the system can be used to measure diffuse reflectance and time-resolved fluorescence spectra in the GI tract in vivo. The development and testing of this system are discussed and data are presented from both ex vivo and in vivo studies of GI cancer. The second broad section of this thesis focuses more closely on confocal endomicroscopy. Firstly current methods used in this field are discussed and the sources of several drawbacks are explained. A novel approach to laser scanning endomicroscopy is then presented, which requires no moving parts and can be implemented without the need for any distal scanners or optics. This technique is similar in concept to the use of adaptive optics to focus through turbid media: it utilises a proximal spatial light modulator to correct for phase variations across a fibre imaging bundle and then to encode for arbitrary wavefronts at the distal end of that fibre bundle. Thus, it is possible to realise both focussing and beam scanning at the output of the fibre bundle with no distal components, permitting extremely compact endoscopic probes to be developed. Proof-of-principle results are presented illustrating the imaging capabilities of this novel system as well as simulations showing the achievable resolution and field of view in several feasible endoscopic configurations. Overall, this thesis contains work from two quite different projects both aimed at developing novel optical techniques for clinical diagnostic use in endoscopic procedures. The first is aimed at investigating the temporal and spectral properties of the fluorescence and reflectance signatures of cancer, while the goal of the second is to develop improved confocal endomicroscopes

Development and clinical translation of optical and software methods for endomicroscopic imaging

Endomicroscopy is an emerging technology that aims to improve clinical diagnostics by allowing for in vivo microscopy in difficult to reach areas of the body. This is most commonly achieved by using coherent fibre bundles to relay light for illumination and imaging to and from the area under investigation. Endomicroscopy’s attraction for researchers and clinicians is two-fold: on the one hand, its use can reduce the invasiveness of a diagnostic procedure by removing the need for biopsies; On the other hand, it allows for structural and functional in vivo imaging. Endomicroscopic images acquired through optical fibre bundles exhibit artefacts that deteriorate image quality and contrast. This thesis aims to improve an existing endomicroscopy imaging system by exploring two methods that mitigate these artefacts. The first, software-based method takes several processing steps from literature and implements them in an existing endomicroscopy device with a focus on real-time application to enable clinical use, after image quality was found to be inadequate without further processing. A contribution to the field is that two different approaches are implemented and compared in quantitative and qualitative means that have not been compared directly in this manner before. This first attempt at improving endomicroscopy image quality relies solely on digital image processing methods and is developed with a strong focus on real-time applicability in clinical use. Both approaches are compared on pre-clinical and clinical human imaging data. The second method targets the effect of inter-core coupling, which reduces contrast in fibre images. A parallelised confocal imaging method is developed in which a sequence of images is acquired while selectively illuminating groups of fibre cores through the use of a spatial light modulator. A bespoke algorithm creates a composite image in a final processing step. In doing so, unwanted light is detected and removed from the final image. This method is shown to reduce the negative impact of inter-core coupling on image contrast on small imaging targets, while no benefit was found in large, scattering samples

Double-clad fiber-based multifunctional biosensors and multimodal bioimaging systems: technology and applications

Optical fibers have been used to probe various tissue properties such as temperature, pH, absorption, and scattering. Combining different sensing and imaging modalities within a single fiber allows for increased sensitivity without compromising the compactness of an optical fiber probe. A double-clad fiber (DCF) can sustain concurrent propagation modes (single-mode, through its core, and multimode, through an inner cladding), making DCFs ideally suited for multimodal approaches. This study provides a technological review of how DCFs are used to combine multiple sensing functionalities and imaging modalities. Specifically, we discuss the working principles of DCF-based sensors and relevant instrumentation as well as fiber probe designs and functionalization schemes. Secondly, we review different applications using a DCF-based probe to perform multifunctional sensing and multimodal bioimaging.Kathy Beaudette, Jiawen Li, Joseph Lamarre, Lucas Majeau and Caroline Boudou

FIBER-OPTIC BUNDLE FLUORESCENCE MICROSCOPY FOR FUNCTIONAL BRAIN ACTIVITY MAPPING

Understanding the relationship between cellular activities in the animal brain and the emerging patterns of animal behavior is a critical step toward completing the Brain Activity Map. This dissertation describes the development of fiber-bundle microscopy capable of high-resolution cellular imaging, for mapping of functional brain activity in freely moving mice. As a part of this work, several fiber-bundle microscope systems and image processing algorithms were proposed and developed. These optical imaging methods and system performance were tested and evaluated by performing in vivo animal brain imaging. Several fiber-bundle imaging devices, including a dual-mode confocal reflectance and fluorescence micro-endoscope, a single ball-lens imaging probe, and a spatially multiplexed fiber-bundle imager, were designed and developed for high-resolution imaging of brain cells and visualization of brain activity. A dual-mode micro-endoscope, simultaneously achieving laser scanning confocal reflectance and fluorescence imaging, was developed to quantitatively assess gene transfection efficacy using human cervical cancer cells. A single ball-lens integrated imaging probe was designed for endoscopic brain imaging. Lastly, a spatially multiplexed fiber-bundle imager that allows concurrent monitoring of astrocytic activities in multiple brain regions and enables optical manipulation with cell-specific targeting was proposed and experimentally demonstrated. Novel image-processing algorithms were used along with the developed imaging systems. Structured illumination employing a digital micro-mirror device (DMD) was integrated into the system to achieve depth-resolved imaging with a wide-field illumination fiber-bundle microscope. Data from super-resolution fiber-bundle microscopy based on the linear structured illumination were numerically processed to extend the lateral resolution beyond the diffraction limit. To evaluate the performance of the developed fiber-bundle microscope systems and image reconstruction algorithms, the systems and methods were each tested and validated on in vivo animal models, namely transgenic mice expressing a genetically encoded Calcium indicator (GCaMP3) within astrocytes. We showed that locomotion triggers simultaneous activation of astrocyte networks in multiple brain regions in mice. We have also demonstrated real-time cellular-resolution dual-color functional brain imaging in mice. Finally, we established a platform that allows real-time and non-invasive imaging of the intact central nervous system of freely behaving mice. Using this platform, we observed, for the first time, physiologically relevant activation of astrocytes during behaviorally relevant tasks and in the natural setting. In addition, we present a proof-of-concept study by using a fiber-bundle ring light-guided portable multispectral imaging (MSI) platform capable of tissue characterization and preoperative surgical planning for intestinal anastomosis

Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery

One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-opera- tive morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilites by observ- ing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted in- struments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This paper reviews the state-of-the-art methods for optical intra-operative 3D reconstruction in laparoscopic surgery and discusses the technical challenges and future perspectives towards clinical translation. With the recent paradigm shift of surgical practice towards MIS and new developments in 3D opti- cal imaging, this is a timely discussion about technologies that could facilitate complex CAS procedures in dynamic and deformable anatomical regions