Study of Computational Image Matching Techniques: Improving Our View of Biomedical Image Data

Image matching techniques are proven to be necessary in various fields of science and engineering, with many new methods and applications introduced over the years. In this PhD thesis, several computational image matching methods are introduced and investigated for improving the analysis of various biomedical image data. These improvements include the use of matching techniques for enhancing visualization of cross-sectional imaging modalities such as Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), denoising of retinal Optical Coherence Tomography (OCT), and high quality 3D reconstruction of surfaces from Scanning Electron Microscope (SEM) images. This work greatly improves the process of data interpretation of image data with far reaching consequences for basic sciences research. The thesis starts with a general notion of the problem of image matching followed by an overview of the topics covered in the thesis. This is followed by introduction and investigation of several applications of image matching/registration in biomdecial image processing: a) registration-based slice interpolation, b) fast mesh-based deformable image registration and c) use of simultaneous rigid registration and Robust Principal Component Analysis (RPCA) for speckle noise reduction of retinal OCT images. Moving towards a different notion of image matching/correspondence, the problem of view synthesis and 3D reconstruction, with a focus on 3D reconstruction of microscopic samples from 2D images captured by SEM, is considered next. Starting from sparse feature-based matching techniques, an extensive analysis is provided for using several well-known feature detector/descriptor techniques, namely ORB, BRIEF, SURF and SIFT, for the problem of multi-view 3D reconstruction. This chapter contains qualitative and quantitative comparisons in order to reveal the shortcomings of the sparse feature-based techniques. This is followed by introduction of a novel framework using sparse-dense matching/correspondence for high quality 3D reconstruction of SEM images. As will be shown, the proposed framework results in better reconstructions when compared with state-of-the-art sparse-feature based techniques. Even though the proposed framework produces satisfactory results, there is room for improvements. These improvements become more necessary when dealing with higher complexity microscopic samples imaged by SEM as well as in cases with large displacements between corresponding points in micrographs. Therefore, based on the proposed framework, a new approach is proposed for high quality 3D reconstruction of microscopic samples. While in case of having simpler microscopic samples the performance of the two proposed techniques are comparable, the new technique results in more truthful reconstruction of highly complex samples. The thesis is concluded with an overview of the thesis and also pointers regarding future directions of the research using both multi-view and photometric techniques for 3D reconstruction of SEM images

Content-aware frame interpolation (CAFI): deep learning-based temporal super-resolution for fast bioimaging

The development of high-resolution microscopes has made it possible to investigate cellular processes in 3D and over time. However, observing fast cellular dynamics remains challenging because of photobleaching and phototoxicity. Here we report the implementation of two content-aware frame interpolation (CAFI) deep learning networks, Zooming SlowMo and Depth-Aware Video Frame Interpolation, that are highly suited for accurately predicting images in between image pairs, therefore improving the temporal resolution of image series post-acquisition. We show that CAFI is capable of understanding the motion context of biological structures and can perform better than standard interpolation methods. We benchmark CAFI’s performance on 12 different datasets, obtained from four different microscopy modalities, and demonstrate its capabilities for single-particle tracking and nuclear segmentation. CAFI potentially allows for reduced light exposure and phototoxicity on the sample for improved long-term live-cell imaging. The models and the training and testing data are available via the ZeroCostDL4Mic platform

The value of remote sensing techniques in supporting effective extrapolation across multiple marine spatial scales

The reporting of ecological phenomena and environmental status routinely required point observations, collected with traditional sampling approaches to be extrapolated to larger reporting scales. This process encompasses difficulties that can quickly entrain significant errors. Remote sensing techniques offer insights and exceptional spatial coverage for observing the marine environment. This review provides guidance on (i) the structures and discontinuities inherent within the extrapolative process, (ii) how to extrapolate effectively across multiple spatial scales, and (iii) remote sensing techniques and data sets that can facilitate this process. This evaluation illustrates that remote sensing techniques are a critical component in extrapolation and likely to underpin the production of high-quality assessments of ecological phenomena and the regional reporting of environmental status. Ultimately, is it hoped that this guidance will aid the production of robust and consistent extrapolations that also make full use of the techniques and data sets that expedite this process

Registration of serial sections: An evaluation method based on distortions of the ground truths

Registration of histological serial sections is a challenging task. Serial sections exhibit distortions and damage from sectioning. Missing information on how the tissue looked before cutting makes a realistic validation of 2D registrations extremely difficult. This work proposes methods for ground-truth-based evaluation of registrations. Firstly, we present a methodology to generate test data for registrations. We distort an innately registered image stack in the manner similar to the cutting distortion of serial sections. Test cases are generated from existing 3D data sets, thus the ground truth is known. Secondly, our test case generation premises evaluation of the registrations with known ground truths. Our methodology for such an evaluation technique distinguishes this work from other approaches. Both under- and over-registration become evident in our evaluations. We also survey existing validation efforts. We present a full-series evaluation across six different registration methods applied to our distorted 3D data sets of animal lungs. Our distorted and ground truth data sets are made publicly available.Comment: Supplemental data available under https://zenodo.org/record/428244

Development and application of molecular and computational tools to image copper in cells

Copper is a trace element which is essential for many biological processes. A deficiency or excess of copper(I) ions, which is its main oxidation state of copper in cellular environment, is increasingly linked to the development of neurodegenerative diseases such as Parkinson’s and Alzheimer’s disease (PD and AD). The regulatory mechanisms for copper(I) are under active investigation and lysosomes which are best known as cellular “incinerators” have been found to play an important role in the trafficking of copper inside the cell. Therefore, it is important to develop reliable experimental methods to detect, monitor and visualise this metal in cells and to develop tools that allow to improve the data quality of microscopy recordings. This would enable the detailed exploration of cellular processes related to copper trafficking through lysosomes. The research presented in this thesis aimed to develop chemical and computational tools that can help to investigate concentration changes of copper(I) in cells (particularly in lysosomes), and it presents a preliminary case study that uses the here developed microscopy image quality enhancement tools to investigate lysosomal mobility changes upon treatment of cells with different PD or AD drugs. Chapter I first reports the synthesis of a previously reported copper(I) probe (CS3). The photophysical properties of this probe and functionality on different cell lines was tested and it was found that this copper(I) sensor predominantly localized in lipid droplets and that its photostability and quantum yield were insufficient to be applied for long term investigations of cellular copper trafficking. Therefore, based on the insights of this probe a new copper(I) selective fluorescent probe (FLCS1) was designed, synthesized, and characterized which showed superior photophysical properties (photostability, quantum yield) over CS3. The probe showed selectivity for copper(I) over other physiological relevant metals and showed strong colocalization in lysosomes in SH-SY5Y cells. This probe was then used to study and monitor lysosomal copper(I) levels via fluorescence lifetime imaging microscopy (FLIM); to the best of my knowledge this is the first copper(I) probe based on emission lifetime. Chapter II explores different computational deep learning approaches for improving the quality of recorded microscopy images. In total two existing networks were tested (fNET, CARE) and four new networks were implemented, tested, and benchmarked for their capabilities of improving the signal-to-noise ratio, upscaling the image size (GMFN, SRFBN-S, Zooming SlowMo) and interpolating image sequences (DAIN, Zooming SlowMo) in z- and t-dimension of multidimensional simulated and real-world datasets. The best performing networks of each category were then tested in combination by sequentially applying them on a low signal-to-noise ratio, low resolution, and low frame-rate image sequence. This image enhancement workstream for investigating lysosomal mobility was established. Additionally, the new frame interpolation networks were implemented in user-friendly Google Colab notebooks and were made publicly available to the scientific community on the ZeroCostDL4Mic platform. Chapter III provides a preliminary case study where the newly developed fluorescent copper(I) probe in combination with the computational enhancement algorithms was used to investigate the effects of five potential Parkinson’s disease drugs (rapamycin, digoxin, curcumin, trehalose, bafilomycin A1) on the mobility of lysosomes in live cells.Open Acces

New Algorithms in Computational Microscopy

Microscopy plays an important role in providing tools to microscopically observe objects and their surrounding areas with much higher resolution ranging from the scale between molecular machineries (angstrom) and individual cells (micrometer). Under microscopes, illumination, such as visible light and electron-magnetic radiation/electron beam, interacts with samples, then they are scattered to a plane and are recorded. Computational microscopy corresponds to image reconstruction from these measurements as well as improving quality of the images. Along with the evolution of microscopy, new studies are discovered and algorithms need development not only to provide high-resolution imaging but also to decipher new and advanced research. In this dissertation, we focus on algorithm development for inverse problems in microscopy, specifically phase retrieval and tomography, and the application of these techniques to machine learning. The four studies in this dissertation demonstrates the use of optimization and calculus of variation in imaging science and other different disciplines.Study 1 focuses on coherent diffractive imaging (CDI) or phase retrieval, a non-linear inverse problem that aims to recover 2D image from it Fourier transforms in modulus taking into account that extra information provided by oversampling as a second constraint. To solve this two-constraint minimization, we proceed from Hamilton-Jacobi partial differential equation (HJ-PDE) and its Hopf-Lax formula. Introducing generalized Bregman distance to the HJ-PDE and applying Legendre transform, we derive our generalized proximal smoothing (GPS) algorithm under the form of primal-dual hybrid gradient (PDHG). While the reflection operator, known as extrapolating momentum, helps overcome local minima, the smoothing by the generalized Bregman distance is adjusted to improve convergence and consistency of phase retrieval.Study 2 focuses on electron tomography, 3D image reconstruction from a set of 2D projections obtained from a transmission electron microscope (TEM) or X-ray microscope. Notice that current tomography algorithms limit to a single tilt axis and fail to work with fully or partially missing data. In the light of calculus of variations and Fourier slice theorem (FST), we develop a highly accurate tomography iterative algorithm that can provide higher resolution imaging and work with missing data as well as has capability to perform multiple-tilt-axis tomography. The algorithm is further developed to work with non-isolated objects and partially-blocked projections which have become more popular in experiment. The success of real space iterative reconstruction engine (RESIRE) opens a new era to the study of tomography in material science and magnetic structures (vector Tomography).Study 3 and 4 are applications of our algorithms to machine learning. Study 3 develops a backward Euler method in a stochastic manner to solve K-mean clustering, a well-known non-convex optimization problem. The algorithm has been shown to improve minimums and consistency, providing a new powerful tool to the class of classification techniques. Study 4 is a direct application of GPS to deep learning gradient descent algorithms. Linearizing the Hopf-Lax formula derived in GPS, we derive our method Laplacian smoothing gradient descent (LSGD), simply known as gradient smoothing. Our experiment shows that LSGD has the ability to search for better and flatter minimums, reduce variation, and obtain higher accuracy and consistency

Volumetric Lissajous confocal microscopy with tunable spatiotemporal resolution

Dynamic biological systems present challenges to existing three-dimensional (3D) optical microscopes because of their continuous temporal and spatial changes. Most techniques are rigid in adapting the acquisition parameters over time, as in confocal microscopy, where a laser beam is sequentially scanned at a predefined spatial sampling rate and pixel dwell time. Such lack of tunability forces a user to provide scan parameters, which may not be optimal, based on the best assumption before an acquisition starts. Here, we developed volumetric Lissajous confocal microscopy to achieve unsurpassed 3D scanning speed with a tunable sampling rate. The system combines an acoustic liquid lens for continuous axial focus translation with a resonant scanning mirror. Accordingly, the excitation beam follows a dynamic Lissajous trajectory enabling sub-millisecond acquisitions of image series containing 3D information at a sub-Nyquist sampling rate. By temporal accumulation and/or advanced interpolation algorithms, the volumetric imaging rate is selectable using a post-processing step at the desired spatiotemporal resolution for events of interest. We demonstrate multicolor and calcium imaging over volumes of tens of cubic microns with 3D acquisition speeds of 30 Hz and frame rates up to 5 kHz

Optically gated beating-heart imaging

The constant motion of the beating heart presents an obstacle to clear optical imaging, especially 3D imaging, in small animals where direct optical imaging would otherwise be possible. Gating techniques exploit the periodic motion of the heart to computationally "freeze" this movement and overcome motion artefacts. Optically gated imaging represents a recent development of this, where image analysis is used to synchronize acquisition with the heartbeat in a completely non-invasive manner. This article will explain the concept of optical gating, discuss a range of different implementation strategies and their strengths and weaknesses. Finally we will illustrate the usefulness of the technique by discussing applications where optical gating has facilitated novel biological findings by allowing 3D in vivo imaging of cardiac myocytes in their natural environment of the beating heart

Improving digital image correlation in the TopoSEM Software Package

Dissertação de mestrado integrado em Informatics EngineeringTopoSEM is a software package with the aim of reconstructing a 3D surface topography of a microscopic sample from a set of 2D Scanning Electron Microscopy (SEM) images. TopoSEM is also able to produce a stability report on the calibration of the SEM hardware based solely on output images. One of the key steps in both of these workflows is the use of a Digital Image Correlation (DIC) algorithm, a no-contact imaging technique, to measure full-field displacements of an input image. A novel DIC implementation fine-tuned for 3D reconstructions was originally developed in MATLAB to satisfy the feature requirement of this project. However, near real-time usability of the TopoSEM is paramount for its users, and the main barrier towards this goal is the under-performing DIC implementation. This dissertation work ported the original MATLAB implementation of TopoSEM to sequential C++ and its performance was further optimised: (i) to improve memory accesses, (ii) to explore the available vector exten sions in each core of current multiprocessor chips processors to perform computationally intensive operations on vectors and matrices of single and double-precision floating point values, and (iii) to additionally improve the execution performance through parallelization on multi-core devices, by using multiple threads with a front wave propagation scheduler. The initial MATLAB implementation took 3279.4 seconds to compute the full-field displacement of a 2576 pixels by 2086 pixels image on a quad-core laptop. With all added improvements, the new parallel C++ version on the same laptop lowered the execution time to 1.52 seconds, achieving an overall speedup of 2158.TopoSEM é um programa cujo objetivo é reconstruir em 3D a topografia de uma amostra capturada por um mi croscópio electrónico de varrimento. Esta ferramenta é também capaz de gerar um relatório sobre a estabilidade da calibração do microscópio com base apenas em imagens capturadas. Um dos passos chave para ambas as funcionalidades trata-se da utilização de um algoritmo de Correlação Digital de Imagens (DIC), uma técnica de visão por computador que não envolve contacto direto com a amostra e que permite medir deslocamentos e deformações entre imagens. Criou-se uma nova implementação de DIC em MATLAB especialmente formulada para reconstrução 3D. No entanto, a capacidade de utilizar o TopoSEM em quase tempo real é fundamental para os seus utilizadores e a principal barreira para tal são os elevados tempos de execução da implementação em MATLAB. Esta dissertação portou o código de MATLAB para código sequencial em C++ e a sua performance foi melho rada: (i) para otimizar acessos a memória, (ii) para explorar extensões de vetorização disponíveis em hardware moderno para otimizar operações sobre vetores e matrizes, e (iii) para através de paralelização em dispositivos multi-core melhorar ainda mais a performance utilizando para isso vários fios de execução com um escalonador de propagação em onda. A implementação inicial em MATLAB demorava 3279.4 segundos para computar uma imagem com resolução de 2576 pixels por 2086 pixels num portátil quad-core. Com todas as melhorias de performance, a nova imple mentação paralela em C++ reduziu o tempo de execução para 1.52 segundos para as mesmas imagens no mesmo computador, atingindo um speedup de 2158