Video Superresolution Reconstruction Using Iterative Back Projection with Critical-Point Filters Based Image Matching

To improve the spatial resolution of reconstructed images/videos, this paper proposes a Superresolution (SR) reconstruction algorithm based on iterative back projection. In the proposed algorithm, image matching using critical-point filters (CPF) is employed to improve the accuracy of image registration. First, a sliding window is used to segment the video sequence. CPF based image matching is then performed between frames in the window to obtain pixel-level motion fields. Finally, high-resolution (HR) frames are reconstructed based on the motion fields using iterative back projection (IBP) algorithm. The CPF based registration algorithm can adapt to various types of motions in real video scenes. Experimental results demonstrate that, compared to optical flow based image matching with IBP algorithm, subjective quality improvement and an average PSNR score of 0.53 dB improvement are obtained by the proposed algorithm, when applied to video sequence

Optical imaging methods for the study of disease models from the nano to the mesoscale

The visualisation of disease phenotypes allows scientists to study fundamental mechanisms of disease. Optical imaging methods are useful not only to observe anatomical features of biological samples, but also to infer interactions between molecular species using fluorescence labelling. This thesis presents the development of imaging and analysis tools to study biological questions in three models of disease, with samples ranging from the sub-cellular to the organ scale. First, the role of the alpha-synuclein (a-syn) protein, whose dysfunction is a hallmark of Parkinson’s Disease, was studied with respect to vesicle trafficking at the synapse. Synaptic vesicles are ∼40 nm in diameter; imaging vesicles therefore requires methods with resolution below the diffraction limit. Single-molecule localisation microscopy (SMLM), which circumvents the diffraction limit by separating fluorophore emission in time to localise individual molecules in space with ∼20 nm precision, was thus implemented to study a-syn in purified synaptic boutons. A software package was developed to analyse the colocalisation of a-syn with internalised vesicles, and the clustering of a-syn under differing synaptic calcium levels. The colocalisation of a-syn and internalised vesicles was found to be temperature independent, suggesting that a-syn is involved in non-canonical trafficking mechanisms. Ground truth simulations from a synaptosome model were used to benchmark two cluster analysis methods. Both methods applied on the experimental data showed that a-syn becomes less clustered at low synaptic calcium levels. Second, the spatiotemporal association of ESCRT-II, a protein complex whose role in the budding of the human immunodeficiency virus (HIV) was previously considered dispensable, and the HIV polyprotein Gag was studied during viral egress using novel image analysis tools. A nearest-neighbour analysis showed the ESCRT-II protein EAP45 colocalises with Gag similarly to ALIX, a protein well known to be involved in HIV budding. However, upon deletion of EAP45’s N-terminus, its colocalisation with Gag was significantly impaired, highlighting the importance of this EAP45 domain in linking to Gag. Single particle tracking was used to trace the trajectories of EAP45 and Gag in live cells, and an algorithm was developed to visualise the simultaneous motion of two particles; these analyses revealed three types of potential dynamic interaction between EAP45 and Gag. Finally, an open-source instrument to visualise phenotypes from large organs in 3D was developed for the study of chronic obstructive pulmonary disease (COPD) models. The instrument implements Optical Projection Tomography, a technique which can reconstruct cross-sectional slices of a transparent object from its orthographic projections, using off-the- shelf components and novel ImageJ plugins for artefact correction and volume reconstructions. Excised and cleared mouse lungs were imaged in which high order airways can be discerned with 50 μm resolution. The raw lung data, instructions for building the instrument, the free ImageJ plugins, and a detailed software manual are available in an online repository to encourage the widespread use of OPT for imaging large samples.Gates Cambridg

MAP-based of Super-resolution Reconstruction Algorithm

引入一种基于关键点滤波(CrITICAl-POInT fIlTErS,CPf)的图像配准方法,并在最大后验概率(MAXIMuM A POSTErIOrI,MAP)框架下提出一种改进的集投影法(PrOJECTIOnS OnTO COnVEXSETS,MAP/POCS)混合算法。算法把POCS的残差约束集合加入到基于CPf图像配准的MAP正则算法中,在每次迭代重建中对重建图像的像素点进行约束,充分利用这三种算法的优点。实验结果表明,相比于传统的重建方法,该算法能够更有效地表达视频中的非平移运动,超分辨图像主观质量有明显改善。An image registration method based on the critical-point filters is introduced in this paper,and an improved MAP/POCS hybrid algorithm based on the framework of MAP has also been proposed,the POCS residual set of constraints is added to the reconstruction algorithm of MAP based on the critical-point filters registration method joined the adaptive regularization parameter so that it can take advantage of the three algorithms.After the test,the results show that compared with the traditional methods,the algorithm of this paper can express the non-translational motion more effectively and the quality of the super-resolution images has been improved obviously.国家自然科学基金项目(61102135); 中国博士后科学基金项目(2012M511433

Coherent Diffraction Imaging and Ptychography of Human Metaphase Chromosomes

Mitotic chromosome structure is highly important in the packing and safe separation of DNA during cell division. The chromatin fibre, the first packaging of DNA around histone proteins, is coiled into a highly condensed state during the metaphase stage of mitosis where the chromosomes form their characteristic ’X’ shape in order to be split into two daughter cells on cell division. Accessing the 3D structure of the condensed chromatin could provide high resolution information on the location of the genes inside the chromosome during metaphase and the placement of the proteins used in the packaging mechanism. We apply lensless X-ray imaging techniques to chromosomes in an attempt to access their structure. Coherent Diffraction Imaging and Ptychography have potential to achieve wavelength limited resolution in 3D. The techniques directly measure phase that can be used to map the electron density in the sample. Sample preparation is highly important to obtain good quality images from any microscopy technique. We start with studying sample preparations with Fluorescence and Scanning Electron Microscopy to observe the suitability and effects of the protocols. We perform lensless imaging on nuclei and chromosomes prepared with newly designed protocols for this type of X-ray imaging. Images of nuclei were retrieved from the ptychography experiments at 34-ID-C, I-13 and cSAXS. Ptychography was performed with a visible light source on a set of 46 human chromosomes. The images provide a map of electron density that can be used to calculate the mass of chromosomes and nuclei. Mass is a macroscopic structural quantity that can be used to observe the changes to the protein and DNA content in the nucleus during the cell cycle given a high enough resolution. The nuclei measured in this work were in an early mitotic state and gave consistent mass values calculated from the phase measured by ptychography. In the case of chromosomes mass calculated from the ptychography images was used to sort and identify them in a form of karyotyping. This method of identification is novel because it takes into account the whole of the chromosome contents, both protein and DNA, whereas standard methods of karyotyping look at positions of bandsin the chromosomes

Advanced VLBI Imaging

Very Long Baseline Interferometry (VLBI) is an observational technique developed in astronomy for combining multiple radio telescopes into a single virtual instrument with an effective aperture reaching up to many thousand kilometers and enabling measurements at highest angular resolutions. The celebrated examples of applying VLBI to astrophysical studies include detailed, high-resolution images of the innermost parts of relativistic outflows (jets) in active galactic nuclei (AGN) and recent pioneering observations of the shadows of supermassive black holes (SMBH) in the center of our Galaxy and in the galaxy M87. Despite these and many other proven successes of VLBI, analysis and imaging of VLBI data still remain difficult, owing in part to the fact that VLBI imaging inherently constitutes an ill-posed inverse problem. Historically, this problem has been addressed in radio interferometry by the CLEAN algorithm, a matching-pursuit inverse modeling method developed in the early 1970-s and since then established as a de-facto standard approach for imaging VLBI data. In recent years, the constantly increasing demand for improving quality and fidelity of interferometric image reconstruction has resulted in several attempts to employ new approaches, such as forward modeling and Bayesian estimation, for application to VLBI imaging. While the current state-of-the-art forward modeling and Bayesian techniques may outperform CLEAN in terms of accuracy, resolution, robustness, and adaptability, they also tend to require more complex structure and longer computation times, and rely on extensive finetuning of a larger number of non-trivial hyperparameters. This leaves an ample room for further searches for potentially more effective imaging approaches and provides the main motivation for this dissertation and its particular focusing on the need to unify algorithmic frameworks and to study VLBI imaging from the perspective of inverse problems in general. In pursuit of this goal, and based on an extensive qualitative comparison of the existing methods, this dissertation comprises the development, testing, and first implementations of two novel concepts for improved interferometric image reconstruction. The concepts combine the known benefits of current forward modeling techniques, develop more automatic and less supervised algorithms for image reconstruction, and realize them within two different frameworks. The first framework unites multiscale imaging algorithms in the spirit of compressive sensing with a dictionary adapted to the uv-coverage and its defects (DoG-HiT, DoB-CLEAN). We extend this approach to dynamical imaging and polarimetric imaging. The core components of this framework are realized in a multidisciplinary and multipurpose software MrBeam, developed as part of this dissertation. The second framework employs a multiobjective genetic evolutionary algorithm (MOEA/D) for the purpose of achieving fully unsupervised image reconstruction and hyperparameter optimization. These new methods are shown to outperform the existing methods in various metrics such as angular resolution, structural sensitivity, and degree of supervision. We demonstrate the great potential of these new techniques with selected applications to frontline VLBI observations of AGN jets and SMBH. In addition to improving the quality and robustness of image reconstruction, DoG-HiT, DoB-CLEAN and MOEA/D also provide such novel capabilities as dynamic reconstruction of polarimetric images on minute time-scales, or near-real time and unsupervised data analysis (useful in particular for application to large imaging surveys). The techniques and software developed in this dissertation are of interest for a wider range of inverse problems as well. This includes such versatile fields such as Ly-alpha tomography (where we improve estimates of the thermal state of the intergalactic medium), the cosmographic search for dark matter (where we improve forecasted bounds on ultralight dilatons), medical imaging, and solar spectroscopy

STED Nanoscopy to Illuminate New Avenues in Cancer Research – From Live Cell Staining and Direct Imaging to Decisive Preclinical Insights for Diagnosis and Therapy

Molecular imaging is established as an indispensable tool in various areas of cancer research, ranging from basic cancer biology and preclinical research to clinical trials and medical practice. In particular, the field of fluorescence imaging has experienced exceptional progress during the last three decades with the development of various in vivo technologies. Within this field, fluorescence microscopy is primarily of experimental use since it is especially qualified for addressing the fundamental questions of molecular oncology. As stimulated emission depletion (STED) nanoscopy combines the highest spatial and temporal resolutions with live specimen compatibility, it is best-suited for real-time investigations of the differences in the molecular machineries of malignant and normal cells to eventually translate the acquired knowledge into increased diagnostic and therapeutic efficacy. This thesis presents the application of STED nanoscopy to two acute topics in cancer research of direct or indirect clinical interest. The first project has investigated the structure of telomeres, the ends of the linear eukaryotic chromosomes, in intact human cells at the nanoscale. To protect genome integrity, a telomere can mask the chromosome end by folding back and sequestering its single-stranded 3’-overhang in an upstream part of the double-stranded DNA repeat region. The formed t-loop structure has so far only been visualized by electron microscopy and fluorescence nanoscopy with cross-linked mammalian telomeric DNA after disruption of cell nuclei and spreading. For the first time, this work demonstrates the existence of t-loops within their endogenous nuclear environment in intact human cells. The identification of further telomere conformations has laid the groundwork for distinguishing cancerous cells that use different telomere maintenance mechanisms based on their individual telomere populations by a combined STED nanoscopy and deep learning approach. The population difference was essentially attributed to the promyelocytic leukemia (PML) protein that significantly perturbs the organization of a subpopulation of telomeres towards an open conformation in cancer cells that employ a telomerase-independent, alternative telomere lengthening mechanism. Elucidating the nanoscale topology of telomeres and associated proteins within the nucleus has provided new insight into telomere structure-function relationships relevant for understanding the deregulation of telomere maintenance in cancer cells. After understanding the molecular foundations, this newly gained knowledge can be exploited to develop novel or refined diagnostic and treatment strategies. The second project has characterized the intracellular distribution of recently developed prostate cancer tracers. These novel prostate-specific membrane antigen (PSMA) inhibitors have revolutionized the treatment regimen of prostate cancer by enabling targeted imaging and therapy approaches. However, the exact internalization mechanism and the subcellular fate of these tracers have remained elusive. By combining STED nanoscopy with a newly developed non-standard live cell staining protocol, this work confirmed cell surface clustering of the targeted membrane antigen upon PSMA inhibitor binding, subsequent clathrin-dependent endocytosis and endosomal trafficking of the antigen-inhibitor complex. PSMA inhibitors accumulate in prostate cancer cells at clinically relevant time points, but strikingly and in contrast to the targeted antigen itself, they eventually distribute homogenously in the cytosol. This project has revealed the subcellular fate of PSMA/PSMA inhibitor complexes for the first time and provides crucial knowledge for the future application of these tracers including the development of new strategies in the field of prostate cancer diagnostics and therapeutics. Relying on the photostability and biocompatibility of the applied fluorophores, the performance of live cell STED nanoscopy in the field of cancer research is boosted by the development of improved fluorophores. The third project in this thesis introduces a biocompatible, small molecule near-infrared dye suitable for live cell STED imaging. By the application of a halogen dance rearrangement, a dihalogenated fluorinatable pyridinyl rhodamine could be synthesized at high yield. The option of subsequent radiolabeling combined with excellent optical properties and a non-toxic profile renders this dye an appropriate candidate for medical and bioimaging applications. Providing an intrinsic and highly specific mitochondrial targeting ability, the radiolabeled analogue is suggested as a vehicle for multimodal (positron emission tomography and optical imaging) medical imaging of mitochondria for cancer diagnosis and therapeutic approaches in patients and biopsy tissue. The absence of cytotoxicity is not only a crucial prerequisite for clinically used fluorophores. To guarantee the generation of meaningful data mirroring biological reality, the absence of cytotoxicity is likewise a decisive property of dyes applied in live cell STED nanoscopy. The fourth project in this thesis proposes a universal approach for cytotoxicity testing based on characterizing the influence of the compound of interest on the proliferation behavior of human cell lines using digital holographic cytometry. By applying this approach to recently developed live cell STED compatible dyes, pronounced cytotoxic effects could be excluded. Looking more closely, some of the tested dyes slightly altered cell proliferation, so this project provides guidance on the right choice of dye for the least invasive live cell STED experiments. Ultimately, live cell STED data should be exploited to extract as much biological information as possible. However, some information might be partially hidden by image degradation due the dynamics of living samples and the deliberate choice of rather conservative imaging parameters in order to preserve sample viability. The fifth project in this thesis presents a novel image restoration method in a Bayesian framework that simultaneously performs deconvolution, denoising as well as super-resolution, to restore images suffering from noise with mixed Poisson-Gaussian statistics. Established deconvolution or denoising methods that consider only one type of noise generally do not perform well on images degraded significantly by mixed noise. The newly introduced method was validated with live cell STED telomere data proving that the method can compete with state-of-the-art approaches. Taken together, this thesis demonstrates the value of an integrated approach for STED nanoscopy imaging studies. A coordinated workflow including sample preparation, image acquisition and data analysis provided a reliable platform for deriving meaningful conclusions for current questions in the field of cancer research. Moreover, this thesis emphasizes the strength of iteratively adapting the individual components in the operational chain and it particularly points towards those components that, if further improved, optimize the significance of the final results rendering live cell STED nanoscopy even more powerful

Medical Image Enhancement using Deep Learning and Tensor Factorization Techniques

La résolution spatiale des images acquises par tomographie volumique à faisceau conique (CBCT) est limitée par la géométrie des capteurs, leur sensibilité, les mouvements du patient, les techniques de reconstruction d'images et la limitation de la dose de rayonnement. Le modèle de dégradation d'image considéré dans cette thèse consiste en un opérateur de ou avec la fonction d'étalement du système d'imagerie (PSF), un opérateur de décimation, et du bruit, qui relient les volumes CBCT à une image 3D super-résolue à estimer. Les méthodes proposées dans cette thèse (SISR - single image super-résolution) ont comme objectif d'inverser ce modèle direct, c'est à dire d'estimer un volume haute résolution à partir d'une image CBCT. Les algorithmes ont été évalués dans le cadre d'une application dentaire, avec comme vérité terrain les images haute résolution acquises par micro CT (µCT), qui utilise des doses de rayonnement très importantes, incompatibles avec les applications cliniques. Nous avons proposé une approche de SISR par deep learning, appliquée individuellement à des coupes CBCT. Deux types de réseaux ont été évalués : U-net et subpixel. Les deux ont amélioré les volumes CBCT, avec un gain en PSNR de 21 à 22 dB et en coefficient de Dice pour la segmentation canalaire de 1 à 2.2 %. Le gain a été plus particulièrement important dans la partie apicale des dents, ce qui représente un résultat important étant donnée son importance pour les applications cliniques. Nous avons proposé des algorithmes de SISR basés sur la décomposition canonique polyadique des tenseurs. Le principal avantage de cette méthode, lié à l'utilisation de la théorie des tenseur, est d'utiliser la structure 3D des volumes CBCT. L'algorithme proposé regroupe plusieurs étapes: débruitage base sur la factorisation des tenseurs, déconvolution et super-résolution, avec un faible nombre d'hyperparamètres. Le temps d'exécution est très faible par rapport aux algorithmes existants (deux ordres de magnitude plus petit), pour des performances légèrement supérieures (gain de 1.2 à 1.5 dB en PSNR). La troisième contribution de la thèse est en lien avec la contribution 2 : l'algorithme de SISR basé sur la décomposition canonique polyadique des tenseurs est combiné avec une méthode d'estimation de la PSF, inconnues dans les applications pratiques. L'algorithme résultant effectue les deux tâche de manière alternée, et s'avère précis et rapide sur des données de simulation et expérimentales. La dernière contribution de la thèse a été d'évaluer l'intérêt d'un autre type de décomposition tensorielle, la décomposition de Tucker, dans le cadre d'un algorithme de SISR. Avant la déconvolution, le volume CBCT est débruité en tronquant sa décomposition de Tucker. Comparé à l'algorithme de la contribution 2, cette approche permet de diminuer encore plus le temps de calcul, d'un facteur 10, pour des performances similaires pour des SNR importants et légèrement supérieures pour de faibles SNR. Le lien entre cette méthode et les algorithmes 2D basés sur une SVD facilite le réglage des hyperparamètres comparé à la décomposition canonique polyadique.The resolution of dental cone beam computed tomography (CBCT) images is imited by detector geometry, sensitivity, patient movement, the reconstruction technique and the need to minimize radiation dose. The corresponding image degradation model assumes that the CBCT image is a blurred (with a point spread function, PSF), downsampled, noisy version of a high resolution image. The quality of the image is crucial for precise diagnosis and treatment planning. The methods proposed in this thesis aim to give a solution for the single image super-resolution (SISR) problem. The algorithms were evaluated on dental CBCT and corresponding highresolution (and high radiation-dose) µCT image pairs of extracted teeth. I have designed a deep learning framework for the SISR problem, applied to CBCT slices. I have tested the U-net and subpixel neural networks, which both improved the PSNR by 21-22 dB, and the Dice coe_cient of the canal segmentation by 1-2.2%, more significantly in the medically critical apical region. I have designed an algorithm for the 3D SISR problem, using the canonical polyadic decomposition of tensors. This implementation conserves the 3D structure of the volume, integrating the factorization-based denoising, deblurring with a known PSF, and upsampling of the image in a lightweight algorithm with a low number of parameters. It outperforms the state-of-the-art 3D reconstruction-based algorithms with two orders of magnitude faster run-time and provides similar PSNR (improvement of 1.2-1.5 dB) and segmentation metrics (Dice coe_cient increased on average to 0.89 and 0.90). Thesis II b: I have implemented a joint alternating recovery of the unknown PSF parameters and of the high-resolution 3D image using CPD-SISR. The algorithm was compared to a state-of-the-art 3D reconstruction-based algorithm, combined with the proposed alternating PSF-optimization. The two algorithms have shown similar improvement in PSNR, but CPD-SISR-blind converged roughly 40 times faster, under 6 minutes both in simulation and on experimental dental computed tomography data. I have proposed a solution for the 3D SISR problem using the Tucker decomposition (TD-SISR). The denoising step is realized _rst by TD in order to mitigate the ill-posedness of the subsequent deconvolution. Compared to CPDSISR the algorithm runs ten times faster. Depending on the amount of noise, higher PSNR (0.3 - 3.5 dB), SSI (0.58 - 2.43%) and segmentation values (Dice coefficient, 2% improvement) were measured. The parameters in TD-SISR are familiar from 2D SVD-based algorithms, so their tuning is easier compared to CPD-SISR

Coherent X-ray di�ffraction imaging of zinc oxide crystals

Zinc Oxide (ZnO) exhibits a plethora of physical properties potentially advantageous in many roles and is why it one of the most studied semiconductor compounds. When doped or in its intrinsic state ZnO demonstrates a multitude of electronic, optical and magnetic properties in a large variety of manufacturable morphologies. Thus it is inherently important to understand why these properties arise and the impact potentially invasive sample preparation methods have for both the function and durability of the material and its devices. Coherent X-ray Diff�raction Imaging (CXDI) is a recently established non-destructive technique which can probe the whole three dimensional structure of small crystalline materials and has the potential for sub angstrom strain resolution. The iterative methods employed to overcome the `phase problem' are described fully. CXDI studies of wurtzite ZnO crystals in the rod morphology with high aspect ratio are presented. ZnO rods synthesised via Chemical Vapour Transport Deposition were studied in post growth state and during in-situ modifi�cation via metal evaporation processing and annealing. Small variations in post growth state were observed, the physical origin of which remains unidentifi�ed. The doping of a ZnO crystal with Iron, Nickel and Cobalt by thermal evaporation and subsequent annealing was studied. The evolution of diff�using ions into the crystal lattice from was not observed, decomposition was found to be the dominant process. Improvements in experimental technique allowed multiple Bragg reflections from a single ZnO crystal to be measured for the fi�rst time. Large aspect ratio ZnO rods were used to probe the coherence properties of the incident beam. The longitudinal coherence function of the illuminating radiation was mapped using the visibility of the interference pattern at each bragg reflection and an accurate estimate of the longitudinal coherence length obtained, \xi(L) = 0.66\pm 0.02 \mu m. The consequences for data analysis are discussed. The combination of multiple Bragg reflections to realise three dimensional displacement �fields was also approached