Feature Extraction for image super-resolution using finite rate of innovation principles

To understand a real-world scene from several multiview pictures, it is necessary to find the disparities existing between each pair of images so that they are correctly related to one another. This process, called image registration, requires the extraction of some specific information about the scene. This is achieved by taking features out of the acquired images. Thus, the quality of the registration depends largely on the accuracy of the extracted features. Feature extraction can be formulated as a sampling problem for which perfect re- construction of the desired features is wanted. The recent sampling theory for signals with finite rate of innovation (FRI) and the B-spline theory offer an appropriate new frame- work for the extraction of features in real images. This thesis first focuses on extending the sampling theory for FRI signals to a multichannel case and then presents exact sampling results for two different types of image features used for registration: moments and edges. In the first part, it is shown that the geometric moments of an observed scene can be retrieved exactly from sampled images and used as global features for registration. The second part describes how edges can also be retrieved perfectly from sampled images for registration purposes. The proposed feature extraction schemes therefore allow in theory the exact registration of images. Indeed, various simulations show that the proposed extraction/registration methods overcome traditional ones, especially at low-resolution. These characteristics make such feature extraction techniques very appropriate for applications like image super-resolution for which a very precise registration is needed. The quality of the super-resolved images obtained using the proposed feature extraction meth- ods is improved by comparison with other approaches. Finally, the notion of polyphase components is used to adapt the image acquisition model to the characteristics of real digital cameras in order to run super-resolution experiments on real images

Nonrigid reconstruction of 3D breast surfaces with a low-cost RGBD camera for surgical planning and aesthetic evaluation

Accounting for 26% of all new cancer cases worldwide, breast cancer remains the most common form of cancer in women. Although early breast cancer has a favourable long-term prognosis, roughly a third of patients suffer from a suboptimal aesthetic outcome despite breast conserving cancer treatment. Clinical-quality 3D modelling of the breast surface therefore assumes an increasingly important role in advancing treatment planning, prediction and evaluation of breast cosmesis. Yet, existing 3D torso scanners are expensive and either infrastructure-heavy or subject to motion artefacts. In this paper we employ a single consumer-grade RGBD camera with an ICP-based registration approach to jointly align all points from a sequence of depth images non-rigidly. Subtle body deformation due to postural sway and respiration is successfully mitigated leading to a higher geometric accuracy through regularised locally affine transformations. We present results from 6 clinical cases where our method compares well with the gold standard and outperforms a previous approach. We show that our method produces better reconstructions qualitatively by visual assessment and quantitatively by consistently obtaining lower landmark error scores and yielding more accurate breast volume estimates

Super resolution and dynamic range enhancement of image sequences

Camera producers try to increase the spatial resolution of a camera by reducing size of sites on sensor array. However, shot noise causes the signal to noise ratio drop as sensor sites get smaller. This fact motivates resolution enhancement to be performed through software. Super resolution (SR) image reconstruction aims to combine degraded images of a scene in order to form an image which has higher resolution than all observations. There is a demand for high resolution images in biomedical imaging, surveillance, aerial/satellite imaging and high-definition TV (HDTV) technology. Although extensive research has been conducted in SR, attention has not been given to increase the resolution of images under illumination changes. In this study, a unique framework is proposed to increase the spatial resolution and dynamic range of a video sequence using Bayesian and Projection onto Convex Sets (POCS) methods. Incorporating camera response function estimation into image reconstruction allows dynamic range enhancement along with spatial resolution improvement. Photometrically varying input images complicate process of projecting observations onto common grid by violating brightness constancy. A contrast invariant feature transform is proposed in this thesis to register input images with high illumination variation. Proposed algorithm increases the repeatability rate of detected features among frames of a video. Repeatability rate is increased by computing the autocorrelation matrix using the gradients of contrast stretched input images. Presented contrast invariant feature detection improves repeatability rate of Harris corner detector around %25 on average. Joint multi-frame demosaicking and resolution enhancement is also investigated in this thesis. Color constancy constraint set is devised and incorporated into POCS framework for increasing resolution of color-filter array sampled images. Proposed method provides fewer demosaicking artifacts compared to existing POCS method and a higher visual quality in final image

Development Of A High Performance Mosaicing And Super-Resolution Algorithm

In this dissertation, a high-performance mosaicing and super-resolution algorithm is described. The scale invariant feature transform (SIFT)-based mosaicing algorithm builds an initial mosaic which is iteratively updated by the robust super resolution algorithm to achieve the final high-resolution mosaic. Two different types of datasets are used for testing: high altitude balloon data and unmanned aerial vehicle data. To evaluate our algorithm, five performance metrics are employed: mean square error, peak signal to noise ratio, singular value decomposition, slope of reciprocal singular value curve, and cumulative probability of blur detection. Extensive testing shows that the proposed algorithm is effective in improving the captured aerial data and the performance metrics are accurate in quantifying the evaluation of the algorithm

Nanoscale Optical and Correlative Microscopies for Quantitative Characterization of DNA Nanostructures

Methods to engineer nanomaterials and devices with uniquely tailored properties are highly sought after in fields such as manufacturing, medicine, energy, and the environment. The macromolecule deoxyribonucleic acid (DNA) enables programmable self-assembly of nanostructures with near arbitrary shape and size and with unprecedented precision and accuracy. Additionally, DNA can be chemically modified to attach molecules and nanoparticles, providing a means to organize active materials into devices with unique or enhanced properties. One particularly powerful form of DNA-based self-assembly, DNA origami, provides robust structures with the potential for nanometer-scale resolution of addressable sites. DNA origami are assembled from one large DNA scaffold strand and many unique, short staple strands; each staple programmatically binds the scaffold at several distant domains, and the coordinated interactions of many staples with the scaffold act to fold the scaffold into a desired shape. The utility of DNA origami has been demonstrated through multiple applications, such as plasmonic and photonic devices, electronic device patterning, information storage, drug delivery, and biosensors. Despite the promise of DNA nanotechnology, few products have successfully translated from the laboratory to industry. Achieving high yield and high-precision synthesis of stable DNA nanostructures is one of the biggest challenges to applications of DNA nanostructures. For adoption in manufacturing, methods to measure and inspect assembled structures (i.e. metrology) are essential. Common high-resolution imaging techniques used to characterize DNA nanostructures, such as atomic force microscopy and transmission electron microscopy, cannot facilitate high-throughput characterization, and few studies have been directed towards the development of improved methods for nanoscale metrology. DNA-PAINT super-resolution microscopy enables high-resolution, multiplexed imaging of reactive sites on DNA nanostructures and offers the potential for inline optical metrology. In this work, nanoscale metrologies utilizing DNA-PAINT were developed for DNA nanostructures and applied to characterize DNA origami arrays and single site defects on DNA origami. For metrology of DNA origami arrays, an embedded, multiplexed optical super-resolution methodology was developed to characterize the periodic structure and defects of two-dimensional arrays. Images revealed the spatial arrangement of structures within the arrays, internal array defects, and grain boundaries between arrays, enabling the reconstruction of arrays from the images. The nature of the imaging technique is also highly compatible with statistical methods, enabling rapid statistical analysis of synthesis conditions. To obtain a greater understanding of DNA origami defects at the scale of individual strands, correlative super-resolution and atomic force microscopies were enabled through the development of a simple and flexible method to bind DNA origami directly to cover glass, simultaneously passivating the surface to single-stranded DNA. High-resolution, correlative microscopy was performed to characterize DNA origami, and spatial correlation in super-resolution optical and topographic images of 5 nm was achieved, validating correlative microscopy for single strand defect metrology. Investigations of single strand defects showed little correlation to structural defects on DNA origami, revealing that most site defects occur on strands that are present in the structure, contrary to prior reports. In addition, the results suggest that the structural stability of DNA origami was decreased by DNA-PAINT imaging. The presented work demonstrated the development and application of advanced characterization techniques for DNA nanostructures, which will accelerate fundamental research and applications of DNA nanotechnology

Evaluation of Interpolation and Registration Techniques in Magnetic Resonance Image for Orthogonal Plane Super Resolution Reconstruction

Super resolution reconstruction (SRR) combines several perspectives of an image (typically low resolution) in order to reconstruct a more complete and comprehensive (higher resolution) image. The aim is to use this concept on magnetic resonance imaging (MRI) data, for which the standard is to scan in several-plane orientation in a 2D fashion. As a result, clinical MRI, functional MRI (FMRI), diffusion weighted imaging (DWI)/diffusion tensor imaging (DTI), and MR angiography (MRA) tend to have high in- plane resolution but low resolution in the slice-select direction. By combining the 2 scans of the orthogonal plane, new 3D images can be reconstructed. This thesis addresses the principal problem of image quality and considers a novel SRR technique that uses the original information from 3 MRI plane orientations in order to enhance the resolution based on prior knowledge of scanning protocol as it relates to voxel resolution. The procedure for validating the MRI data algorithm is executed using MRI dataset of a human brain. The mean squared error (MSE) and peak signal-to-noise ratio (PSNR) were computed for quantitative assessment, whereas the qualitative assessment was performed by visually comparing the SR images to the original HR

Super-resolução em vídeos de baixa qualidade para aplicações forenses, de vigilância e móveis

Orientadores: Siome Klein Goldenstein, Anderson de Rezende RochaTese (doutorado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Algoritmos de super-resolução (SR) são métodos para obter um aumento da resolução de imagens compostas por pixels. Na super-resolução por múltiplas imagens, um conjunto de imagens de baixa resolução de uma cena é combinado para construir uma imagem de resolução superior. Super-resolução é uma solução barata para superar as limitações dos sistemas de aquisição de imagens, e pode ser útil em diversos casos em que o dispositivo não pode ser melhorado ou substituído - mas em que é possível obter diversas capturas da mesma cena. Neste trabalho, é explorada a super-resolução por múltiplas imagens para imagens naturais, em cenários nos quais é possível obter diversas imagens de uma cena. São propostas cinco variações de um método que explora propriedades geométricas de múltiplas imagens de baixa resolução para combiná-las em uma imagem de resolução superior; duas variações de um método que combina técnicas de inpainting e super-resolução; e mais três variações de um método que utiliza filtros adaptativos e regularização para resolver um problema de mínimos quadrados. Super-resolução por múltiplas imagens é possível quando existe movimento e informações não redundantes entre as imagens de baixa resolução. Entretanto, combiná-las em uma imagem de resolução superior pode não ser computacionalmente viável por técnicas complexas de super-resolução. A primeira aplicação dos métodos propostos é para um conjunto de imagens capturadas pelos dispositivos móveis mais recentes. Este tipo de ambiente requer algoritmos eficazes que sejam executados rapidamente e utilizando baixo consumo de memória. A segunda aplicação é na Ciência Forense. Câmeras de vigilância espalhadas pelas cidades poderiam fornecer dicas importantes para identificar um suspeito, por exemplo, em uma cena de crime. Entretanto, o reconhecimento dos caracteres de placas veiculares é especialmente difícil quando a resolução das imagens é baixa. Por isso, este trabalho também propõe um arcabouço que realiza a super-resolução de placas veiculares em vídeos reais de vigilância, capturados por câmeras de baixa qualidade e não projetadas especificamente para esta tarefa, ajudando o especialista forense a compreender um evento de interesse. O arcabouço realiza todas as etapas necessárias para rastrear, alinhar, reconstruir e reconhecer automaticamente os caracteres de uma placa suspeita. O usuário recebe, como saída, a imagem de alta resolução reconstruída, mais rica em detalhes, e também a sequência de caracteres reconhecida automaticamente nesta imagem. São apresentadas validações quantitativas e qualitativas dos algoritmos propostos e de suas aplicações. Os experimentos mostram, por exemplo, que é possível aumentar o número de caracteres reconhecidos corretamente, colocando o arcabouço proposto como uma ferramenta importante para fornecer aos peritos uma solução para o reconhecimento de placas veiculares sob condições adversas de aquisição. Por fim, também é sugerido o número mínimo de imagens a ser utilizada como entrada em cada aplicaçãoAbstract: Super-resolution (SR) algorithms are methods for achieving high-resolution (HR) enlargements of pixel-based images. In multi-frame super resolution, a set of low-resolution (LR) images of a scene are combined to construct an image with higher resolution. Super resolution is an inexpensive solution to overcome the limitations of image acquisition hardware systems, and can be useful in several cases in which the device cannot be upgraded or replaced, but multiple frames of the same scene can be obtained. In this work, we explore SR possibilities for natural images, in scenarios wherein we have multiple frames of a same scene. We design and develop five variations of an algorithm which rely on exploring geometric properties in order to combine pixels from LR observations into an HR grid; two variations of a method that combines inpainting techniques to multi-frame super resolution; and three variations of an algorithm that uses adaptive filtering and Tikhonov regularization to solve a least-square problem. Multi-frame super resolution is possible when there is motion and non-redundant information from LR observations. However, combining a large number of frames into a higher resolution image may not be computationally feasible by complex super-resolution techniques. The first application of the proposed methods is in consumer-grade photography with a setup in which several low-resolution images gathered by recent mobile devices can be combined to create a much higher resolution image. Such always-on low-power environment requires effective high-performance algorithms, that run fastly and with a low-memory footprint. The second application is in Digital Forensic, with a setup in which low-quality surveillance cameras throughout the cities could provide important cues to identify a suspect vehicle, for example, in a crime scene. However, license-plate recognition is especially difficult under poor image resolutions. Hence, we design and develop a novel, free and open-source framework underpinned by SR and Automatic License-Plate Recognition (ALPR) techniques to identify license-plate characters in low-quality real-world traffic videos, captured by cameras not designed for the ALPR task, aiding forensic analysts in understanding an event of interest. The framework handles the necessary conditions to identify a target license plate, using a novel methodology to locate, track, align, super resolve, and recognize its alphanumerics. The user receives as outputs the rectified and super-resolved license-plate, richer in details, and also the sequence of license-plates characters that have been automatically recognized in the super-resolved image. We present quantitative and qualitative validations of the proposed algorithms and its applications. Our experiments show, for example, that SR can increase the number of correctly recognized characters posing the framework as an important step toward providing forensic experts and practitioners with a solution for the license-plate recognition problem under difficult acquisition conditions. Finally, we also suggest a minimum number of images to use as input in each applicationDoutoradoCiência da ComputaçãoDoutor em Ciência da Computação1197478,146886153996/3-2015CAPESCNP

High-Magnification Digital Image Correlation Techniques for Aged Nuclear Fuel Cladding Testing

Nuclear fuel cladding in light water reactors, often made of zirconium alloys, is naturally made more brittle by exposure to the water coolant during normal reactor operation. However, this embrittlement by zirconium hydrides changes the mechanical behavior of the cladding material, affecting how it will deform and what may cause it to fail. Because the cladding already has different properties in different material directions, mechanical testing also needs to be direction specific. In addition, to understand the effects that these microscale hydride features have, measurements of deforming cladding need to be at a microscale. This dissertation describes several high-magnification innovations and advancements in digital image correlation (DIC), a non-contact method for measuring displacement and strain of test specimens during experiments. First, a high-magnification UV lens is demonstrated to be capable of DIC measurements with improved spatial resolution and at high temperatures. Second, previously developed super resolution imaging techniques are applied to DIC measurements of directional ring test specimens, again improving resolution and measurement quality. Third, image capture settings are optimized to balance a tradeoff between poor depth of field and the diffraction of light, both of which cause blurred images and poorer DIC measurements. Fourth, several test arrangements are analyzed with computer modelling to determine the best method for directional tests of the cladding. Finally, the techniques are used to perform high-magnification tension tests for hydrided ring cladding specimens

Feature Extraction for Image Super-resolution using Finite Rate of Innovation Principles

To understand a real-world scene from several multiview pictures, it is necessary to find the disparities existing between each pair of images so that they are correctly related to one another., This process. called image registration, reguires the extraction of some specific information about the scene. This is achieved by taking features out of the acquired imaqes. Thus, the quality of the, registration depends largely on the accuracy of the extracted features. Feature extraction can be formulated as a sampling problem for which perfect reconstruction of the, desired features is wanted. The recent sampling theory for signals with finite rate of innovation (FR/), and the B-spline theory offer an appropriate new framework for the extraction of features in real, images. This thesis first focuses on extending the sampling theory for FRI signals to a multichannel, case and then presents exact sampling results for two different types of image features used for, registration: moments and edges. In the first part, it is shown that the geometric moments of an observed scene can be retrieved exactly from sampled images and used as global features for registration. The second part describes how edges can also be retrieved perfectly from sampled images for registration purposes. The proposed feature extraction schemes therefore allow in theory the exact registration of images. Indeed, various simulations show that the proposed extraction/registration methods overcome traditional ones, especially at low-resolution. These characteristics make such feature extraction techniques very appropriate for applications like image super-resolution for which a very precise registration is needed. The quality of the superresolved images obtained using the proposed feature extraction methods is improved by comparison with other approaches. Finally, the notion of polyphase components is used to adapt the imaqe acquisition model to the characteristics of real digital cameras in order to run super-resolution experiments on real images