Face superresolution from image sequence

Táto práce se zabývá použitím hlubokého učení neuronových sítí ke zvýšení rozlišení obrázků, které obsahují obličeje. Tato metoda najde uplatnění v různých oblastech, zejména v bezpečnosti, například, při bezpečnostním incidentu, kdy policie potřebuje identifikovat podezřelého z nahraného videa ze sledovací kamery. Cílem této práce je navrhnout minimálně dvě architektury neuronových sítí, které budou pracovat se sekvencí snímků, a porovnat je s metodami zpracování jediného snímku. Pro tento účel je také vytvořena nová trénovací množina, obsahující sekvenci snímku obličeje. Metody zpracování jednoho snímku jsou natrénované na nové množině. Dále jsou navrženy nové metody zvětšení obrázků na základě sekvence snímků. Tyto metody jsou založené na U-Net modelu, který je úspěšný v segmentaci, ale také v superrozlišení. Pro zlepšení architektury byly použity reziduální bloky a jejich modifikace, a navíc také percepční ztrátová funkce, která dovoluje vyhnout se rozmazání a získání více detailů. První čast této práce je věnovana popisu neuronových sítí a některých architektur, jejichž modifikace mohou být použity v superrozlišení. Druhá část se poté zabývá popisem metod pro zvýšení rozlišení obrazu pomocí jednoho snímku, několika snímků a videa. Ve třetí části jsou popsány navržené metody a experimenty a v poslední části porovnaná metod založených na jednom snímku a několika snímcích. Navržené metody jsou schopny získat více detailů v obraze, ale mohou produkovat artefakty. Ty lze ale poté eliminovat pomocí filtru, například Gaussova. Nové metody méně selhávají při detekci obličejů, a to je podstatné u identifikace člověka v případě incidentu.This work is focused on application of deep learning in increasing resolution of images containing face. This can be applied in different fields, including security. For example, in case of incident, the police needs to identify a culprit from the records of security camera. The aim of this work is to propose neural network models, which would work with sequence of frames, and to compare these models with existing methods for a single image super-resolution. For this purpose, a new dataset with sequences of the images with faces is created. The methods for the single super-resolution are trained on the new dataset. The new architectures for multiframe super-resolution are proposed. They are based on U-Net model. This model is successful for segmentation tasks, but it can be also applied for super-resolution tasks. To improve this architecture, the residual blocks and its modification are used. To avoid blurring effect and recover more details, the perceptual loss function is applied. In the first part of this work, the description of neural networks and overview of the architectures, which can be applied in super-resolution, is provided. The second part contains the methods for super-resolution of a single frame, multiframe, video. In the next section, there is a description of proposed architectures and description of the experiment. In the last part of the work, multiframe methods and single frame methods are compared. In the result, the proposed methods recover more details, however, some architectures produce artefacts, which can be reduced using a filter, for example, Gaussian. New methods allow to reduce the number of failed face recognition. This fact is necessary for person identification in case of incidents.

Accurate and robust image superresolution by neural processing of local image representations

Image superresolution involves the processing of an image sequence to generate a still image with higher resolution. Classical approaches, such as bayesian MAP methods, require iterative minimization procedures, with high computational costs. Recently, the authors proposed a method to tackle this problem, based on the use of a hybrid MLP-PNN architecture. In this paper, we present a novel superresolution method, based on an evolution of this concept, to incorporate the use of local image models. A neural processing stage receives as input the value of model coefficients on local windows. The data dimension-ality is firstly reduced by application of PCA. An MLP, trained on synthetic se-quences with various amounts of noise, estimates the high-resolution image data. The effect of varying the dimension of the network input space is exam-ined, showing a complex, structured behavior. Quantitative results are presented showing the accuracy and robustness of the proposed method

A New Method for Superresolution Image Reconstruction Based on Surveying Adjustment

A new method for superresolution image reconstruction based on surveying adjustment method is described in this paper. The main idea of such new method is that a sequence of low-resolution images are taken firstly as observations, and then observation equations are established for the superresolution image reconstruction. The gray function of the object surface can be found by using surveying adjustment method from the observation equations. High-resolution pixel value of the corresponding area can be calculated by using the gray function. The results show that the proposed algorithm converges much faster than that of conventional superresolution image reconstruction method. By using the new method, the visual feeling of reconstructed image can be greatly improved compared to that of iterative back projection algorithm, and its peak signal-to-noise ratio can also be improved by nearly 1 dB higher than the projection onto convex sets algorithm. Furthermore, this method can successfully avoid the ill-posed problems in reconstruction process

Depth Superresolution using Motion Adaptive Regularization

Spatial resolution of depth sensors is often significantly lower compared to that of conventional optical cameras. Recent work has explored the idea of improving the resolution of depth using higher resolution intensity as a side information. In this paper, we demonstrate that further incorporating temporal information in videos can significantly improve the results. In particular, we propose a novel approach that improves depth resolution, exploiting the space-time redundancy in the depth and intensity using motion-adaptive low-rank regularization. Experiments confirm that the proposed approach substantially improves the quality of the estimated high-resolution depth. Our approach can be a first component in systems using vision techniques that rely on high resolution depth information

Multiplane 3D superresolution optical fluctuation imaging

By switching fluorophores on and off in either a deterministic or a stochastic manner, superresolution microscopy has enabled the imaging of biological structures at resolutions well beyond the diffraction limit. Superresolution optical fluctuation imaging (SOFI) provides an elegant way of overcoming the diffraction limit in all three spatial dimensions by computing higher-order cumulants of image sequences of blinking fluorophores acquired with a conventional widefield microscope. So far, three-dimensional (3D) SOFI has only been demonstrated by sequential imaging of multiple depth positions. Here we introduce a versatile imaging scheme which allows for the simultaneous acquisition of multiple focal planes. Using 3D cross-cumulants, we show that the depth sampling can be increased. Consequently, the simultaneous acquisition of multiple focal planes reduces the acquisition time and hence the photo-bleaching of fluorescent markers. We demonstrate multiplane 3D SOFI by imaging the mitochondria network in fixed C2C12 cells over a total volume of $65\times65\times3.5 \mu\textrm{m}^3$ without depth scanning.Comment: 7 pages, 3 figure

A Framework for Fast Image Deconvolution with Incomplete Observations

In image deconvolution problems, the diagonalization of the underlying operators by means of the FFT usually yields very large speedups. When there are incomplete observations (e.g., in the case of unknown boundaries), standard deconvolution techniques normally involve non-diagonalizable operators, resulting in rather slow methods, or, otherwise, use inexact convolution models, resulting in the occurrence of artifacts in the enhanced images. In this paper, we propose a new deconvolution framework for images with incomplete observations that allows us to work with diagonalized convolution operators, and therefore is very fast. We iteratively alternate the estimation of the unknown pixels and of the deconvolved image, using, e.g., an FFT-based deconvolution method. This framework is an efficient, high-quality alternative to existing methods of dealing with the image boundaries, such as edge tapering. It can be used with any fast deconvolution method. We give an example in which a state-of-the-art method that assumes periodic boundary conditions is extended, through the use of this framework, to unknown boundary conditions. Furthermore, we propose a specific implementation of this framework, based on the alternating direction method of multipliers (ADMM). We provide a proof of convergence for the resulting algorithm, which can be seen as a "partial" ADMM, in which not all variables are dualized. We report experimental comparisons with other primal-dual methods, where the proposed one performed at the level of the state of the art. Four different kinds of applications were tested in the experiments: deconvolution, deconvolution with inpainting, superresolution, and demosaicing, all with unknown boundaries.Comment: IEEE Trans. Image Process., to be published. 15 pages, 11 figures. MATLAB code available at https://github.com/alfaiate/DeconvolutionIncompleteOb

A Compressive Multi-Mode Superresolution Display

Compressive displays are an emerging technology exploring the co-design of new optical device configurations and compressive computation. Previously, research has shown how to improve the dynamic range of displays and facilitate high-quality light field or glasses-free 3D image synthesis. In this paper, we introduce a new multi-mode compressive display architecture that supports switching between 3D and high dynamic range (HDR) modes as well as a new super-resolution mode. The proposed hardware consists of readily-available components and is driven by a novel splitting algorithm that computes the pixel states from a target high-resolution image. In effect, the display pixels present a compressed representation of the target image that is perceived as a single, high resolution image.Comment: Technical repor