Robust Super-resolution by Fusion of Interpolated Frames for Color and Grayscale Images

Multi-frame super-resolution (SR) processing seeks to overcome undersampling issues that can lead to undesirable aliasing artifacts in imaging systems. A key factor in effective multi-frame SR is accurate subpixel inter-frame registration. Accurate registration is more difficult when frame-to-frame motion does not contain simple global translation and includes locally moving scene objects. SR processing is further complicated when the camera captures full color by using a Bayer color filter array (CFA). Various aspects of these SR challenges have been previously investigated. Fast SR algorithms tend to have difficulty accommodating complex motion and CFA sensors. Furthermore, methods that can tolerate these complexities tend to be iterative in nature and may not be amenable to real-time processing. In this paper, we present a new fast approach for performing SR in the presence of these challenging imaging conditions. We refer to the new approach as Fusion of Interpolated Frames (FIF) SR. The FIF SR method decouples the demosaicing, interpolation, and restoration steps to simplify the algorithm. Frames are first individually demosaiced and interpolated to the desired resolution. Next, FIF uses a novel weighted sum of the interpolated frames to fuse them into an improved resolution estimate. Finally, restoration is applied to improve any degrading camera effects. The proposed FIF approach has a lower computational complexity than many iterative methods, making it a candidate for real-time implementation. We provide a detailed description of the FIF SR method and show experimental results using synthetic and real datasets in both constrained and complex imaging scenarios. Experiments include airborne grayscale imagery and Bayer CFA image sets with affine background motion plus local motion

Robust Super-resolution by Fusion of Interpolated Frames for Color and Grayscale Images

Multi-frame super-resolution (SR) processing seeks to overcome undersampling issues that can lead to undesirable aliasing artifacts in imaging systems. A key factor in effective multi-frame SR is accurate subpixel inter-frame registration. Accurate registration is more difficult when frame-to-frame motion does not contain simple global translation and includes locally moving scene objects. SR processing is further complicated when the camera captures full color by using a Bayer color filter array (CFA). Various aspects of these SR challenges have been previously investigated. Fast SR algorithms tend to have difficulty accommodating complex motion and CFA sensors. Furthermore, methods that can tolerate these complexities tend to be iterative in nature and may not be amenable to real-time processing. In this paper, we present a new fast approach for performing SR in the presence of these challenging imaging conditions. We refer to the new approach as Fusion of Interpolated Frames (FIF) SR. The FIF SR method decouples the demosaicing, interpolation, and restoration steps to simplify the algorithm. Frames are first individually demosaiced and interpolated to the desired resolution. Next, FIF uses a novel weighted sum of the interpolated frames to fuse them into an improved resolution estimate. Finally, restoration is applied to improve any degrading camera effects. The proposed FIF approach has a lower computational complexity than many iterative methods, making it a candidate for real-time implementation. We provide a detailed description of the FIF SR method and show experimental results using synthetic and real datasets in both constrained and complex imaging scenarios. Experiments include airborne grayscale imagery and Bayer CFA image sets with affine background motion plus local motion

Adaptive Wiener Filter Super-Resolution of Color Filter Array Images

Digital color cameras using a single detector array with a Bayer color filter array (CFA) require interpolation or demosaicing to estimate missing color information and provide full-color images. However, demosaicing does not specifically address fundamental undersampling and aliasing inherent in typical camera designs. Fast non-uniform interpolation based super-resolution (SR) is an attractive approach to reduce or eliminate aliasing and its relatively low computational load is amenable to real-time applications. The adaptive Wiener filter (AWF) SR algorithm was initially developed for grayscale imaging and has not previously been applied to color SR demosaicing. Here, we develop a novel fast SR method for CFA cameras that is based on the AWF SR algorithm and uses global channel-to-channel statistical models. We apply this new method as a stand-alone algorithm and also as an initialization image for a variational SR algorithm. This paper presents the theoretical development of the color AWF SR approach and applies it in performance comparisons to other SR techniques for both simulated and real data

Block Matching and Wiener Filtering Approach to Optical Turbulence Mitigation and Its Application to Simulated and Real Imagery with Quantitative Error Analysis

We present a block-matching and Wiener filtering approach to atmospheric turbulence mitigation for long-range imaging of extended scenes. We evaluate the proposed method, along with some benchmark methods, using simulated and real-image sequences. The simulated data are generated with a simulation tool developed by one of the authors. These data provide objective truth and allow for quantitative error analysis. The proposed turbulence mitigation method takes a sequence of short-exposure frames of a static scene and outputs a single restored image. A block-matching registration algorithm is used to provide geometric correction for each of the individual input frames. The registered frames are then averaged, and the average image is processed with a Wiener filter to provide deconvolution. An important aspect of the proposed method lies in how we model the degradation point spread function (PSF) for the purposes of Wiener filtering. We use a parametric model that takes into account the level of geometric correction achieved during image registration. This is unlike any method we are aware of in the literature. By matching the PSF to the level of registration in this way, the Wiener filter is able to fully exploit the reduced blurring achieved by registration. We also describe a method for estimating the atmospheric coherence diameter (or Fried parameter) from the estimated motion vectors. We provide a detailed performance analysis that illustrates how the key tuning parameters impact system performance. The proposed method is relatively simple computationally, yet it has excellent performance in comparison with state-of-the-art benchmark methods in our study

Deep learning for anisoplanatic optical turbulence mitigation in long-range imaging

We present a deep learning approach for restoring images degraded by atmospheric optical turbulence. We consider the case of terrestrial imaging over long ranges with a wide field-of-view. This produces an anisoplanatic imaging scenario where turbulence warping and blurring vary spatially across the image. The proposed turbulence mitigation (TM) method assumes that a sequence of short-exposure images is acquired. A block matching (BM) registration algorithm is applied to the observed frames for dewarping, and the resulting images are averaged. A convolutional neural network (CNN) is then employed to perform spatially adaptive restoration. We refer to the proposed TM algorithm as the block matching and CNN (BM-CNN) method. Training the CNN is accomplished using simulated data from a fast turbulence simulation tool capable of producing a large amount of degraded imagery from declared truth images rapidly. Testing is done using independent data simulated with a different well-validated numerical wave-propagation simulator. Our proposed BM-CNN TM method is evaluated in a number of experiments using quantitative metrics. The quantitative analysis is made possible by virtue of having truth imagery from the simulations. A number of restored images are provided for subjective evaluation. We demonstrate that the BM-CNN TM method outperforms the benchmark methods in the scenarios tested

Super-resolution in the presence of atmospheric optical turbulence

The design of imaging systems involves navigating a complex trade space. As a result, many imaging systems employ focal plane arrays with a detector pitch that is insufficient to meet the Nyquist sampling criterion under diffraction-limited imaging conditions. This undersampling may result in aliasing artifacts and prevent the imaging system from achieving the full resolution afforded by the optics. Another potential source of image degradation, especially for long-range imaging, is atmospheric optical turbulence. Optical turbulence gives rise to spatially and temporally varying image blur and warping from fluctuations in the index of refraction along with optical path. Under heavy turbulence, the blurring from the turbulence acts as an anti-aliasing filter, and undersampling does not generally occur. However, under light to moderate turbulence, many imaging systems will exhibit both aliasing artifacts and turbulence degradation. Few papers in the literature have analyzed or addressed both of these degradations together. In this paper, we provide a novel analysis of undersampling in the presence of optical turbulence. Specifically, we provide an optical transfer function analysis that illustrates regimes where aliasing and turbulence are both present, and where they are not. We also propose and evaluate a super-resolution (SR) method for combating aliasing that offers robustness to optical turbulence. The method has a tuning parameter that allows it to transition from traditional diffraction-limited SR, to pure turbulence mitigation with no SR. The proposed method is based on Fusion of Interpolated Frames (FIF) SR, recently proposed by two of the current authors. We quantitatively evaluate the SR method with varying levels of optical turbulence using simulated sequences. We also presented results using real infrared imagery

Fusion of interpolated frames superresolution in the presence of atmospheric optical turbulence

An extension of the fusion of interpolated frames superresolution (FIF SR) method to perform SR in the presence of atmospheric optical turbulence is presented. The goal of such processing is to improve the performance of imaging systems impacted by turbulence. We provide an optical transfer function analysis that illustrates regimes where significant degradation from both aliasing and turbulence may be present in imaging systems. This analysis demonstrates the potential need for simultaneous SR and turbulence mitigation (TM). While the FIF SR method was not originally proposed to address this joint restoration problem, we believe it is well suited for this task. We propose a variation of the FIF SR method that has a fusion parameter that allows it to transition from traditional diffraction-limited SR to pure TM with no SR as well as a continuum in between. This fusion parameter balances subpixel resolution, needed for SR, with the amount of temporal averaging, needed for TM and noise reduction. In addition, we develop a model of the interpolation blurring that results from the fusion process, as a function of this tuning parameter. The blurring model is then incorporated into the overall degradation model that is addressed in the restoration step of the FIF SR method. This innovation benefits the FIF SR method in all applications. We present a number of experimental results to demonstrate the efficacy of the FIF SR method in different levels of turbulence. Simulated imagery with known ground truth is used for a detailed quantitative analysis. Three real infrared image sequences are also used. Two of these include bar targets that allow for a quantitative resolution enhancement assessment

Differential Tilt Variance Effects of Turbulence in Imagery: Comparing Simulation with Theory

Differential tilt variance is a useful metric for interpreting the distorting effects of turbulence in incoherent imaging systems. In this paper, we compare the theoretical model of differential tilt variance to simulations. Simulation is based on a Monte Carlo wave optics approach with split step propagation. Results show that the simulation closely matches theory. The results also show that care must be taken when selecting a method to estimate tilts

Microglial brain region−dependent diversity and selective regional sensitivities to aging

Microglia play critical roles in neural development, homeostasis and neuroinflammation and are increasingly implicated in age-related neurological dysfunction. Neurodegeneration often occurs in disease-specific spatially-restricted patterns, the origins of which are unknown. We performed the first genome-wide analysis of microglia from discrete brain regions across the adult lifespan of the mouse and reveal that microglia have distinct region-dependent transcriptional identities and age in a regionally variable manner. In the young adult brain, differences in bioenergetic and immunoregulatory pathways were the major sources of heterogeneity and suggested that cerebellar and hippocampal microglia exist in a more immune vigilant state. Immune function correlated with regional transcriptional patterns. Augmentation of the distinct cerebellar immunophenotype and a contrasting loss in distinction of the hippocampal phenotype among forebrain regions were key features during ageing. Microglial diversity may enable regionally localised homeostatic functions but could also underlie region-specific sensitivities to microglial dysregulation and involvement in age-related neurodegeneration

Media 1: Adaptive Wiener filter super-resolution of color filter array images

Originally published in Optics Express on 12 August 2013 (oe-21-16-18820