Multiframe Super-Resolution of Color Image Sequences Using a Global Motion Model

The development of efficient software tools capable of super- resolving multi-spectral image sequences on-the-fly is an important step toward the production of imaging systems capable of acquiring vital imagery of hostile environments at an affordable price. A number of image processing tools already available for use in target recognition and identification rely on the availability of high-resolution imagery which cannot be safely acquired at a reasonable price. This thesis investigates the use of multiframe super-resolution as a tool to increase the spatial resolution of image sequences acquired with sensors commonly used in consumer video cameras. Multiframe super-resolution is the branch of imaging science which tries to restore high-resolution estimates of a scene utilizing a sequence of under-sampled images of that scene. Although a number of algorithms have already been developed to deal with this problem, they have unfortunately not been extended to deal with multi-spectral images acquired from moving imaging platforms. This thesis performs such extension for one of the most successful super-resolution algorithm and demonstrates that it can be used to improve the performance of common multi-spectral imaging systems utilizing Color Filter Arrays to acquire spectral data

Super resolution and dynamic range enhancement of image sequences

Camera producers try to increase the spatial resolution of a camera by reducing size of sites on sensor array. However, shot noise causes the signal to noise ratio drop as sensor sites get smaller. This fact motivates resolution enhancement to be performed through software. Super resolution (SR) image reconstruction aims to combine degraded images of a scene in order to form an image which has higher resolution than all observations. There is a demand for high resolution images in biomedical imaging, surveillance, aerial/satellite imaging and high-definition TV (HDTV) technology. Although extensive research has been conducted in SR, attention has not been given to increase the resolution of images under illumination changes. In this study, a unique framework is proposed to increase the spatial resolution and dynamic range of a video sequence using Bayesian and Projection onto Convex Sets (POCS) methods. Incorporating camera response function estimation into image reconstruction allows dynamic range enhancement along with spatial resolution improvement. Photometrically varying input images complicate process of projecting observations onto common grid by violating brightness constancy. A contrast invariant feature transform is proposed in this thesis to register input images with high illumination variation. Proposed algorithm increases the repeatability rate of detected features among frames of a video. Repeatability rate is increased by computing the autocorrelation matrix using the gradients of contrast stretched input images. Presented contrast invariant feature detection improves repeatability rate of Harris corner detector around %25 on average. Joint multi-frame demosaicking and resolution enhancement is also investigated in this thesis. Color constancy constraint set is devised and incorporated into POCS framework for increasing resolution of color-filter array sampled images. Proposed method provides fewer demosaicking artifacts compared to existing POCS method and a higher visual quality in final image

Adaptive Wiener Filter Super-Resolution of Color Filter Array Images

Digital color cameras using a single detector array with a Bayer color filter array (CFA) require interpolation or demosaicing to estimate missing color information and provide full-color images. However, demosaicing does not specifically address fundamental undersampling and aliasing inherent in typical camera designs. Fast non-uniform interpolation based super-resolution (SR) is an attractive approach to reduce or eliminate aliasing and its relatively low computational load is amenable to real-time applications. The adaptive Wiener filter (AWF) SR algorithm was initially developed for grayscale imaging and has not previously been applied to color SR demosaicing. Here, we develop a novel fast SR method for CFA cameras that is based on the AWF SR algorithm and uses global channel-to-channel statistical models. We apply this new method as a stand-alone algorithm and also as an initialization image for a variational SR algorithm. This paper presents the theoretical development of the color AWF SR approach and applies it in performance comparisons to other SR techniques for both simulated and real data

Robust Super-resolution by Fusion of Interpolated Frames for Color and Grayscale Images

Multi-frame super-resolution (SR) processing seeks to overcome undersampling issues that can lead to undesirable aliasing artifacts in imaging systems. A key factor in effective multi-frame SR is accurate subpixel inter-frame registration. Accurate registration is more difficult when frame-to-frame motion does not contain simple global translation and includes locally moving scene objects. SR processing is further complicated when the camera captures full color by using a Bayer color filter array (CFA). Various aspects of these SR challenges have been previously investigated. Fast SR algorithms tend to have difficulty accommodating complex motion and CFA sensors. Furthermore, methods that can tolerate these complexities tend to be iterative in nature and may not be amenable to real-time processing. In this paper, we present a new fast approach for performing SR in the presence of these challenging imaging conditions. We refer to the new approach as Fusion of Interpolated Frames (FIF) SR. The FIF SR method decouples the demosaicing, interpolation, and restoration steps to simplify the algorithm. Frames are first individually demosaiced and interpolated to the desired resolution. Next, FIF uses a novel weighted sum of the interpolated frames to fuse them into an improved resolution estimate. Finally, restoration is applied to improve any degrading camera effects. The proposed FIF approach has a lower computational complexity than many iterative methods, making it a candidate for real-time implementation. We provide a detailed description of the FIF SR method and show experimental results using synthetic and real datasets in both constrained and complex imaging scenarios. Experiments include airborne grayscale imagery and Bayer CFA image sets with affine background motion plus local motion

Robust Super-resolution by Fusion of Interpolated Frames for Color and Grayscale Images

Multi-frame super-resolution (SR) processing seeks to overcome undersampling issues that can lead to undesirable aliasing artifacts in imaging systems. A key factor in effective multi-frame SR is accurate subpixel inter-frame registration. Accurate registration is more difficult when frame-to-frame motion does not contain simple global translation and includes locally moving scene objects. SR processing is further complicated when the camera captures full color by using a Bayer color filter array (CFA). Various aspects of these SR challenges have been previously investigated. Fast SR algorithms tend to have difficulty accommodating complex motion and CFA sensors. Furthermore, methods that can tolerate these complexities tend to be iterative in nature and may not be amenable to real-time processing. In this paper, we present a new fast approach for performing SR in the presence of these challenging imaging conditions. We refer to the new approach as Fusion of Interpolated Frames (FIF) SR. The FIF SR method decouples the demosaicing, interpolation, and restoration steps to simplify the algorithm. Frames are first individually demosaiced and interpolated to the desired resolution. Next, FIF uses a novel weighted sum of the interpolated frames to fuse them into an improved resolution estimate. Finally, restoration is applied to improve any degrading camera effects. The proposed FIF approach has a lower computational complexity than many iterative methods, making it a candidate for real-time implementation. We provide a detailed description of the FIF SR method and show experimental results using synthetic and real datasets in both constrained and complex imaging scenarios. Experiments include airborne grayscale imagery and Bayer CFA image sets with affine background motion plus local motion

Partition-based Interpolation for Color Filter Array Demosaicking and Super-Resolution Reconstruction

A class of partition-based interpolators that addresses a variety of image interpolation applications are proposed. The proposed interpolators first partition an image into a finite set of partitions that capture local image structures. Missing high resolution pixels are then obtained through linear operations on neighboring pixels that exploit the captured image structure. By exploiting the local image structure, the proposed algorithm produces excellent performance on both edge and uniform regions. The presented results demonstrate that partition-based interpolation yields results superior to traditional and advanced algorithms in the applications of color filter array (CFA) demosaicking and super-resolution reconstruction

De-velopment of Demosaicking Techniques for Multi-Spectral Imaging Using Mosaic Focal Plane Arrays

The use of mosaicked array technology in commercial digital cameras has madethem smaller, cheaper and mechanically more robust. In a mosaicked sensor, each pixel detector is covered with a wavelength-specific optical filter. Since only one spectral band is sensed per pixel location, there is an absence of information from the rest of the spectral bands. These unmeasured spectral bands are estimated by using information obtained from the neighborhood pixels. This process of estimating the unmeasured spectral band information is called demosaicking. The demosaicking process uses interpolation strategies to estimate the missing pixels. Sophisticated interpolation methods have been developed for performing this task in digital color cameras.In this thesis we propose to evaluate the adaptation of the mosaicked technol- ogy for multi-spectral cameras. Existing multi-spectral cameras use traditional methods like imaging spectrometers to capture a multi-spectral image. These methods are very expensive and delicate in nature. The objective of using the mosaicked technology for multi-spectral cameras is to reap the same benefits it offers in the commercial digital color cameras. However, the problem in using the mosaicked technology for multi-spectral images is the huge amount of missing pixels that need to be estimated in order to form the multi-spectral image. The estimation process becomes even more complicated as the number of bands in the multi-spectral image increases. Traditional demosaicking algorithms cannot be used because they have been specifically designed to suit three-band color images.This thesis focuses on developing new demosaicking algorithms for multi- spectral images. The existing demosaicking algorithms for color images have been extended for multi-spectral images. A new variation of the bilinear interpolationbased strategy has been developed to perform demosaicking. This demosaicking method uses variable neighborhood definitions to interpolate the missing spectral band values at each pixel locations in a multi-spectral image. A novel Maximum a-Posteriori (MAP) based demosaicking method has also been developed. This method treats demosaicking as an image restoration problem. It can derive op- timal estimation result that resembles the original image the best. In addition, it can simultaneously perform interpolation of missing spectral bands at pixel locations and also remove noise and degradations in the image.Extensive experimentation and comparisons have shown that the new demo- saicking methods for multi-spectral images developed in this thesis perform better than the traditional interpolation trategies. The outputs from the demosaicking methods have been shown to be better reconstructed estimates of the original im- ages and also have the ability to produce good classification results in applicationslike target recognition and discrimination