High performance high quality image demosaicing hardware designs

Since capturing three color channels (red, green, and blue) per pixel increases the cost of digital cameras, most digital cameras capture only one color channel per pixel using a single image sensor. The images pass through a color filter array before being captured by the image sensor. Demosaicing is the process of reconstructing the missing color channels of the pixels in the color filtered image using their available neighboring pixels. There are many image demosaicing algorithms with varying reconstructed image quality and computational complexity. In this thesis, high performance hardware architectures are designed for two high quality image demosaicing algorithms with high computational complexity. The proposed hardware architectures are implemented on an FPGA. A high performance Alternating Projections (AP) image demosaicing hardware is proposed. This is the first AP image demosaicing hardware in the literature. A high performance Enhanced Effective Color Interpolation (EECI) image demosaicing hardware is proposed. This is the first EECI image demosaicing hardware in the literature. The proposed hardware architectures are implemented using Verilog HDL. The Verilog RTL codes are mapped to a Xilinx Virtex 6 FPGA. The proposed FPGA implementations are verified with post place & route simulations. They can process 31 and 94 full HD (1920x1080) images per second, respectively

High performance image demosaicing hardware designs

Most digital cameras capture only one color channel (red, green, or blue) per pixel, because capturing three color channels per pixel would require three image sensors which increases the cost of digital cameras. Therefore, only one image sensor is used, and images pass through a color filter array (CFA) before being captured by the image sensor. Bayer pattern is the most commonly used CFA pattern in digital cameras. Demosaicing is the process of reconstructing the missing color channels of the pixels in the color filtered image using their available neighboring pixels. There are many image demosaicing algorithms with varying reconstructed image quality and computational complexity. In this thesis, high performance hardware architectures are designed for three high quality image demosaicing algorithms, and the proposed hardware architectures are implemented on FPGA. A high performance hardware architecture for Effective Color Interpolation (ECI) demosaicing algorithm is proposed. A modified version of Enhanced ECI demosaicing algorithm and a high performance hardware architecture for this image demosaicing algorithm are proposed. A hybrid ECI and Alternating Projections demosaicing algorithm and a high performance hardware architecture for this image demosaicing algorithm are proposed. The proposed hardware architectures are implemented using Verilog HDL. The Verilog RTL codes are mapped to Xilinx Virtex 6 FPGA. The proposed FPGA implementations are verified with post place & route simulations. They are capable of processing 160, 118, and 119 full HD images per second

Low-Cost Compressive Sensing for Color Video and Depth

A simple and inexpensive (low-power and low-bandwidth) modification is made to a conventional off-the-shelf color video camera, from which we recover {multiple} color frames for each of the original measured frames, and each of the recovered frames can be focused at a different depth. The recovery of multiple frames for each measured frame is made possible via high-speed coding, manifested via translation of a single coded aperture; the inexpensive translation is constituted by mounting the binary code on a piezoelectric device. To simultaneously recover depth information, a {liquid} lens is modulated at high speed, via a variable voltage. Consequently, during the aforementioned coding process, the liquid lens allows the camera to sweep the focus through multiple depths. In addition to designing and implementing the camera, fast recovery is achieved by an anytime algorithm exploiting the group-sparsity of wavelet/DCT coefficients.Comment: 8 pages, CVPR 201

The effect of the color filter array layout choice on state-of-the-art demosaicing

Interpolation from a Color Filter Array (CFA) is the most common method for obtaining full color image data. Its success relies on the smart combination of a CFA and a demosaicing algorithm. Demosaicing on the one hand has been extensively studied. Algorithmic development in the past 20 years ranges from simple linear interpolation to modern neural-network-based (NN) approaches that encode the prior knowledge of millions of training images to fill in missing data in an inconspicious way. CFA design, on the other hand, is less well studied, although still recognized to strongly impact demosaicing performance. This is because demosaicing algorithms are typically limited to one particular CFA pattern, impeding straightforward CFA comparison. This is starting to change with newer classes of demosaicing that may be considered generic or CFA-agnostic. In this study, by comparing performance of two state-of-the-art generic algorithms, we evaluate the potential of modern CFA-demosaicing. We test the hypothesis that, with the increasing power of NN-based demosaicing, the influence of optimal CFA design on system performance decreases. This hypothesis is supported with the experimental results. Such a finding would herald the possibility of relaxing CFA requirements, providing more freedom in the CFA design choice and producing high-quality cameras

Noise Power Spectrum Scene-Dependency in Simulated Image Capture Systems

The Noise Power Spectrum (NPS) is a standard measure for image capture system noise. It is derived traditionally from captured uniform luminance patches that are unrepresentative of pictorial scene signals. Many contemporary capture systems apply non- linear content-aware signal processing, which renders their noise scene-dependent. For scene-dependent systems, measuring the NPS with respect to uniform patch signals fails to characterize with accuracy: i) system noise concerning a given input scene, ii) the average system noise power in real-world applications. The scene- and-process-dependent NPS (SPD-NPS) framework addresses these limitations by measuring temporally varying system noise with respect to any given input signal. In this paper, we examine the scene-dependency of simulated camera pipelines in-depth by deriving SPD-NPSs from fifty test scenes. The pipelines apply either linear or non-linear denoising and sharpening, tuned to optimize output image quality at various opacity levels and exposures. Further, we present the integrated area under the mean of SPD-NPS curves over a representative scene set as an objective system noise metric, and their relative standard deviation area (RSDA) as a metric for system noise scene-dependency. We close by discussing how these metrics can also be computed using scene-and-process- dependent Modulation Transfer Functions (SPD-MTF)

Color reproduction from noisy CFA data of single sensor digital cameras

2007-2008 > Academic research: refereed > Publication in refereed journalVersion of RecordPublishe