Color Filter Array Image Analysis for Joint Denoising and Demosaicking

Noise is among the worst artifacts that affect the perceptual quality of the output from a digital camera. While cost-effective and popular, single-sensor solutions to camera architectures are not adept at noise suppression. In this scheme, data are typically obtained via a spatial subsampling procedure implemented as a color filter array (CFA), a physical construction whereby each pixel location measures the intensity of the light corresponding to only a single color. Aside from undersampling, observations made under noisy conditions typically deteriorate the estimates of the full-color image in the reconstruction process commonly referred to as demosaicking or CFA interpolation in the literature. A typical CFA scheme involves the canonical color triples (i.e., red, green, blue), and the most prevalent arrangement is called Bayer pattern. As the general trend of increased image resolution continues due to prevalence of multimedia, the importance of interpolation is de-emphasized while the concerns for computational efficiency, noise, and color fidelity play an increasingly prominent role in the decision making of a digital camera architect. For instance, the interpolation artifacts become less noticeable as the size of the pixel shrinks with respect to the image features, while the decreased dimensionality of the pixel sensors on the complementary metal oxide semiconductor (CMOS) and charge coupled device (CCD) sensors make the pixels more susceptible to noise. Photon-limited influences are also evident in low-light photography, ranging from a specialty camera for precision measurement to indoor consumer photography. Sensor data, which can be interpreted as subsampled or incomplete image data, undergo a series of image processing procedures in order to produce a digital photograph. However, these same steps may amplify noise introduced during image acquisition. Specifically, the demosaicking step is a major source of conflict between the image processing pipeline and image sensor noise characterization because the interpolation methods give high priority to preserving the sharpness of edges and textures. In the presence of noise, noise patterns may form false edge structures; therefore, the distortions at the output are typically correlated with the signal in a complicated manner that makes noise modelling mathematically intractable. Thus, it is natural to conceive of a rigorous tradeoff between demosaicking and image denoising

Sparse Modeling for Image and Vision Processing

In recent years, a large amount of multi-disciplinary research has been conducted on sparse models and their applications. In statistics and machine learning, the sparsity principle is used to perform model selection---that is, automatically selecting a simple model among a large collection of them. In signal processing, sparse coding consists of representing data with linear combinations of a few dictionary elements. Subsequently, the corresponding tools have been widely adopted by several scientific communities such as neuroscience, bioinformatics, or computer vision. The goal of this monograph is to offer a self-contained view of sparse modeling for visual recognition and image processing. More specifically, we focus on applications where the dictionary is learned and adapted to data, yielding a compact representation that has been successful in various contexts.Comment: 205 pages, to appear in Foundations and Trends in Computer Graphics and Visio

Informative sensing : theory and applications

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2012.Cataloged from PDF version of thesis.Includes bibliographical references (p. 145-156).Compressed sensing is a recent theory for the sampling and reconstruction of sparse signals. Sparse signals only occupy a tiny fraction of the entire signal space and thus have a small amount of information, relative to their dimension. The theory tells us that the information can be captured faithfully with few random measurement samples, even far below the Nyquist rate. Despite the successful story, we question how the theory would change if we had a more precise prior than the simple sparsity model. Hence, we consider the settings where the prior is encoded as a probability density. In a Bayesian perspective, we see the signal recovery as an inference, in which we estimate the unmeasured dimensions of the signal given the incomplete measurements. We claim that good sensors should somehow be designed to minimize the uncertainty of the inference. In this thesis, we primarily use Shannon's entropy to measure the uncertainty and in effect pursue the InfoMax principle, rather than the restricted isometry property, in optimizing the sensors. By approximate analysis on sparse signals, we found random projections, typical in the compressed sensing literature, to be InfoMax optimal if the sparse coefficients are independent and identically distributed (i.i.d.). If not, however, we could find a different set of projections which, in signal reconstruction, consistently outperformed random or other types of measurements. In particular, if the coefficients are groupwise i.i.d., groupwise random projections with nonuniform sampling rate per group prove asymptotically Info- Max optimal. Such a groupwise i.i.d. pattern roughly appears in natural images when the wavelet basis is partitioned into groups according to the scale. Consequently, we applied the groupwise random projections to the sensing of natural images. We also considered designing an optimal color filter array for single-chip cameras. In this case, the feasible set of projections is highly restricted because multiplexing across pixels is not allowed. Nevertheless, our principle still applies. By minimizing the uncertainty of the unmeasured colors given the measured ones, we could find new color filter arrays which showed better demosaicking performance in comparison with Bayer or other existing color filter arrays.by Hyun Sung Chang.Ph.D

Kuva-anturien tunnistaminen valovasteen epäyhdenmukaisuutta hyödyntäen

This thesis shows a method to identify a camera source by examining the noise inherent to the imaging process of the camera. The noise is caused by the imaging hardware, e.g. physical properties of charge-coupled device (CCD), the lens, and the Bayer pattern filter. The noise is then altered by the algorithms of the imaging pipeline. After the imaging pipeline, the noise can be isolated from the image by calculating the difference between noisy and denoised image. Noise can be used to form a camera fingerprint by calculating mean noise of a number of training images from same camera, pixel by pixel. The fingerprint can be used to identify the camera by calculating the correlation coefficient between the fingerprints from the cameras and a test image. The image is then assigned to the camera with highest correlation. The key factors affecting the recognition accuracy and stability are the de- noising algorithm and number of training images. It was shown that the best results are achieved with 60 training images and wavelet filter. This thesis evaluates the identification process in four cases. Firstly, between cameras chosen so that each is from different model. Secondly, between different individual cameras from the same model. Thirdly, between all individual cameras without considering the camera model. Finally, forming a fingerprint from one camera from each model, and then using them to identify the rest of the cameras from that model. It was shown that in the first two cases the identification process is feasible, accurate and reasonably stabile. In the latter two cases, the identification process failed to achieve sufficient accuracy to be feasible.Tässä työssä esitetään menetelmä kuvalähteenä olevan kameran tunnistamiseksi tutkimalla kuvausprosessissa sinällään syntyvää kohinaa. Kohina syntyy kuvauksessa käytettävästä laitteistosta, esim. kuva-anturista (CCD), linssistä ja Bayer-suotimesta. Kohinaa muokkaavat kameran automaattisesti kuvanparannukseen käyttämät algoritmit. Kuvanparannuksen jälkeen kohinan voi eristää muodostamalla erotuksen kohinan sisältävän kuvan ja suodatetun kuvan välillä. Kameran sormenjäljen voi muodostaa laskemalla pikseleittäin keskiarvon opetuskuvien kohinasta. Sormenjälkeä käytetään laskemaan korrelaatio testikuvan ja sormenjäljen välillä. Kuvan ottaneeksi kameraksi tunnistetaan se, jonka sormenjäljen ja testikuvan kohinan välillä on suurin korrelaatio. Tärkeimmät tunnistuksen tarkkuuteen ja vakauteen vaikuttavat tekijät ovat kohinanpoistoalgoritmi ja opetuskuvien määrä. Työssä osoitetaan, että parhaat tulokset saadaan käyttämällä 60:tä opetuskuvaa ja aallokesuodatusta. Tässä työssä arvioidaan tunnistusprosessia neljässä tapauksessa. Ensiksi eri malleista valittujen yksittäisten kameroiden suhteen, toiseksi saman kameramallin yksilöiden välillä, kolmanneksi kaikkien yksittäisten kameroiden välillä jättäen huomiotta kameramallin, ja viimeiseksi pyritään yhtä kameraa käyttäen muodostamaan prototyyppisormenjälki, jolla tunnistaa muut samanmalliset kamerat. Työssä osoitettiin, että kahdessa ensinmainitussa tapauksessa tunnistus toimii riittävän tarkasti ja vakaasti. Jälkimmäisissä kahdessa tapauksessa tunnistus ei saavuttanut riittävää tarkkuutta

Multi-channel coded-aperture photography

Thesis (M. Eng.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2008.Includes bibliographical references (p. 87-89).This thesis describes the multi-channel coded-aperture photography, a modified camera system that can extract an all-focus image of the scene along with a depth estimate over the scene. The modification consists of inserting a set of patterned color filters into the aperture of the camera lens. This work generalizes the previous research on a single-channel coded aperture, by deploying distinct filters in the three primary color channels, in order to cope better with the effect of a Bayer filter and to exploit the correlation among the channels. We derive the model and algorithms for the multi-channel coded aperture, comparing the simulated performance of the reconstruction algorithm against that of the original single-channel coded aperture. We also demonstrate a physical prototype, discussing the challenges arising from the use of multiple filters. We provide a comparison with the single-channel coded aperture in performance, and present results on several scenes of cluttered objects at various depths.by Jongmin Baek.M.Eng

Sensor Signal and Information Processing II

In the current age of information explosion, newly invented technological sensors and software are now tightly integrated with our everyday lives. Many sensor processing algorithms have incorporated some forms of computational intelligence as part of their core framework in problem solving. These algorithms have the capacity to generalize and discover knowledge for themselves and learn new information whenever unseen data are captured. The primary aim of sensor processing is to develop techniques to interpret, understand, and act on information contained in the data. The interest of this book is in developing intelligent signal processing in order to pave the way for smart sensors. This involves mathematical advancement of nonlinear signal processing theory and its applications that extend far beyond traditional techniques. It bridges the boundary between theory and application, developing novel theoretically inspired methodologies targeting both longstanding and emergent signal processing applications. The topic ranges from phishing detection to integration of terrestrial laser scanning, and from fault diagnosis to bio-inspiring filtering. The book will appeal to established practitioners, along with researchers and students in the emerging field of smart sensors processing

Multimedia Social Networks: Game Theoretic Modeling and Equilibrium Analysis

Multimedia content sharing and distribution over multimedia social networks is more popular now than ever before: we download music from Napster, share our images on Flickr, view user-created video on YouTube, and watch peer-to-peer television using Coolstreaming, PPLive and PPStream. Within these multimedia social networks, users share, exchange, and compete for scarce resources such as multimedia data and bandwidth, and thus influence each other's decision and performance. Therefore, to provide fundamental guidelines for the better system design, it is important to analyze the users' behaviors and interactions in a multimedia social network, i.e., how users interact with and respond to each other. Game theory is a mathematical tool that analyzes the strategic interactions among multiple decision makers. It is ideal and essential for studying, analyzing, and modeling the users' behaviors and interactions in social networking. In this thesis, game theory will be used to model users' behaviors in social networks and analyze the corresponding equilibria. Specifically, in this thesis, we first illustrate how to use game theory to analyze and model users' behaviors in multimedia social networks by discussing the following three different scenarios. In the first scenario, we consider a non-cooperative multimedia social network where users in the social network compete for the same resource. We use multiuser rate allocation social network as an example for this scenario. In the second scenario, we consider a cooperative multimedia social network where users in the social network cooperate with each other to obtain the content. We use cooperative peer-to-peer streaming social network as an example for this scenario. In the third scenario, we consider how to use the indirect reciprocity game to stimulate cooperation among users. We use the packet forwarding social network as an example. Moreover, the concept of ``multimedia social networks" can be applied into the field of signal and image processing. If each pixel/sample is treated as a user, then the whole image/signal can be regarded as a multimedia social network. From such a perspective, we introduce a new paradigm for signal and image processing, and develop generalized and unified frameworks for classical signal and image problems. In this thesis, we use image denoising and image interpolation as examples to illustrate how to use game theory to re-formulate the classical signal and image processing problems