On the Information Rates of the Plenoptic Function

The {\it plenoptic function} (Adelson and Bergen, 91) describes the visual information available to an observer at any point in space and time. Samples of the plenoptic function (POF) are seen in video and in general visual content, and represent large amounts of information. In this paper we propose a stochastic model to study the compression limits of the plenoptic function. In the proposed framework, we isolate the two fundamental sources of information in the POF: the one representing the camera motion and the other representing the information complexity of the "reality" being acquired and transmitted. The sources of information are combined, generating a stochastic process that we study in detail. We first propose a model for ensembles of realities that do not change over time. The proposed model is simple in that it enables us to derive precise coding bounds in the information-theoretic sense that are sharp in a number of cases of practical interest. For this simple case of static realities and camera motion, our results indicate that coding practice is in accordance with optimal coding from an information-theoretic standpoint. The model is further extended to account for visual realities that change over time. We derive bounds on the lossless and lossy information rates for this dynamic reality model, stating conditions under which the bounds are tight. Examples with synthetic sources suggest that in the presence of scene dynamics, simple hybrid coding using motion/displacement estimation with DPCM performs considerably suboptimally relative to the true rate-distortion bound.Comment: submitted to IEEE Transactions in Information Theor

Fast Random Access to Wavelet Compressed Volumetric Data Using Hashing

We present a new approach to lossy storage of the coefficients of wavelet transformed data. While it is common to store the coefficients of largest magnitude (and let all other coefficients be zero), we allow a slightly different set of coefficients to be stored. This brings into play a recently proposed hashing technique that allows space efficient storage and very efficient retrieval of coefficients. Our approach is applied to compression of volumetric data sets. For the ``Visible Man'' volume we obtain up to 80% improvement in compression ratio over previously suggested schemes. Further, the time for accessing a random voxel is quite competitive

Accurate and discernible photocollages

There currently exist several techniques for selecting and combining images from a digital image library into a single image so that the result meets certain prespecified visual criteria. Image mosaic methods, first explored by Connors and Trivedi[18], arrange library images according to some tiling arrangement, often a regular grid, so that the combination of images, when viewed as a whole, resembles some input target image. Other techniques, such as Autocollage of Rother et al.[78], seek only to combine images in an interesting and visually pleasing manner, according to certain composition principles, without attempting to approximate any target image. Each of these techniques provide a myriad of creative options for artists who wish to combine several levels of meaning into a single image or who wish to exploit the meaning and symbolism contained in each of a large set of images through an efficient and easy process. We first examine the most notable and successful of these methods, and summarize the advantages and limitations of each. We then formulate a set of goals for an image collage system that combines the advantages of these methods while addressing and mitigating the drawbacks. Particularly, we propose a system for creating photocollages that approximate a target image as an aggregation of smaller images, chosen from a large library, so that interesting visual correspondences between images are exploited. In this way, we allow users to create collages in which multiple layers of meaning are encoded, with meaningful visual links between each layer. In service of this goal, we ensure that the images used are as large as possible and are combined in such a way that boundaries between images are not immediately apparent, as in Autocollage. This has required us to apply a multiscale approach to searching and comparing images from a large database, which achieves both speed and accuracy. We also propose a new framework for color post-processing, and propose novel techniques for decomposing images according to object and texture information

Compressing the illumination-adjustable images with principal component analysis.

Pun-Mo Ho.Thesis (M.Phil.)--Chinese University of Hong Kong, 2003.Includes bibliographical references (leaves 90-95).Abstracts in English and Chinese.Chapter 1 --- Introduction --- p.1Chapter 1.1 --- Background --- p.1Chapter 1.2 --- Existing Approaches --- p.2Chapter 1.3 --- Our Approach --- p.3Chapter 1.4 --- Structure of the Thesis --- p.4Chapter 2 --- Related Work --- p.5Chapter 2.1 --- Compression for Navigation --- p.5Chapter 2.1.1 --- Light Field/Lumigraph --- p.5Chapter 2.1.2 --- Surface Light Field --- p.6Chapter 2.1.3 --- Concentric Mosaics --- p.6Chapter 2.1.4 --- On the Compression --- p.7Chapter 2.2 --- Compression for Relighting --- p.7Chapter 2.2.1 --- Previous Approaches --- p.7Chapter 2.2.2 --- Our Approach --- p.8Chapter 3 --- Image-Based Relighting --- p.9Chapter 3.1 --- Plenoptic Illumination Function --- p.9Chapter 3.2 --- Sampling and Relighting --- p.11Chapter 3.3 --- Overview --- p.13Chapter 3.3.1 --- Codec Overview --- p.13Chapter 3.3.2 --- Image Acquisition --- p.15Chapter 3.3.3 --- Experiment Data Sets --- p.16Chapter 4 --- Data Preparation --- p.18Chapter 4.1 --- Block Division --- p.18Chapter 4.2 --- Color Model --- p.23Chapter 4.3 --- Mean Extraction --- p.24Chapter 5 --- Principal Component Analysis --- p.29Chapter 5.1 --- Overview --- p.29Chapter 5.2 --- Singular Value Decomposition --- p.30Chapter 5.3 --- Dimensionality Reduction --- p.34Chapter 5.4 --- Evaluation --- p.37Chapter 6 --- Eigenimage Coding --- p.39Chapter 6.1 --- Transform Coding --- p.39Chapter 6.1.1 --- Discrete Cosine Transform --- p.40Chapter 6.1.2 --- Discrete Wavelet Transform --- p.47Chapter 6.2 --- Evaluation --- p.49Chapter 6.2.1 --- Statistical Evaluation --- p.49Chapter 6.2.2 --- Visual Evaluation --- p.52Chapter 7 --- Relighting Coefficient Coding --- p.57Chapter 7.1 --- Quantization and Bit Allocation --- p.57Chapter 7.2 --- Evaluation --- p.62Chapter 7.2.1 --- Statistical Evaluation --- p.62Chapter 7.2.2 --- Visual Evaluation --- p.62Chapter 8 --- Relighting --- p.65Chapter 8.1 --- Overview --- p.66Chapter 8.2 --- First-Phase Decoding --- p.66Chapter 8.3 --- Second-Phase Decoding --- p.68Chapter 8.3.1 --- Software Relighting --- p.68Chapter 8.3.2 --- Hardware-Assisted Relighting --- p.71Chapter 9 --- Overall Evaluation --- p.81Chapter 9.1 --- Compression of IAIs --- p.81Chapter 9.1.1 --- Statistical Evaluation --- p.81Chapter 9.1.2 --- Visual Evaluation --- p.86Chapter 9.2 --- Hardware-Assisted Relighting --- p.86Chapter 10 --- Conclusion --- p.89Bibliography --- p.9

Efficient acquisition, representation and rendering of light fields

In this thesis we discuss the representation of three-dimensional scenes using image data (image-based rendering), and more precisely the so-called light field approach. We start with an up-to-date survey on previous work in this young field of research. Then we propose a light field representation based on image data and additional per-pixel depth values. This enables us to reconstruct arbitrary views of the scene in an efficient way and with high quality. Furtermore, we can use the same representation to determine optimal reference views during the acquisition of a light field. We further present the so-called free form parameterization, which allows for a relatively free placement of reference views. Finally, we demonstrate a prototype of the Lumi-Shelf system, which acquires, transmits, and renders the light field of a dynamic scene at multiple frames per second.Diese Doktorarbeit beschäftigt sich mit der Repräsentierung dreidimensionaler Szenen durch Bilddaten (engl. image-based rendering, deutsch bildbasierte Bildsynthese), speziell mit dem Ansatz des sog. Lichtfelds. Nach einem aktuellen Überblick über bisherige Arbeiten in diesem jungen Forschungsgebiet stellen wir eine Datenrepräsentation vor, die auf Bilddaten mit zusätzlichen Tiefenwerten basiert. Damit sind wir in der Lage, beliebige Ansichten der Szene effizient und in hoher Qualität zu rekonstruieren sowie die optimalen Referenz-Ansichten bei der Akquisition eines Lichtfelds zu bestimmen. Weiterhin präsentieren wir die sog. Freiform-Parametrisierung, die eine relativ freie Anordnung der Referenz-Ansichten erlaubt. Abschließend demonstrieren wir einen Prototyp des Lumishelf-Systems, welches die Aufnahme, Übertragung und Darstellung des Lichtfeldes einer dynamischen Szene mit mehreren Bildern pro Sekunde ermöglicht

Efficient acquisition, representation and rendering of light fields

In this thesis we discuss the representation of three-dimensional scenes using image data (image-based rendering), and more precisely the so-called light field approach. We start with an up-to-date survey on previous work in this young field of research. Then we propose a light field representation based on image data and additional per-pixel depth values. This enables us to reconstruct arbitrary views of the scene in an efficient way and with high quality. Furtermore, we can use the same representation to determine optimal reference views during the acquisition of a light field. We further present the so-called free form parameterization, which allows for a relatively free placement of reference views. Finally, we demonstrate a prototype of the Lumi-Shelf system, which acquires, transmits, and renders the light field of a dynamic scene at multiple frames per second.Diese Doktorarbeit beschäftigt sich mit der Repräsentierung dreidimensionaler Szenen durch Bilddaten (engl. image-based rendering, deutsch bildbasierte Bildsynthese), speziell mit dem Ansatz des sog. Lichtfelds. Nach einem aktuellen Überblick über bisherige Arbeiten in diesem jungen Forschungsgebiet stellen wir eine Datenrepräsentation vor, die auf Bilddaten mit zusätzlichen Tiefenwerten basiert. Damit sind wir in der Lage, beliebige Ansichten der Szene effizient und in hoher Qualität zu rekonstruieren sowie die optimalen Referenz-Ansichten bei der Akquisition eines Lichtfelds zu bestimmen. Weiterhin präsentieren wir die sog. Freiform-Parametrisierung, die eine relativ freie Anordnung der Referenz-Ansichten erlaubt. Abschließend demonstrieren wir einen Prototyp des Lumishelf-Systems, welches die Aufnahme, Übertragung und Darstellung des Lichtfeldes einer dynamischen Szene mit mehreren Bildern pro Sekunde ermöglicht