The Plenoptic videos: Capturing, Rendering and Compression

This paper presents a system for capturing and rendering a dynamic image-based representation called the plenoptic videos. It is a simplified version of light fields for dynamic environment, where user viewpoints are constrained along the camera plane of a linear array of video cameras. The system consists of a camera array of 8 Sony CCX-Z11 CCD cameras and eight Pentium 41.8 GHz computers connected together through a 100 baseT LAN. Important issues such as multiple camera calibration, real-time compression, decompression and rendering are addressed. Experimental results demonstrated the usefulness of the proposed parallel processing based system in capturing and rendering high quality dynamic image-based representation using off-the-shelf equipment, and its potential applications in visualization and immersive television systems.published_or_final_versio

RLFC: Random Access Light Field Compression using Key Views and Bounded Integer Encoding

We present a new hierarchical compression scheme for encoding light field images (LFI) that is suitable for interactive rendering. Our method (RLFC) exploits redundancies in the light field images by constructing a tree structure. The top level (root) of the tree captures the common high-level details across the LFI, and other levels (children) of the tree capture specific low-level details of the LFI. Our decompressing algorithm corresponds to tree traversal operations and gathers the values stored at different levels of the tree. Furthermore, we use bounded integer sequence encoding which provides random access and fast hardware decoding for compressing the blocks of children of the tree. We have evaluated our method for 4D two-plane parameterized light fields. The compression rates vary from 0.08 - 2.5 bits per pixel (bpp), resulting in compression ratios of around 200:1 to 20:1 for a PSNR quality of 40 to 50 dB. The decompression times for decoding the blocks of LFI are 1 - 3 microseconds per channel on an NVIDIA GTX-960 and we can render new views with a resolution of 512X512 at 200 fps. Our overall scheme is simple to implement and involves only bit manipulations and integer arithmetic operations.Comment: Accepted for publication at Symposium on Interactive 3D Graphics and Games (I3D '19

A multi-camera approach to image-based rendering and 3-D/Multiview display of ancient chinese artifacts

published_or_final_versio

Data compression and transmission aspects of panoramic videos

Panoramic videos are effective means for representing static or dynamic scenes along predefined paths. They allow users to change their viewpoints interactively at points in time or space defined by the paths. High-resolution panoramic videos, while desirable, consume a significant amount of storage and bandwidth for transmission. They also make real-time decoding computationally very intensive. This paper proposes efficient data compression and transmission techniques for panoramic videos. A high-performance MPEG-2-like compression algorithm, which takes into account the random access requirements and the redundancies of panoramic videos, is proposed. The transmission aspects of panoramic videos over cable networks, local area networks (LANs), and the Internet are also discussed. In particular, an efficient advanced delivery sharing scheme (ADSS) for reducing repeated transmission and retrieval of frequently requested video segments is introduced. This protocol was verified by constructing an experimental VOD system consisting of a video server and eight Pentium 4 computers. Using the synthetic panoramic video Village at a rate of 197 kb/s and 7 f/s, nearly two-thirds of the memory access and transmission bandwidth of the video server were saved under normal network traffic.published_or_final_versio

Survey of image-based representations and compression techniques

In this paper, we survey the techniques for image-based rendering (IBR) and for compressing image-based representations. Unlike traditional three-dimensional (3-D) computer graphics, in which 3-D geometry of the scene is known, IBR techniques render novel views directly from input images. IBR techniques can be classified into three categories according to how much geometric information is used: rendering without geometry, rendering with implicit geometry (i.e., correspondence), and rendering with explicit geometry (either with approximate or accurate geometry). We discuss the characteristics of these categories and their representative techniques. IBR techniques demonstrate a surprising diverse range in their extent of use of images and geometry in representing 3-D scenes. We explore the issues in trading off the use of images and geometry by revisiting plenoptic-sampling analysis and the notions of view dependency and geometric proxies. Finally, we highlight compression techniques specifically designed for image-based representations. Such compression techniques are important in making IBR techniques practical.published_or_final_versio

Compression of Lumigraph with Multiple Reference Frame (MRF) Prediction and Just-in-time Rendering

In the form of 2D image array, Lumigraph captures the complete appearance of an object or a scene, and is able to quickly render a novel view independent of the scene/object complexity. Since the data amount of Lumigraph is huge, the efficient storage and access of Lumigraph are essential. In this paper, we propose a multiple reference frame (MRF) structure to compress the Lumigraph data. By predicting the Lumigraph view from multiple neighbor views, a higher compression ratio is achieved. We also implement the key functionality of just-in-time (JIT) Lumigraph rendering, in which only a small portion of the compressed bitstream necessary for rendering the current view is accessed and decoded. JIT rendering eliminates the need to predecode the entire Lumigraph data set, thus greatly reduces the memory requirement of Lumigraph rendering. A decoder cache has been implemented to speed up rendering by reus ing the decoded data. The trade off between the computational speed and cache size of the decoder is discussed in the paper. KEYWORDS: image-based rendering (IBR), Lumigraph, multiple reference frame (MRF) structure, data compression, just-in-time (JIT) rendering 1

Compressing the illumination-adjustable images with principal component analysis.

Pun-Mo Ho.Thesis (M.Phil.)--Chinese University of Hong Kong, 2003.Includes bibliographical references (leaves 90-95).Abstracts in English and Chinese.Chapter 1 --- Introduction --- p.1Chapter 1.1 --- Background --- p.1Chapter 1.2 --- Existing Approaches --- p.2Chapter 1.3 --- Our Approach --- p.3Chapter 1.4 --- Structure of the Thesis --- p.4Chapter 2 --- Related Work --- p.5Chapter 2.1 --- Compression for Navigation --- p.5Chapter 2.1.1 --- Light Field/Lumigraph --- p.5Chapter 2.1.2 --- Surface Light Field --- p.6Chapter 2.1.3 --- Concentric Mosaics --- p.6Chapter 2.1.4 --- On the Compression --- p.7Chapter 2.2 --- Compression for Relighting --- p.7Chapter 2.2.1 --- Previous Approaches --- p.7Chapter 2.2.2 --- Our Approach --- p.8Chapter 3 --- Image-Based Relighting --- p.9Chapter 3.1 --- Plenoptic Illumination Function --- p.9Chapter 3.2 --- Sampling and Relighting --- p.11Chapter 3.3 --- Overview --- p.13Chapter 3.3.1 --- Codec Overview --- p.13Chapter 3.3.2 --- Image Acquisition --- p.15Chapter 3.3.3 --- Experiment Data Sets --- p.16Chapter 4 --- Data Preparation --- p.18Chapter 4.1 --- Block Division --- p.18Chapter 4.2 --- Color Model --- p.23Chapter 4.3 --- Mean Extraction --- p.24Chapter 5 --- Principal Component Analysis --- p.29Chapter 5.1 --- Overview --- p.29Chapter 5.2 --- Singular Value Decomposition --- p.30Chapter 5.3 --- Dimensionality Reduction --- p.34Chapter 5.4 --- Evaluation --- p.37Chapter 6 --- Eigenimage Coding --- p.39Chapter 6.1 --- Transform Coding --- p.39Chapter 6.1.1 --- Discrete Cosine Transform --- p.40Chapter 6.1.2 --- Discrete Wavelet Transform --- p.47Chapter 6.2 --- Evaluation --- p.49Chapter 6.2.1 --- Statistical Evaluation --- p.49Chapter 6.2.2 --- Visual Evaluation --- p.52Chapter 7 --- Relighting Coefficient Coding --- p.57Chapter 7.1 --- Quantization and Bit Allocation --- p.57Chapter 7.2 --- Evaluation --- p.62Chapter 7.2.1 --- Statistical Evaluation --- p.62Chapter 7.2.2 --- Visual Evaluation --- p.62Chapter 8 --- Relighting --- p.65Chapter 8.1 --- Overview --- p.66Chapter 8.2 --- First-Phase Decoding --- p.66Chapter 8.3 --- Second-Phase Decoding --- p.68Chapter 8.3.1 --- Software Relighting --- p.68Chapter 8.3.2 --- Hardware-Assisted Relighting --- p.71Chapter 9 --- Overall Evaluation --- p.81Chapter 9.1 --- Compression of IAIs --- p.81Chapter 9.1.1 --- Statistical Evaluation --- p.81Chapter 9.1.2 --- Visual Evaluation --- p.86Chapter 9.2 --- Hardware-Assisted Relighting --- p.86Chapter 10 --- Conclusion --- p.89Bibliography --- p.9