A spectral analysis for light field rendering

Image based rendering using the plenoptic function is an efficient technique for re-rendering at different viewpoints. In this paper, we study the sampling and reconstruction problem of plenoptic function as a multidimensional sampling problem. The spectral support of plenoptic function is found to be an important quantity in the efficient sampling and reconstruction of such function. A spectral analysis for the light field, a 4D plenoptic function, is performed. Its spectrum, as a function of the depth function of the scene, is then derived. This result enables us to estimate the spectral support of the light field given some prior estimate of the depth function. Results using a piecewise constant depth model show significant improvement in rendering of the light field images. The design of the reconstruction filter is also discussed.published_or_final_versio

A spectral analysis for light field rendering

Image based rendering using the plenoptic function is an efficient technique for re-rendering at different viewpoints. In this paper, we study the sampling and reconstruction problem of plenoptic function as a multidimensional sampling problem. The spectral support of plenoptic function is found to be an important quantity in the efficient sampling and reconstruction of such function. A spectral analysis for the light field, a 4D plenoptic function, is performed. Its spectrum, as a function of the depth function of the scene, is then derived. This result enables us to estimate the spectral support of the light field given some prior estimate of the depth function. Results using a piecewise constant depth model show significant improvement in rendering of the light field images. The design of the reconstruction filter is also discussed.published_or_final_versio

Improved methods for object-based coding of plenoptic videos

2005 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS 2005), Hong Kong, 13-16 December 2005Plenoptic videos (PVs) are a class of dynamic image-based representations, where the videos are taken at regularly spaced locations along a line. To yield the better rendering quality in scenes with large depth variations and support the functionalities at the object level for rendering, an object-based coding scheme is employed for the coding of PVs. Upon this object-based coding framework, the paper studies the improved coding methods for the texture and depth coding to achieve better compression efficiency. Experimental results show that considerable improvements in texture coding performance are obtained for both synthetic and real scenes. The improved depth coding quality is also illustrated. © 2005 IEEE.published_or_final_versio

Object tracking and matting for A class of dynamic image-based representations

Image-based rendering (IBR) is an emerging technology for photo-realistic rendering of scenes from a collection of densely sampled images and videos. Recently, an object-based approach for a class of dynamic image-based representations called plenoptic videos was proposed. This paper proposes an automatic object tracking approach using the level-set method. Our tracking method, which utilizes both local and global features of the image sequences instead of global features exploited in previous approach, can achieve better tracking results for objects, especially with non-uniform energy distribution. Due to possible segmentation errors around object boundaries, natural matting with Bayesian approach is also incorporated into our system. Furthermore, a MPEG-4 like object-based algorithm is developed for compressing the plenoptic videos, which consist of the alpha maps, depth maps and textures of the segmented image-based objects from different video plenoptic streams. Experimental results show that satisfactory renderings can be obtained by the proposed approaches. © 2005 IEEE.published_or_final_versio

The compression issues of panoramic video

The paper proposes efficient data compression techniques for panoramic video. Panoramic videos have been used as a means for representing dynamic scenes or paths along a static environment. They allow the user to change viewpoints interactively at a point in time or space. High-resolution panoramic videos, while desirable, consume a significant amount of storage and bandwidth for transmission, and make real-time decoding very computationally intensive. A high performance MPEG-like compression algorithm, which takes into account the random access requirements and the redundancies of the panoramic video, is presented. The transmission aspects of panoramic video over cable network, LAN and Internet are also briefly discussed.published_or_final_versio

The application of nonlinear filter banks to efficient rendering and progressive transmission of light fields

This paper studies the application of perfect reconstruction nonlinear filter banks (NFB) to the efficient rendering and progressive transmission of light fields. The reference pictures in the conventional disparity-compensated prediction encoder is decomposed using the NFB to reduce the amount of main memory needed to support fast rendering. The NFB has very low arithmetic complexity for reconstruction and small filter support which considerably simplifies the random access operations. It can also be applied to the predicted light field images to support progressive transmission. Different prediction and reconstruction strategies are also investigated to achieve different tradeoffs between memory requirement and decoding speed.published_or_final_versio

Image-based rendering and synthesis

Multiview imaging (MVI) is currently the focus of some research as it has a wide range of applications and opens up research in other topics and applications, including virtual view synthesis for three-dimensional (3D) television (3DTV) and entertainment. However, a large amount of storage is needed by multiview systems and are difficult to construct. The concept behind allowing 3D scenes and objects to be visualized in a realistic way without full 3D model reconstruction is image-based rendering (IBR). Using images as the primary substrate, IBR has many potential applications including for video games, virtual travel and others. The technique creates new views of scenes which are reconstructed from a collection of densely sampled images or videos. The IBR concept has different classification such as knowing 3D models and the lighting conditions and be rendered using conventional graphic techniques. Another is lightfield or lumigraph rendering which depends on dense sampling with no or very little geometry for rendering without recovering the exact 3D-models.published_or_final_versio

Data compression and transmission aspects of panoramic videos

Panoramic videos are effective means for representing static or dynamic scenes along predefined paths. They allow users to change their viewpoints interactively at points in time or space defined by the paths. High-resolution panoramic videos, while desirable, consume a significant amount of storage and bandwidth for transmission. They also make real-time decoding computationally very intensive. This paper proposes efficient data compression and transmission techniques for panoramic videos. A high-performance MPEG-2-like compression algorithm, which takes into account the random access requirements and the redundancies of panoramic videos, is proposed. The transmission aspects of panoramic videos over cable networks, local area networks (LANs), and the Internet are also discussed. In particular, an efficient advanced delivery sharing scheme (ADSS) for reducing repeated transmission and retrieval of frequently requested video segments is introduced. This protocol was verified by constructing an experimental VOD system consisting of a video server and eight Pentium 4 computers. Using the synthetic panoramic video Village at a rate of 197 kb/s and 7 f/s, nearly two-thirds of the memory access and transmission bandwidth of the video server were saved under normal network traffic.published_or_final_versio

Survey of image-based representations and compression techniques

In this paper, we survey the techniques for image-based rendering (IBR) and for compressing image-based representations. Unlike traditional three-dimensional (3-D) computer graphics, in which 3-D geometry of the scene is known, IBR techniques render novel views directly from input images. IBR techniques can be classified into three categories according to how much geometric information is used: rendering without geometry, rendering with implicit geometry (i.e., correspondence), and rendering with explicit geometry (either with approximate or accurate geometry). We discuss the characteristics of these categories and their representative techniques. IBR techniques demonstrate a surprising diverse range in their extent of use of images and geometry in representing 3-D scenes. We explore the issues in trading off the use of images and geometry by revisiting plenoptic-sampling analysis and the notions of view dependency and geometric proxies. Finally, we highlight compression techniques specifically designed for image-based representations. Such compression techniques are important in making IBR techniques practical.published_or_final_versio

A virtual reality system using the concentric mosaic: Construction, rendering, and data compression

This paper proposes a new image-based rendering (IBR) technique called "concentric mosaic" for virtual reality applications. IBR using the plenoptic function is an efficient technique for rendering new views of a scene from a collection of sample images previously captured. It provides much better image quality and lower computational requirement for rendering than conventional three-dimensional (3-D) model-building approaches. The concentric mosaic is a 3-D plenoptic function with viewpoints constrained on a plane. Compared with other more sophisticated four-dimensional plenoptic functions such as the light field and the lumigraph, the file size of a concentric mosaic is much smaller. In contrast to a panorama, the concentric mosaic allows users to move freely in a circular region and observe significant parallax and lighting changes without recovering the geometric and photometric scene models. The rendering of concentric mosaics is very efficient, and involves the reordering and interpolating of previously captured slit images in the concentric mosaic. It typically consists of hundreds of high-resolution images which consume a significant amount of storage and bandwidth for transmission. An MPEG-like compression algorithm is therefore proposed in this paper taking into account the access patterns and redundancy of the mosaic images. The compression algorithms of two equivalent representations of the concentric mosaic, namely the multiperspective panoramas and the normal setup sequence, are investigated. A multiresolution representation of concentric mosaics using a nonlinear filter bank is also proposed.published_or_final_versio