Acquisition, compression and rendering of depth and texture for multi-view video

Three-dimensional (3D) video and imaging technologies is an emerging trend in the development of digital video systems, as we presently witness the appearance of 3D displays, coding systems, and 3D camera setups. Three-dimensional multi-view video is typically obtained from a set of synchronized cameras, which are capturing the same scene from different viewpoints. This technique especially enables applications such as freeviewpoint video or 3D-TV. Free-viewpoint video applications provide the feature to interactively select and render a virtual viewpoint of the scene. A 3D experience such as for example in 3D-TV is obtained if the data representation and display enable to distinguish the relief of the scene, i.e., the depth within the scene. With 3D-TV, the depth of the scene can be perceived using a multi-view display that renders simultaneously several views of the same scene. To render these multiple views on a remote display, an efficient transmission, and thus compression of the multi-view video is necessary. However, a major problem when dealing with multiview video is the intrinsically large amount of data to be compressed, decompressed and rendered. We aim at an efficient and flexible multi-view video system, and explore three different aspects. First, we develop an algorithm for acquiring a depth signal from a multi-view setup. Second, we present efficient 3D rendering algorithms for a multi-view signal. Third, we propose coding techniques for 3D multi-view signals, based on the use of an explicit depth signal. This motivates that the thesis is divided in three parts. The first part (Chapter 3) addresses the problem of 3D multi-view video acquisition. Multi-view video acquisition refers to the task of estimating and recording a 3D geometric description of the scene. A 3D description of the scene can be represented by a so-called depth image, which can be estimated by triangulation of the corresponding pixels in the multiple views. Initially, we focus on the problem of depth estimation using two views, and present the basic geometric model that enables the triangulation of corresponding pixels across the views. Next, we review two calculation/optimization strategies for determining corresponding pixels: a local and a one-dimensional optimization strategy. Second, to generalize from the two-view case, we introduce a simple geometric model for estimating the depth using multiple views simultaneously. Based on this geometric model, we propose a new multi-view depth-estimation technique, employing a one-dimensional optimization strategy that (1) reduces the noise level in the estimated depth images and (2) enforces consistent depth images across the views. The second part (Chapter 4) details the problem of multi-view image rendering. Multi-view image rendering refers to the process of generating synthetic images using multiple views. Two different rendering techniques are initially explored: a 3D image warping and a mesh-based rendering technique. Each of these methods has its limitations and suffers from either high computational complexity or low image rendering quality. As a consequence, we present two image-based rendering algorithms that improves the balance on the aforementioned issues. First, we derive an alternative formulation of the relief texture algorithm which was extented to the geometry of multiple views. The proposed technique features two advantages: it avoids rendering artifacts ("holes") in the synthetic image and it is suitable for execution on a standard Graphics Processor Unit (GPU). Second, we propose an inverse mapping rendering technique that allows a simple and accurate re-sampling of synthetic pixels. Experimental comparisons with 3D image warping show an improvement of rendering quality of 3.8 dB for the relief texture mapping and 3.0 dB for the inverse mapping rendering technique. The third part concentrates on the compression problem of multi-view texture and depth video (Chapters 5–7). In Chapter 5, we extend the standard H.264/MPEG-4 AVC video compression algorithm for handling the compression of multi-view video. As opposed to the Multi-view Video Coding (MVC) standard that encodes only the multi-view texture data, the proposed encoder peforms the compression of both the texture and the depth multi-view sequences. The proposed extension is based on exploiting the correlation between the multiple camera views. To this end, two different approaches for predictive coding of views have been investigated: a block-based disparity-compensated prediction technique and a View Synthesis Prediction (VSP) scheme. Whereas VSP relies on an accurate depth image, the block-based disparity-compensated prediction scheme can be performed without any geometry information. Our encoder adaptively selects the most appropriate prediction scheme using a rate-distortion criterion for an optimal prediction-mode selection. We present experimental results for several texture and depth multi-view sequences, yielding a quality improvement of up to 0.6 dB for the texture and 3.2 dB for the depth, when compared to solely performing H.264/MPEG-4AVC disparitycompensated prediction. Additionally, we discuss the trade-off between the random-access to a user-selected view and the coding efficiency. Experimental results illustrating and quantifying this trade-off are provided. In Chapter 6, we focus on the compression of a depth signal. We present a novel depth image coding algorithm which concentrates on the special characteristics of depth images: smooth regions delineated by sharp edges. The algorithm models these smooth regions using parameterized piecewiselinear functions and sharp edges by a straight line, so that it is more efficient than a conventional transform-based encoder. To optimize the quality of the coding system for a given bit rate, a special global rate-distortion optimization balances the rate against the accuracy of the signal representation. For typical bit rates, i.e., between 0.01 and 0.25 bit/pixel, experiments have revealed that the coder outperforms a standard JPEG-2000 encoder by 0.6-3.0 dB. Preliminary results were published in the Proceedings of 26th Symposium on Information Theory in the Benelux. In Chapter 7, we propose a novel joint depth-texture bit-allocation algorithm for the joint compression of texture and depth images. The described algorithm combines the depth and texture Rate-Distortion (R-D) curves, to obtain a single R-D surface that allows the optimization of the joint bit-allocation in relation to the obtained rendering quality. Experimental results show an estimated gain of 1 dB compared to a compression performed without joint bit-allocation optimization. Besides this, our joint R-D model can be readily integrated into an multi-view H.264/MPEG-4 AVC coder because it yields the optimal compression setting with a limited computation effort

Coding of depth-maps using piecewise linear functions

An efficient. way t.o t.mnsmil, mul/.'i-view images is to send a single te:Di.uTe image togelheT with a cOTTesponding depth-map. The dept.h-map spec~fies the distance vetween each pi.1;d and the cam em,. With t.his inforrnat.ion, aTvit.m.7·y 3-D views can be genemt.cd at the decodeT. In this papeT, we pmpose a new algor'ithm for the coding of depth-maps that. pTOvides an efficient. re]wesent.ation of smooth regions as well as geome/,ric featuTes such as object. contO'llTs. OUT alg07it,hm uses a segmcnt.ation pmCed'llTC based on a quadt.Tee decomposition and TTI,odds the depth-map content. wit.h piecewise linear' funct.ions. We achieved a vit,-rat.e as low as 0.33 vii/pi.Tel, wit.hout any cntrvpy coding. The attmct.ivity (~r the coding algorit.hm is that., vy e.Tploiting spec~fic pTOper·ties of depth-maps, no degmdat.ions are shown along discont.iTl.1Lities, which is import.ant. few depth percept.ion

Quality improving techniques in DIBR for free-viewpoint video

This paper evaluates our 3D view interpolation rendering algorithm and proposes a few performance improving techniques. We aim at developing a rendering method for free-viewpoint 3DTV, based on depth image warping from surrounding cameras. The key feature of our approach is warping texture and depth in the first stage simultaneously and postpone blending the new view to a later stage, thereby avoiding errors in the virtual depth map. We evaluate the rendering quality in two ways. Firstly, it is measured by varying the distance between the two nearest cameras. We have obtained a PSNR gain of 3 dB and 4.5 dB for the 'Breakdancers' and 'Ballet' sequences, respectively, compared to the performance of a recent algorithm. A second series of tests in measuring the rendering quality were performed using compressed video or images from surrounding cameras. The overall quality of the system is dominated by rendering quality and not by coding

Precipitation-Redispersion of Cerium Oxide Nanoparticles with Poly(Acrylic Acid) : Towards Stable Dispersions

We exploit a precipitation-redispersion mechanism for complexation of short chain polyelectrolytes with cerium oxide nanoparticles to extend their stability ranges. As synthesized, cerium oxide sols at pH 1.4 consist of monodisperse cationic nanocrystalline particles having a hydrodynamic diameter of 10 nm and a molecular weight 400000 gmol-1. We show that short chain uncharged poly(acrylic acid) at low pH when added to a cerium oxide sols leads to macroscopic precipitation. As the pH is increased, the solution spontaneously redisperses into a clear solution of single particles with an anionic poly(acrylic acid) corona. The structure and dynamics of cerium oxide nanosols and their hybrid polymer-inorganic complexes in solution are investigated by static and dynamic light scattering, X-ray scattering, and by chemical analysis. Quantitative analysis of the redispersed sol gives rise to an estimate of 40 - 50 polymer chains per particle for stable suspension. This amount represents 20 % of the mass of the polymer-nanoparticle complexes. This complexation adds utility to the otherwise unstable cerium oxide dispersions by extending the range of stability of the sols in terms of pH, ionic strength and concentration.Comment: 6 Figures, 23 pages, submitte

Mean curvature flow of monotone Lagrangian submanifolds

We use holomorphic disks to describe the formation of singularities in the mean curvature flow of monotone Lagrangian submanifolds in $\mathbb C^{n}$.Comment: 37 pages, 3 figure

Towards a Multi-Scaled Functional-Structural Model of Apple, Linking Ecophysiology at the Fruit and Branch Scales

A multitude of data on eco-physiological processes in apple (Malus x domestica) is available, concerning various aspects of fruit growth and development, fruit quality, or leaf photosynthesis. However, despite the wealth of data and studies many processes leading to (inter-annual and intra-arboreal) heterogeneity in quantity of fruit production as well as fruit quality are still only poorly understood at the branch level Current Functional-Structural Plant Models of apple have targeted canopy architecture, i.e. development of vegetative structures. Here we will present a concept to apply the FSPM paradigm to the simulation of assimilation (source), transport and consumption (sink) of carbon in the context of a static structure representing the limb (fruit-bearing branch)

Effects of accretion flow on the chemical structure in the inner regions of protoplanetary disks

We have studied the dependence of the profiles of molecular abundances and line emission on the accretion flow in the hot (\ga 100K) inner region of protoplanetary disks. The gas-phase reactions initiated by evaporation of the ice mantle on dust grains are calculated along the accretion flow. We focus on methanol, a molecule that is formed predominantly through the evaporation of warm ice mantles, to show how the abundance profile and line emission depend on the accretion flow. Our results show that some evaporated molecules keep high abundances only when the accretion velocity is large enough, and that methanol could be useful as a diagnostic of the accretion flow by means of ALMA observations at the disk radius of \la 10AU.Comment: 6 pages, 5 figures, Accepted for publication in A&

Combined BIMA and OVRO observations of comet C/1999 S4 (LINEAR)

We present results from an observing campaign of the molecular content of the coma of comet C/1999 S4 (LINEAR) carried out jointly with the millimeter-arrays of the Berkeley-Illinois-Maryland Association (BIMA) and the Owens Valley Radio Observatory (OVRO). Using the BIMA array in autocorrelation (`single-dish') mode, we detected weak HCN J=1-0 emission from comet C/1999 S4 (LINEAR) at 14 +- 4 mK km/s averaged over the 143" beam. The three days over which emission was detected, 2000 July 21.9-24.2, immediately precede the reported full breakup of the nucleus of this comet. During this same period, we find an upper limit for HCN 1-0 of 144 mJy/beam km/s (203 mK km/s) in the 9"x12" synthesized beam of combined observations of BIMA and OVRO in cross-correlation (`imaging') mode. Together with reported values of HCN 1-0 emission in the 28" IRAM 30-meter beam, our data probe the spatial distribution of the HCN emission from radii of 1300 to 19,000 km. Using literature results of HCN excitation in cometary comae, we find that the relative line fluxes in the 12"x9", 28" and 143" beams are consistent with expectations for a nuclear source of HCN and expansion of the volatile gases and evaporating icy grains following a Haser model.Comment: 18 pages, 3 figures. Uses aastex. AJ in pres