17 research outputs found
RLFC: Random Access Light Field Compression using Key Views and Bounded Integer Encoding
We present a new hierarchical compression scheme for encoding light field
images (LFI) that is suitable for interactive rendering. Our method (RLFC)
exploits redundancies in the light field images by constructing a tree
structure. The top level (root) of the tree captures the common high-level
details across the LFI, and other levels (children) of the tree capture
specific low-level details of the LFI. Our decompressing algorithm corresponds
to tree traversal operations and gathers the values stored at different levels
of the tree. Furthermore, we use bounded integer sequence encoding which
provides random access and fast hardware decoding for compressing the blocks of
children of the tree. We have evaluated our method for 4D two-plane
parameterized light fields. The compression rates vary from 0.08 - 2.5 bits per
pixel (bpp), resulting in compression ratios of around 200:1 to 20:1 for a PSNR
quality of 40 to 50 dB. The decompression times for decoding the blocks of LFI
are 1 - 3 microseconds per channel on an NVIDIA GTX-960 and we can render new
views with a resolution of 512X512 at 200 fps. Our overall scheme is simple to
implement and involves only bit manipulations and integer arithmetic
operations.Comment: Accepted for publication at Symposium on Interactive 3D Graphics and
Games (I3D '19
Self-Supervised Light Field Reconstruction Using Shearlet Transform and Cycle Consistency
The image-based rendering approach using Shearlet Transform (ST) is one of
the state-of-the-art Densely-Sampled Light Field (DSLF) reconstruction methods.
It reconstructs Epipolar-Plane Images (EPIs) in image domain via an iterative
regularization algorithm restoring their coefficients in shearlet domain.
Consequently, the ST method tends to be slow because of the time spent on
domain transformations for dozens of iterations. To overcome this limitation,
this letter proposes a novel self-supervised DSLF reconstruction method,
CycleST, which applies ST and cycle consistency to DSLF reconstruction.
Specifically, CycleST is composed of an encoder-decoder network and a residual
learning strategy that restore the shearlet coefficients of densely-sampled
EPIs using EPI reconstruction and cycle consistency losses. Besides, CycleST is
a self-supervised approach that can be trained solely on Sparsely-Sampled Light
Fields (SSLFs) with small disparity ranges ( 8 pixels). Experimental
results of DSLF reconstruction on SSLFs with large disparity ranges (16 - 32
pixels) from two challenging real-world light field datasets demonstrate the
effectiveness and efficiency of the proposed CycleST method. Furthermore,
CycleST achieves ~ 9x speedup over ST, at least
LFSRDiff: Light Field Image Super-Resolution via Diffusion Models
Light field (LF) image super-resolution (SR) is a challenging problem due to
its inherent ill-posed nature, where a single low-resolution (LR) input LF
image can correspond to multiple potential super-resolved outcomes. Despite
this complexity, mainstream LF image SR methods typically adopt a deterministic
approach, generating only a single output supervised by pixel-wise loss
functions. This tendency often results in blurry and unrealistic results.
Although diffusion models can capture the distribution of potential SR results
by iteratively predicting Gaussian noise during the denoising process, they are
primarily designed for general images and struggle to effectively handle the
unique characteristics and information present in LF images. To address these
limitations, we introduce LFSRDiff, the first diffusion-based LF image SR
model, by incorporating the LF disentanglement mechanism. Our novel
contribution includes the introduction of a disentangled U-Net for diffusion
models, enabling more effective extraction and fusion of both spatial and
angular information within LF images. Through comprehensive experimental
evaluations and comparisons with the state-of-the-art LF image SR methods, the
proposed approach consistently produces diverse and realistic SR results. It
achieves the highest perceptual metric in terms of LPIPS. It also demonstrates
the ability to effectively control the trade-off between perception and
distortion. The code is available at
\url{https://github.com/chaowentao/LFSRDiff}
OccCasNet: Occlusion-aware Cascade Cost Volume for Light Field Depth Estimation
Light field (LF) depth estimation is a crucial task with numerous practical
applications. However, mainstream methods based on the multi-view stereo (MVS)
are resource-intensive and time-consuming as they need to construct a finer
cost volume. To address this issue and achieve a better trade-off between
accuracy and efficiency, we propose an occlusion-aware cascade cost volume for
LF depth (disparity) estimation. Our cascaded strategy reduces the sampling
number while keeping the sampling interval constant during the construction of
a finer cost volume. We also introduce occlusion maps to enhance accuracy in
constructing the occlusion-aware cost volume. Specifically, we first obtain the
coarse disparity map through the coarse disparity estimation network. Then, the
sub-aperture images (SAIs) of side views are warped to the center view based on
the initial disparity map. Next, we propose photo-consistency constraints
between the warped SAIs and the center SAI to generate occlusion maps for each
SAI. Finally, we introduce the coarse disparity map and occlusion maps to
construct an occlusion-aware refined cost volume, enabling the refined
disparity estimation network to yield a more precise disparity map. Extensive
experiments demonstrate the effectiveness of our method. Compared with
state-of-the-art methods, our method achieves a superior balance between
accuracy and efficiency and ranks first in terms of MSE and Q25 metrics among
published methods on the HCI 4D benchmark. The code and model of the proposed
method are available at https://github.com/chaowentao/OccCasNet
Light Field Depth Estimation Based on Stitched-EPI
Depth estimation is one of the most essential problems for light field
applications. In EPI-based methods, the slope computation usually suffers low
accuracy due to the discretization error and low angular resolution. In
addition, recent methods work well in most regions but often struggle with
blurry edges over occluded regions and ambiguity over texture-less regions. To
address these challenging issues, we first propose the stitched-EPI and
half-stitched-EPI algorithms for non-occluded and occluded regions,
respectively. The algorithms improve slope computation by shifting and
concatenating lines in different EPIs but related to the same point in 3D
scene, while the half-stitched-EPI only uses non-occluded part of lines.
Combined with the joint photo-consistency cost proposed by us, the more
accurate and robust depth map can be obtained in both occluded and non-occluded
regions. Furthermore, to improve the depth estimation in texture-less regions,
we propose a depth propagation strategy that determines their depth from the
edge to interior, from accurate regions to coarse regions. Experimental and
ablation results demonstrate that the proposed method achieves accurate and
robust depth maps in all regions effectively.Comment: 15 page