298 research outputs found
Fast and Efficient Lenslet Image Compression
Light field imaging is characterized by capturing brightness, color, and
directional information of light rays in a scene. This leads to image
representations with huge amount of data that require efficient coding schemes.
In this paper, lenslet images are rendered into sub-aperture images. These
images are organized as a pseudo-sequence input for the HEVC video codec. To
better exploit redundancy among the neighboring sub-aperture images and
consequently decrease the distances between a sub-aperture image and its
references used for prediction, sub-aperture images are divided into four
smaller groups that are scanned in a serpentine order. The most central
sub-aperture image, which has the highest similarity to all the other images,
is used as the initial reference image for each of the four regions.
Furthermore, a structure is defined that selects spatially adjacent
sub-aperture images as prediction references with the highest similarity to the
current image. In this way, encoding efficiency increases, and furthermore it
leads to a higher similarity among the co-located Coding Three Units (CTUs).
The similarities among the co-located CTUs are exploited to predict Coding Unit
depths.Moreover, independent encoding of each group division enables parallel
processing, that along with the proposed coding unit depth prediction decrease
the encoding execution time by almost 80% on average. Simulation results show
that Rate-Distortion performance of the proposed method has higher compression
gain than the other state-of-the-art lenslet compression methods with lower
computational complexity
Fast and Efficient Lenslet Image Compression
Light field imaging is characterized by capturing brightness, color, and
directional information of light rays in a scene. This leads to image
representations with huge amount of data that require efficient coding schemes.
In this paper, lenslet images are rendered into sub-aperture images. These
images are organized as a pseudo-sequence input for the HEVC video codec. To
better exploit redundancy among the neighboring sub-aperture images and
consequently decrease the distances between a sub-aperture image and its
references used for prediction, sub-aperture images are divided into four
smaller groups that are scanned in a serpentine order. The most central
sub-aperture image, which has the highest similarity to all the other images,
is used as the initial reference image for each of the four regions.
Furthermore, a structure is defined that selects spatially adjacent
sub-aperture images as prediction references with the highest similarity to the
current image. In this way, encoding efficiency increases, and furthermore it
leads to a higher similarity among the co-located Coding Three Units (CTUs).
The similarities among the co-located CTUs are exploited to predict Coding Unit
depths.Moreover, independent encoding of each group division enables parallel
processing, that along with the proposed coding unit depth prediction decrease
the encoding execution time by almost 80% on average. Simulation results show
that Rate-Distortion performance of the proposed method has higher compression
gain than the other state-of-the-art lenslet compression methods with lower
computational complexity
Steered mixture-of-experts for light field images and video : representation and coding
Research in light field (LF) processing has heavily increased over the last decade. This is largely driven by the desire to achieve the same level of immersion and navigational freedom for camera-captured scenes as it is currently available for CGI content. Standardization organizations such as MPEG and JPEG continue to follow conventional coding paradigms in which viewpoints are discretely represented on 2-D regular grids. These grids are then further decorrelated through hybrid DPCM/transform techniques. However, these 2-D regular grids are less suited for high-dimensional data, such as LFs. We propose a novel coding framework for higher-dimensional image modalities, called Steered Mixture-of-Experts (SMoE). Coherent areas in the higher-dimensional space are represented by single higher-dimensional entities, called kernels. These kernels hold spatially localized information about light rays at any angle arriving at a certain region. The global model consists thus of a set of kernels which define a continuous approximation of the underlying plenoptic function. We introduce the theory of SMoE and illustrate its application for 2-D images, 4-D LF images, and 5-D LF video. We also propose an efficient coding strategy to convert the model parameters into a bitstream. Even without provisions for high-frequency information, the proposed method performs comparable to the state of the art for low-to-mid range bitrates with respect to subjective visual quality of 4-D LF images. In case of 5-D LF video, we observe superior decorrelation and coding performance with coding gains of a factor of 4x in bitrate for the same quality. At least equally important is the fact that our method inherently has desired functionality for LF rendering which is lacking in other state-of-the-art techniques: (1) full zero-delay random access, (2) light-weight pixel-parallel view reconstruction, and (3) intrinsic view interpolation and super-resolution
Dense light field coding: a survey
Light Field (LF) imaging is a promising solution for providing more immersive and closer to reality multimedia experiences to end-users with unprecedented creative freedom and flexibility for applications in different areas, such as virtual and augmented reality. Due to the recent technological advances in optics, sensor manufacturing and available transmission bandwidth, as well as the investment of many tech giants in this area, it is expected that soon many LF transmission systems will be available to both consumers and professionals. Recognizing this, novel standardization initiatives have recently emerged in both the Joint Photographic Experts Group (JPEG) and the Moving Picture Experts Group (MPEG), triggering the discussion on the deployment of LF coding solutions to efficiently handle the massive amount of data involved in such systems.
Since then, the topic of LF content coding has become a booming research area, attracting the attention of many researchers worldwide. In this context, this paper provides a comprehensive survey of the most relevant LF coding solutions proposed in the literature, focusing on angularly dense LFs. Special attention is placed on a thorough description of the different LF coding methods and on the main concepts related to this relevant area. Moreover, comprehensive insights are presented into open research challenges and future research directions for LF coding.info:eu-repo/semantics/publishedVersio
Non-disruptive use of light fields in image and video processing
In the age of computational imaging, cameras capture not only an image but also data. This captured additional data can be best used for photo-realistic renderings facilitating numerous post-processing possibilities such as perspective shift, depth scaling, digital refocus, 3D reconstruction, and much more. In computational photography, the light field imaging technology captures the complete volumetric information of a scene. This technology has the highest potential to accelerate immersive experiences towards close-toreality. It has gained significance in both commercial and research domains. However, due to lack of coding and storage formats and also the incompatibility of the tools to process and enable the data, light fields are not exploited to its full potential. This dissertation approaches the integration of light field data to image and video processing. Towards this goal, the representation of light fields using advanced file formats designed for 2D image assemblies to facilitate asset re-usability and interoperability between applications and devices is addressed. The novel 5D light field acquisition and the on-going research on coding frameworks are presented. Multiple techniques for optimised sequencing of light field data are also proposed. As light fields contain complete 3D information of a scene, large amounts of data is captured and is highly redundant in nature. Hence, by pre-processing the data using the proposed approaches, excellent coding performance can be achieved.Im Zeitalter der computergestützten Bildgebung erfassen Kameras nicht mehr nur ein Bild, sondern vielmehr auch Daten. Diese erfassten Zusatzdaten lassen sich optimal für fotorealistische Renderings nutzen und erlauben zahlreiche Nachbearbeitungsmöglichkeiten, wie Perspektivwechsel, Tiefenskalierung, digitale Nachfokussierung, 3D-Rekonstruktion und vieles mehr. In der computergestützten Fotografie erfasst die Lichtfeld-Abbildungstechnologie die vollständige volumetrische Information einer Szene. Diese Technologie bietet dabei das größte Potenzial, immersive Erlebnisse zu mehr Realitätsnähe zu beschleunigen. Deshalb gewinnt sie sowohl im kommerziellen Sektor als auch im Forschungsbereich zunehmend an Bedeutung. Aufgrund fehlender Kompressions- und Speicherformate sowie der Inkompatibilität derWerkzeuge zur Verarbeitung und Freigabe der Daten, wird das Potenzial der Lichtfelder nicht voll ausgeschöpft. Diese Dissertation ermöglicht die Integration von Lichtfelddaten in die Bild- und Videoverarbeitung. Hierzu wird die Darstellung von Lichtfeldern mit Hilfe von fortschrittlichen für 2D-Bilder entwickelten Dateiformaten erarbeitet, um die Wiederverwendbarkeit von Assets- Dateien und die Kompatibilität zwischen Anwendungen und Geräten zu erleichtern. Die neuartige 5D-Lichtfeldaufnahme und die aktuelle Forschung an Kompressions-Rahmenbedingungen werden vorgestellt. Es werden zudem verschiedene Techniken für eine optimierte Sequenzierung von Lichtfelddaten vorgeschlagen. Da Lichtfelder die vollständige 3D-Information einer Szene beinhalten, wird eine große Menge an Daten, die in hohem Maße redundant sind, erfasst. Die hier vorgeschlagenen Ansätze zur Datenvorverarbeitung erreichen dabei eine ausgezeichnete Komprimierleistung
Light field image processing : overview and research issues
Light field (LF) imaging first appeared in the computer graphics community with the goal of photorealistic 3D rendering [1]. Motivated by a variety of potential applications in various domains (e.g., computational photography, augmented reality, light field microscopy, medical imaging, 3D robotic, particle image velocimetry), imaging from real light fields has recently gained in popularity, both at the research and industrial level.peer-reviewe
Convex Optimization Based Bit Allocation for Light Field Compression under Weighting and Consistency Constraints
Compared with conventional image and video, light field images introduce the
weight channel, as well as the visual consistency of rendered view, information
that has to be taken into account when compressing the pseudo-temporal-sequence
(PTS) created from light field images. In this paper, we propose a novel frame
level bit allocation framework for PTS coding. A joint model that measures
weighted distortion and visual consistency, combined with an iterative encoding
system, yields the optimal bit allocation for each frame by solving a convex
optimization problem. Experimental results show that the proposed framework is
effective in producing desired distortion distribution based on weights, and
achieves up to 24.7% BD-rate reduction comparing to the default rate control
algorithm.Comment: published in IEEE Data Compression Conference, 201
MiNL: Micro-images based Neural Representation for Light Fields
Traditional representations for light fields can be separated into two types:
explicit representation and implicit representation. Unlike explicit
representation that represents light fields as Sub-Aperture Images (SAIs) based
arrays or Micro-Images (MIs) based lenslet images, implicit representation
treats light fields as neural networks, which is inherently a continuous
representation in contrast to discrete explicit representation. However, at
present almost all the implicit representations for light fields utilize SAIs
to train an MLP to learn a pixel-wise mapping from 4D spatial-angular
coordinate to pixel colors, which is neither compact nor of low complexity.
Instead, in this paper we propose MiNL, a novel MI-wise implicit neural
representation for light fields that train an MLP + CNN to learn a mapping from
2D MI coordinates to MI colors. Given the micro-image's coordinate, MiNL
outputs the corresponding micro-image's RGB values. Light field encoding in
MiNL is just training a neural network to regress the micro-images and the
decoding process is a simple feedforward operation. Compared with common
pixel-wise implicit representation, MiNL is more compact and efficient that has
faster decoding speed (\textbf{80180} speed-up) as well as better
visual quality (\textbf{14dB} PSNR improvement on average)
Compressive light field photography using overcomplete dictionaries and optimized projections
Light field photography has gained a significant research interest in the last two decades; today, commercial light field cameras are widely available. Nevertheless, most existing acquisition approaches either multiplex a low-resolution light field into a single 2D sensor image or require multiple photographs to be taken for acquiring a high-resolution light field. We propose a compressive light field camera architecture that allows for higher-resolution light fields to be recovered than previously possible from a single image. The proposed architecture comprises three key components: light field atoms as a sparse representation of natural light fields, an optical design that allows for capturing optimized 2D light field projections, and robust sparse reconstruction methods to recover a 4D light field from a single coded 2D projection. In addition, we demonstrate a variety of other applications for light field atoms and sparse coding, including 4D light field compression and denoising.Natural Sciences and Engineering Research Council of Canada (NSERC postdoctoral fellowship)United States. Defense Advanced Research Projects Agency (DARPA SCENICC program)Alfred P. Sloan Foundation (Sloan Research Fellowship)United States. Defense Advanced Research Projects Agency (DARPA Young Faculty Award
- …