Improved inter-layer prediction for Light field content coding with display scalability

Light field imaging based on microlens arrays - also known as plenoptic, holoscopic and integral imaging - has recently risen up as feasible and prospective technology due to its ability to support functionalities not straightforwardly available in conventional imaging systems, such as: post-production refocusing and depth of field changing. However, to gradually reach the consumer market and to provide interoperability with current 2D and 3D representations, a display scalable coding solution is essential. In this context, this paper proposes an improved display scalable light field codec comprising a three-layer hierarchical coding architecture (previously proposed by the authors) that provides interoperability with 2D (Base Layer) and 3D stereo and multiview (First Layer) representations, while the Second Layer supports the complete light field content. For further improving the compression performance, novel exemplar-based inter-layer coding tools are proposed here for the Second Layer, namely: (i) an inter-layer reference picture construction relying on an exemplar-based optimization algorithm for texture synthesis, and (ii) a direct prediction mode based on exemplar texture samples from lower layers. Experimental results show that the proposed solution performs better than the tested benchmark solutions, including the authors' previous scalable codec.info:eu-repo/semantics/acceptedVersio

Quality assessment of 2D image rendering for 4D light field content

Light Field (LF) technology, comprising visual data representations with huge amount of information, can be used to solve some of the current 3D technology limitations while enabling also new image functionalities not straightforwardly supported by traditional 2D imaging. However, current displays are not ready to process this kind of content, which means that rendering algorithms are necessary to present this type of visual content in 2D or 3D multi-view displays. However, the visual quality experienced by the user is highly dependent on the rendering approach adopted. Therefore, LF rendering technology requires appropriate quality assessment tests with real people, as there is no better and reliable way to assess the quality of these type of algorithms. In this context, this dissertation aims to study, implement, improve and compare various LF rendering algorithms and rendering approaches. Performance evaluation is done through subjective quality assessment tests aiming to understand which algorithm performs better in certain situations and the subjective quality impact of some of those algorithm parameters. Additionally, a comparison of single plane of focus versus all-infocus LF rendering approaches is also evaluated.A tecnologia de campos de luz – Light Field (LF), composta por representações visuais de dados com grande quantidade de informação, pode ser usada para solucionar algumas das limitações atuais da tecnologia 3D, além de permitir novas funcionalidades que não são suportadas diretamente pela imagem 2D tradicional. No entanto, os dispositivos de visualização actuais não estão preparados para processar este tipo de conteúdo, o que significa que são necessários algoritmos de renderização para apresentar este tipo de conteúdo visual em versão 2D ou em versão 3D com múltiplas vistas. No entanto, a qualidade visual do ponto vista da percepção do utilizador é altamente dependente da abordagem de renderização adotada. Portanto, a tecnologia de renderização LF requer avaliação de qualidade adequada com pessoas reais, já que não há maneira melhor e mais confiável de avaliar a qualidade deste tipo de algoritmos. Neste contexto, esta dissertação tem como objetivo estudar, implementar e comparar diversos algoritmos e abordagens de renderização LF. A avaliação de desempenho é feita recorrendo a testes subjetivos de avaliação de qualidade para entender qual algoritmo que apresenta melhor desempenho em determinadas situações e a influência, em termos da qualidade subjetiva, de alguns parâmetros de input em certos algoritmos. Além disso, também é avaliada uma comparação de abordagens de renderização com focagem em apenas um plano versus renderização com focagem em todos os planos

Depth scene estimation from images captured with a plenoptic câmera

Monografia (graduação)—Université de Bordeaux, ENSEIRB-MATMECA, Universidade de Brasília, 2013.Uma câmera plenóptica, também conhecida como \textit{light field camera}, é um dispositivo que emprega uma rede de microlentes colocada entre a lente principal e o sensor da câmera para capturar a informação 4D da luz de uma cena. Este \textit{light field} nos permite conhecer a posição e o ângulo de incidência dos raios de luz capturados pela câmera e pode ser usado para melhorar as soluções de problemas relacionados com gráfico computacional e visão por computador. Com um campo de luz amostrado adquirido pela câmera, várias imagens da cena em baixa resolução estão disponíveis das quais é possível inferir a profundidade. Diferentemente do estéreo multivisão tradicional, estas vistas são capturadas pelo mesmo sensor, implicando que elas são adquiridas com os mesmos parâmetros da câmera. Da mesma forma, as vistas estão em geometria epipolar perfeita. Entretanto, outros problemas aparecem devido a esta configuração. O sensor da câmera usa um filtro de Bayer e a dematriçagem da imagem bruta implica em interferência entre as vistas, criando artefatos de imagem. A construção das vistas modifica o padrão de cores, adicionando complexidade para a dematriçagem. A resolução das vistas que podemos obter é outro problema. Como a informação angular e espacial são amostrados pelo mesmo sensor, existe um compromisso entre a resolução das vistas e o número de vistas disponíveis. Para a câmera Lytro, por exemplo, as vistas são construídas com uma resolução de aproximadamente 0,12 megapixels, implicando em \textit{aliasing} para a maioria das cenas. Este trabalho apresenta: Um técnica para construir as vistas a partir da imagem bruta capturada pela câmera; um método de estimação de disparidade adaptado às câmeras plenópticas que permite a estimação mesmo sem a dematriçagem; um novo conceito para representar a disparidade no caso do estéreo multi-vistas; um esquema de reconstrução e dematriçagem usando a informação da disparidade e os pixels de vistas vizinhas.A plenoptic camera, also known as light field camera, is a device that employs a microlens array placed between the main lens and the camera sensor to capture the 4D light field information about a scene. Such light field enable us to know the position and angle of incidence of the light rays captured by the camera and can be used to improve the solution of computer graphics and computer vision-related problems. With a sampled light field acquired from a plenoptic camera, several low-resolution views of the scene are available from which to infer depth. Unlike traditional multiview stereo, these views are captured by the same sensor, implying that they are acquired with the same camera parameters. Also the views are in perfect epipolar geometry. However, other problems arises with such configuration. The camera sensor uses a Bayer color filter and demosaicing the RAW data implies view cross-talk creating image artifacts. The rendering of the views modify the color pattern, adding complexity for demosaicing. The resolution of the views we can get is another problem. As the angular and spatial position of the light rays are sampled by the same sensor, there is a trade off between view resolution and number of available views. For Lytro camera, for example, the views are rendered with about 0.12 megapixels of resolution, implying in aliasing on the views for most of the scenes. This work present: an approach to render the views from the RAW image captured by the camera; a method of disparity estimation adapted to plenoptic cameras that enables the estimation even without executing the demosaicing; a new concept of representing the disparity information on the case of multiview stereo; a reconstruction and demosaicing scheme using the disparity information and the pixels of neighbouring views

Plenoptic Signal Processing for Robust Vision in Field Robotics

This thesis proposes the use of plenoptic cameras for improving the robustness and simplicity of machine vision in field robotics applications. Dust, rain, fog, snow, murky water and insufficient light can cause even the most sophisticated vision systems to fail. Plenoptic cameras offer an appealing alternative to conventional imagery by gathering significantly more light over a wider depth of field, and capturing a rich 4D light field structure that encodes textural and geometric information. The key contributions of this work lie in exploring the properties of plenoptic signals and developing algorithms for exploiting them. It lays the groundwork for the deployment of plenoptic cameras in field robotics by establishing a decoding, calibration and rectification scheme appropriate to compact, lenslet-based devices. Next, the frequency-domain shape of plenoptic signals is elaborated and exploited by constructing a filter which focuses over a wide depth of field rather than at a single depth. This filter is shown to reject noise, improving contrast in low light and through attenuating media, while mitigating occluders such as snow, rain and underwater particulate matter. Next, a closed-form generalization of optical flow is presented which directly estimates camera motion from first-order derivatives. An elegant adaptation of this "plenoptic flow" to lenslet-based imagery is demonstrated, as well as a simple, additive method for rendering novel views. Finally, the isolation of dynamic elements from a static background is considered, a task complicated by the non-uniform apparent motion caused by a mobile camera. Two elegant closed-form solutions are presented dealing with monocular time-series and light field image pairs. This work emphasizes non-iterative, noise-tolerant, closed-form, linear methods with predictable and constant runtimes, making them suitable for real-time embedded implementation in field robotics applications

Plenoptic Signal Processing for Robust Vision in Field Robotics

This thesis proposes the use of plenoptic cameras for improving the robustness and simplicity of machine vision in field robotics applications. Dust, rain, fog, snow, murky water and insufficient light can cause even the most sophisticated vision systems to fail. Plenoptic cameras offer an appealing alternative to conventional imagery by gathering significantly more light over a wider depth of field, and capturing a rich 4D light field structure that encodes textural and geometric information. The key contributions of this work lie in exploring the properties of plenoptic signals and developing algorithms for exploiting them. It lays the groundwork for the deployment of plenoptic cameras in field robotics by establishing a decoding, calibration and rectification scheme appropriate to compact, lenslet-based devices. Next, the frequency-domain shape of plenoptic signals is elaborated and exploited by constructing a filter which focuses over a wide depth of field rather than at a single depth. This filter is shown to reject noise, improving contrast in low light and through attenuating media, while mitigating occluders such as snow, rain and underwater particulate matter. Next, a closed-form generalization of optical flow is presented which directly estimates camera motion from first-order derivatives. An elegant adaptation of this "plenoptic flow" to lenslet-based imagery is demonstrated, as well as a simple, additive method for rendering novel views. Finally, the isolation of dynamic elements from a static background is considered, a task complicated by the non-uniform apparent motion caused by a mobile camera. Two elegant closed-form solutions are presented dealing with monocular time-series and light field image pairs. This work emphasizes non-iterative, noise-tolerant, closed-form, linear methods with predictable and constant runtimes, making them suitable for real-time embedded implementation in field robotics applications

Implementación y evaluación de algoritmos para la visualización de imágenes de campos de luz

Tesis (Magister en Análisis y Procesamiento de Imágenes)-- Universidad Nacional de Córdoba, Facultad de Matemática, Astronomía, Física y Computación, 2018.Maestría conjunta con la Facultad de Cs. Exactas Físicas y Naturales-UNC.En la presente tesis determinamos las posibilidades de la utilización del modelo de campos de luz para generar nuevas representaciones de una escena 3D analizando la información espacio-angular que contiene la función plenóptica y su codificación en una matriz 4D, seleccionando parametrización de dos planos paralelos. Implementamos dicha codificación, visualización mutiperspectiva y reenfoque en el dominio espacial y frecuencial, basados en numerosos autores y un dispositivo experimental. Evaluamos los algoritmos en base a tiempos de proceso, preservación de los atributos fotométricos de la escena y rangos de reenfoque. Concluimos que la fotografía plenóptica es una potente herramienta para visualización 3D.In this thesis we determine the possibilities of using the light field model to generate new representations of a 3D scene by analyzing the space-angular information that contains the plenoptic function and its coding in a 4D matrix, selecting parametrization of two parallel planes. We implemented this coding, mutiperspective visualization and refocusing in the spatial and frequency domain, based on numerous authors and an experimental device. We evaluate the algorithms based on process times, preservation of the photometric attributes of the scene and refocus ranges. We conclude that the plenopic photography is a powerful tool for 3D visualizatio

An Analysis of Color Demosaicing in Plenoptic Cameras

A plenoptic camera captures the 4D radiance about a scene. Recent practical solutions mount a microlens array on top of a commodity SLR to directly acquire these rays. However, they suffer from low resolution as hundreds of thousands of views need to be captured in a single shot. In this paper, we develop a simple but effective technique for improving the image resolution of the plenoptic camera by maneuvering the demosaicing process. We first show that the traditional solution by demosaicing each individual microlens image and then blending them for view synthesis is suboptimal. In particular, this demosaicing process often suffers from aliasing artifacts, and it damages high frequency information recorded by each microlens image hence degrades the image quality. We instead propose to demosaic the synthesized view at the rendering stage. Specifically, we first transform the radiance to the desired focal plane and then apply frequency domain plenoptic resampling. A full resolution color filtered image is then created by performing a 2D integral projection from the reparameterized radiance. Finally, we conduct demosacing to obtain the color result. We show that our solution can achieve visible resolution enhancement on dynamic refocusing and depth-assisted deep focus rendering. 1