Light field processor: a lytro illum imaging application

Light field imaging technology is at the intersection of three main research areas: computer graphics, computational photography and computer vision. This technology has the potential to allow functionalities that were previously impracticable, if not impossible, like refocusing photographic images after the capture or moving around in a VR scene produced by a real-time game engine, with 6DoF. Traditional photography produces one single output whenever a user presses the shot button. Light field photography may have several different outputs because it collects much more data about a scene. Thus it requires post-processing in order to extract any piece of useful information, like 2D images, and that is a characteristic feature that makes this technology substantially different from all others in the field of image making. Post processing means using a specialised application and, since this technology is still in its infancy, those applications are scarce. This context presented a good opportunity for such a development. Light Field Processor is the main outcome of this work. It is a computer application able to open and decode images from Lytro Illum light field cameras, which it may then store as a new file format (Decoded Light Field), proposed in this dissertation, for later use. It is able to extract 2D viewpoints, 2D maps of viewpoints or the microlens array, videos showing the intrinsic parallax of the light field and metadata, as well as do some basic image processing.A tecnologia de imagem de campo de luz está na intersecção de três grandes áreas de investigação: gráficos por computador, fotografia computacional e visão por computador. Esta tecnologia tem o potencial de possibilitar funcionalidades que eram anteriormente impraticáveis, senão mesmo impossíveis, tais como refocar imagens fotográficas após a captura ou movimentar-se numa cena de RV produzida por um motor de jogos em tempo real, com 6 graus de liberdade. A fotografia tradicional produz uma única saída sempre que um utilizador prime o botão de disparo. A fotografia campo de luz pode ter várias saídas diferentes porque junta muito mais dados acerca da cena. Logo ela requer pós-processamento por forma a extrair qualquer informação útil, como imagens 2D, e essa é uma funcionalidade característica que faz desta tecnologia substancialmente diferente de todas as outras no ramo da produção de imagem. Pós-processamento significa usar uma aplicação especializada e, uma vez que esta tecnologia ainda está na sua infância, essas aplicações são escassas. Este contexto proporcionou uma boa oportunidade para tal desenvolvimento. Light Field Processor é o principal resultado deste trabalho. É uma aplicação para computador capaz de abrir e descodificar imagens de cameras Lytro Illum campo de luz, que podem então ser armazenadas como um novo formato de ficheiro (Campo de Luz Descodificado), proposto nesta dissertação, para uso posterior. É capaz de extrair pontos de vista 2D, mapas 2D de pontos de vista ou conjunto de microlentes, vídeos mostrando a paralaxe intrínseca do campo de luz e metadados, assim como fazer algum processamento de imagem básico

Enhanced processing methods for light field imaging

The light field camera provides rich textural and geometric information, but it is still challenging to use it efficiently and accurately to solve computer vision problems. Light field image processing is divided into multiple levels. First, low-level processing technology mainly includes the acquisition of light field images and their preprocessing. Second, the middle-level process consists of the depth estimation, light field encoding, and the extraction of cues from the light field. Third, high-level processing involves 3D reconstruction, target recognition, visual odometry, image reconstruction, and other advanced applications. We propose a series of improved algorithms for each of these levels. The light field signal contains rich angular information. By contrast, traditional computer vision methods, as used for 2D images, often cannot make full use of the high-frequency part of the light field angular information. We propose a fast pre-estimation algorithm to enhance the light field feature to improve its speed and accuracy when keeping full use of the angular information.Light field filtering and refocusing are essential cues in light field signal processing. Modern frequency domain filtering technology and wavelet technology have effectively improved light field filtering accuracy but may fail at object edges. We adapted the sub-window filtering with the light field to improve the reconstruction of object edges. Light field images can analyze the effects of scattering and refraction phenomena, and there are still insufficient metrics to evaluate the results. Therefore, we propose a physical rendering-based light field dataset that simulates the distorted light field image through a transparent medium, such as atmospheric turbulence or water surface. The neural network is an essential method to process complex light field data. We propose an efficient 3D convolutional autoencoder network for the light field structure. This network overcomes the severe distortion caused by high-intensity turbulence with limited angular resolution and solves the difficulty of pixel matching between distorted images. This work emphasizes the application and usefulness of light field imaging in computer vision whilst improving light field image processing speed and accuracy through signal processing, computer graphics, computer vision, and artificial neural networks

Plenoptic Signal Processing for Robust Vision in Field Robotics

This thesis proposes the use of plenoptic cameras for improving the robustness and simplicity of machine vision in field robotics applications. Dust, rain, fog, snow, murky water and insufficient light can cause even the most sophisticated vision systems to fail. Plenoptic cameras offer an appealing alternative to conventional imagery by gathering significantly more light over a wider depth of field, and capturing a rich 4D light field structure that encodes textural and geometric information. The key contributions of this work lie in exploring the properties of plenoptic signals and developing algorithms for exploiting them. It lays the groundwork for the deployment of plenoptic cameras in field robotics by establishing a decoding, calibration and rectification scheme appropriate to compact, lenslet-based devices. Next, the frequency-domain shape of plenoptic signals is elaborated and exploited by constructing a filter which focuses over a wide depth of field rather than at a single depth. This filter is shown to reject noise, improving contrast in low light and through attenuating media, while mitigating occluders such as snow, rain and underwater particulate matter. Next, a closed-form generalization of optical flow is presented which directly estimates camera motion from first-order derivatives. An elegant adaptation of this "plenoptic flow" to lenslet-based imagery is demonstrated, as well as a simple, additive method for rendering novel views. Finally, the isolation of dynamic elements from a static background is considered, a task complicated by the non-uniform apparent motion caused by a mobile camera. Two elegant closed-form solutions are presented dealing with monocular time-series and light field image pairs. This work emphasizes non-iterative, noise-tolerant, closed-form, linear methods with predictable and constant runtimes, making them suitable for real-time embedded implementation in field robotics applications

Plenoptic Signal Processing for Robust Vision in Field Robotics

This thesis proposes the use of plenoptic cameras for improving the robustness and simplicity of machine vision in field robotics applications. Dust, rain, fog, snow, murky water and insufficient light can cause even the most sophisticated vision systems to fail. Plenoptic cameras offer an appealing alternative to conventional imagery by gathering significantly more light over a wider depth of field, and capturing a rich 4D light field structure that encodes textural and geometric information. The key contributions of this work lie in exploring the properties of plenoptic signals and developing algorithms for exploiting them. It lays the groundwork for the deployment of plenoptic cameras in field robotics by establishing a decoding, calibration and rectification scheme appropriate to compact, lenslet-based devices. Next, the frequency-domain shape of plenoptic signals is elaborated and exploited by constructing a filter which focuses over a wide depth of field rather than at a single depth. This filter is shown to reject noise, improving contrast in low light and through attenuating media, while mitigating occluders such as snow, rain and underwater particulate matter. Next, a closed-form generalization of optical flow is presented which directly estimates camera motion from first-order derivatives. An elegant adaptation of this "plenoptic flow" to lenslet-based imagery is demonstrated, as well as a simple, additive method for rendering novel views. Finally, the isolation of dynamic elements from a static background is considered, a task complicated by the non-uniform apparent motion caused by a mobile camera. Two elegant closed-form solutions are presented dealing with monocular time-series and light field image pairs. This work emphasizes non-iterative, noise-tolerant, closed-form, linear methods with predictable and constant runtimes, making them suitable for real-time embedded implementation in field robotics applications

Implementación y evaluación de algoritmos para la visualización de imágenes de campos de luz

Tesis (Magister en Análisis y Procesamiento de Imágenes)-- Universidad Nacional de Córdoba, Facultad de Matemática, Astronomía, Física y Computación, 2018.Maestría conjunta con la Facultad de Cs. Exactas Físicas y Naturales-UNC.En la presente tesis determinamos las posibilidades de la utilización del modelo de campos de luz para generar nuevas representaciones de una escena 3D analizando la información espacio-angular que contiene la función plenóptica y su codificación en una matriz 4D, seleccionando parametrización de dos planos paralelos. Implementamos dicha codificación, visualización mutiperspectiva y reenfoque en el dominio espacial y frecuencial, basados en numerosos autores y un dispositivo experimental. Evaluamos los algoritmos en base a tiempos de proceso, preservación de los atributos fotométricos de la escena y rangos de reenfoque. Concluimos que la fotografía plenóptica es una potente herramienta para visualización 3D.In this thesis we determine the possibilities of using the light field model to generate new representations of a 3D scene by analyzing the space-angular information that contains the plenoptic function and its coding in a 4D matrix, selecting parametrization of two parallel planes. We implemented this coding, mutiperspective visualization and refocusing in the spatial and frequency domain, based on numerous authors and an experimental device. We evaluate the algorithms based on process times, preservation of the photometric attributes of the scene and refocus ranges. We conclude that the plenopic photography is a powerful tool for 3D visualizatio