Light field image processing: an overview

Light field imaging has emerged as a technology allowing to capture richer visual information from our world. As opposed to traditional photography, which captures a 2D projection of the light in the scene integrating the angular domain, light fields collect radiance from rays in all directions, demultiplexing the angular information lost in conventional photography. On the one hand, this higher dimensional representation of visual data offers powerful capabilities for scene understanding, and substantially improves the performance of traditional computer vision problems such as depth sensing, post-capture refocusing, segmentation, video stabilization, material classification, etc. On the other hand, the high-dimensionality of light fields also brings up new challenges in terms of data capture, data compression, content editing, and display. Taking these two elements together, research in light field image processing has become increasingly popular in the computer vision, computer graphics, and signal processing communities. In this paper, we present a comprehensive overview and discussion of research in this field over the past 20 years. We focus on all aspects of light field image processing, including basic light field representation and theory, acquisition, super-resolution, depth estimation, compression, editing, processing algorithms for light field display, and computer vision applications of light field data

End-to-end 3D video communication over heterogeneous networks

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.Three-dimensional technology, more commonly referred to as 3D technology, has revolutionised many fields including entertainment, medicine, and communications to name a few. In addition to 3D films, games, and sports channels, 3D perception has made tele-medicine a reality. By the year 2015, 30% of the all HD panels at home will be 3D enabled, predicted by consumer electronics manufacturers. Stereoscopic cameras, a comparatively mature technology compared to other 3D systems, are now being used by ordinary citizens to produce 3D content and share at a click of a button just like they do with the 2D counterparts via sites like YouTube. But technical challenges still exist, including with autostereoscopic multiview displays. 3D content requires many complex considerations--including how to represent it, and deciphering what is the best compression format--when considering transmission or storage, because of its increased amount of data. Any decision must be taken in the light of the available bandwidth or storage capacity, quality and user expectations. Free viewpoint navigation also remains partly unsolved. The most pressing issue getting in the way of widespread uptake of consumer 3D systems is the ability to deliver 3D content to heterogeneous consumer displays over the heterogeneous networks. Optimising 3D video communication solutions must consider the entire pipeline, starting with optimisation at the video source to the end display and transmission optimisation. Multi-view offers the most compelling solution for 3D videos with motion parallax and freedom from wearing headgear for 3D video perception. Optimising multi-view video for delivery and display could increase the demand for true 3D in the consumer market. This thesis focuses on an end-to-end quality optimisation in 3D video communication/transmission, offering solutions for optimisation at the compression, transmission, and decoder levels.Brunel University - Isambard Research Scholarshi

Three-dimensional media for mobile devices

Cataloged from PDF version of article.This paper aims at providing an overview of the core technologies enabling the delivery of 3-D Media to next-generation mobile devices. To succeed in the design of the corresponding system, a profound knowledge about the human visual system and the visual cues that form the perception of depth, combined with understanding of the user requirements for designing user experience for mobile 3-D media, are required. These aspects are addressed first and related with the critical parts of the generic system within a novel user-centered research framework. Next-generation mobile devices are characterized through their portable 3-D displays, as those are considered critical for enabling a genuine 3-D experience on mobiles. Quality of 3-D content is emphasized as the most important factor for the adoption of the new technology. Quality is characterized through the most typical, 3-D-specific visual artifacts on portable 3-D displays and through subjective tests addressing the acceptance and satisfaction of different 3-D video representation, coding, and transmission methods. An emphasis is put on 3-D video broadcast over digital video broadcasting-handheld (DVB-H) in order to illustrate the importance of the joint source-channel optimization of 3-D video for its efficient compression and robust transmission over error-prone channels. The comparative results obtained identify the best coding and transmission approaches and enlighten the interaction between video quality and depth perception along with the influence of the context of media use. Finally, the paper speculates on the role and place of 3-D multimedia mobile devices in the future internet continuum involving the users in cocreation and refining of rich 3-D media content

Content-adaptive parallax barriers: optimizing dual-layer 3D displays using low-rank light field factorization

We optimize automultiscopic displays built by stacking a pair of modified LCD panels. To date, such dual-stacked LCDs have used heuristic parallax barriers for view-dependent imagery: the front LCD shows a fixed array of slits or pinholes, independent of the multi-view content. While prior works adapt the spacing between slits or pinholes, depending on viewer position, we show both layers can also be adapted to the multi-view content, increasing brightness and refresh rate. Unlike conventional barriers, both masks are allowed to exhibit non-binary opacities. It is shown that any 4D light field emitted by a dual-stacked LCD is the tensor product of two 2D masks. Thus, any pair of 1D masks only achieves a rank-1 approximation of a 2D light field. Temporal multiplexing of masks is shown to achieve higher-rank approximations. Non-negative matrix factorization (NMF) minimizes the weighted Euclidean distance between a target light field and that emitted by the display. Simulations and experiments characterize the resulting content-adaptive parallax barriers for low-rank light field approximation.National Science Foundation (U.S.) (grant CCF-0729126)National Research Foundation of Korea (grant 2009-352-D00232

다시점 디스플레이와 핀홀 방식 집적영상에 기반한 3차원 디스플레이의 개선에 관한 연구

학위논문 (박사)-- 서울대학교 대학원 : 전기공학부, 2012. 8. 이병호.최근 10년간, 디스플레이 시장은 미세 공정과 유기 및 무기 발광 소자의 발달로 인하여 급속도로 평판 디스플레이 시장으로 전환되었고, 보다 현실감 있는 영상을 재생하기 위한 노력을 통하여 2차원 고화질 영상의 시대에서 3차원 입체 영상의 시대로 전환되어 가고 있다. 이미 TV시장과 영화 시장에는 편광 방식의 안경을 이용한 3차원 디스플레이가 성공적으로 자리를 잡고 있으며, 이들을 활용한 컨텐츠 또한 지속적으로 제작되고 있다. 그러나 궁극적인 3차원 디스플레이를 향한 연구는 현재의 안경식 방식에서 안경을 착용하지 않으면서도 다시점의 3차원 영상을 관측할 수 있는 무안경식 3차원 디스플레이 방식으로 집중 되고 있다. 본 논문은 무안경식 3차원 디스플레이 방식 중 다시점 디스플레이와 핀홀형 집적영상에 기반을 둔 3차원 디스플레이의 개선에 관한 연구이다. 지금까지 연구된 다양한 3차원 디스플레이 방식 중, 패럴랙스 배리어, 렌티큘러와 같은 다시점 디스플레이와 집적영상은 가장 상용화에 가까운 무안경식 3차원 디스플레이 방식이다. 그러나 무안경식 3차원 디스플레이의 상용화를 위해서는 디스플레이 기술, 관찰자의 인지요인, 그리고 광학적 정보처리 기술의 세 가지 측면에서의 문제점들에 대한 논의가 필요하다. 이 논문은 무안경식 3차원 디스플레이 중, 다시점 디스플레이와 핀홀(pinhole)형 집적영상을 이 세 가지 측면에서 개선하는 연구에 대한 내용을 담고 있다. 먼저 디스플레이 기술적인 측면에서는, 전계발광소자(electroluminescent film) 혹은 평판 디스플레이에 배열된 컬러 필터를 이용하여 핀홀형 집적영상의 시야각, 해상도, 광선 집적도 그리고 2차원/3차원 변환을 개선하는 방법들을 제안한다. 먼저 깊이 표현 범위가 기본적으로 넓은 핀홀형 집적영상의 2차원/3차원 변환으로의 확장을 위하여 전계발광소자에 기반을 둔 새로운 광원 변환 기술을 제안한다. 전계발광소자는 기본적으로 자르거나 휘거나 혹은 구멍을 뚫어도 발광에 영향을 받지 않는 특성을 가지고 있어, 전계발광소자에 균일한 핀홀을 배열하여 기존의 핀홀형 집적영상으로 이용이 가능하다. 따라서 이를 이용하여, 2차원/3차원 변환이 가능한 집적영상을 설계할 수 있고, 구부리면서 발광 상태를 유지하는 특성을 이용하여 광시야각을 갖는 반원 형태의 3차원 디스플레이로 제작할 수 있다. 또한 360도 전체에서 관찰이 가능한 집적영상 기반의 3차원 디스플레이로 확장하여 제안한다. 그리고 현재 대중적으로 사용되는 액정 디스플레이상의 컬러 필터 층을 핀홀 배열과 빔 프로젝터를 이용하여 광선 집적도, 해상도 그리고 깊이 표현 범위를 확장한 핀홀형 집적영상으로 제안한다. 실제 액정 디스플레이상의 컬러 필터는 3가지 색으로 나뉘어져 있고, 이를 이용하여 각각 색 영역이 다른 핀홀 배열로 사용이 가능하다. 이와 함께 빔 프로젝터를 이용하여, 한 요소 영상의 영역이 기존의 방식에 비해 넓어지고 서로 침범할 수 있게 되어 해상도와 광선 집적도가 최대 3배 향상되는 것을 확인할 수 있다. 집적영상의 개선과 함께 디스플레이 측면에서의 개선에서는, 무안경식 3차원 디스플레이의 특성을 개선하기 위하여 다시점 디스플레이와 1차원 집적영상을 분석하고 융합하는 연구도 진행한다. 두 가지의 다른 무안경식 3차원 디스플레이 방식을 융합하기 위하여, light field를 이용한 공간-각도 분포 분석과 이의 주파수영역에서의 분석을 진행한다. 분석을 통해 얻어진 두 방식의 공간-각도 분포 특징을 이용하여 다시점 디스플레이와 1차원 집적영상을 시분할 및 부픽셀분할을 통해 한 디스플레이 상에서 구현하여, 보다 품질이 향상된 무안경식 3차원 디스플레이를 제안한다. 또 다른 무안경식 3차원 디스플레이의 과제는 관찰자의 인지 요소로, 본 논문에서는 다시점 디스플레이와 집적영상에서의 깊이 해상도 및 조절력 반응에 대한 연구를 진행한다. 먼저 다시점 디스플레이 상에서 3차원 디스플레이 자체의 해상도와 cardboard 효과에 의한 인지 깊이 해상도 변화를 살펴보기 위해, 다시점 디스플레이 기반의 3차원 방송 환경을 구축하고, 디스플레이 자체의 깊이 해상도 표현의 제한, 다시점 영상 합성 과정 중의 깊이 해상도 표현의 제한 그리고 cardboard 효과에 의한 깊이 해상도 제한을 수식적으로 분석하고, 관찰자를 대상으로 한 설문을 진행한다. 이를 통해, 다시점 3차원 디스플레이 방송 환경에서 3차원 영상을 위한 깊이정보의 해상도를 위한 기준을 제시한다. 또한 무안경식 3차원 디스플레이에서 수렴-조절의 충돌을 해결하기 위해 논의되고 있는 초다시점 조건과 집적영상의 상관관계를 밝히기 위하여, 집적영상에서 초다시점 조건을 만족하는 영역을 분석 및 시뮬레이션을 진행하고 이 영역 안과 밖에서 관찰자의 조절력을 측정한다. 이 과정을 통해, 집적영상에서 초다시점 조건과 관찰자의 조절력 변화와의 상관관계를 찾을 수 있다. 그리고 무안경식 3차원 디스플레이를 이용한 또 다른 과제인 광학적 정보 처리에서는, 요소영상과 다시점 영상을 이용한 깊이 추출과 3차원 영상의 재획득에 대한 연구를 진행한다. Optical flow에 기반을 둔 보다 정확한 깊이 추출 알고리즘을 이용하여 요소영상에서 추출한 부영상의 깊이를 추출하는 방법과 이의 응용에 대해 본 논문에서 설명한다. Optical flow에 기반을 둔 3차원 물체의 깊이 추출을 컴퓨터를 이용한 실사 3차원 영상의 재획득에 활용하는 경우, 집적영상의 본질적인 문제인 pseudoscopic 문제를 해결할 수 있다. 본 논문을 통한 무안경식 3차원 디스플레이의 디스플레이 기술, 관찰자의 인지요인, 그리고 광학적 정보처리 기술적인 측면에서의 개선 방법들은 추후 무안경식 3차원 디스플레이 시장의 확대와 가정용 TV시장으로의 진출에 크게 기여할 수 있을 것으로 보인다.This dissertation presents studies on improvement of three-dimensional (3D) displays based on multi-view display and pinhole-type integral imaging. Among various types of 3D displays, integral imaging and multi-view display such as parallax barrier and lenticular 3D display are almost commercialized autostereoscopic 3D display. For commercialization, autostereoscopic 3D display has three issues that are limitation of display technology, human factors related on human visual system and optical information processing technology. In this dissertation, the author will address the studies about the improvement methods for multi-view display and pinhole-type integral imaging in three issues. In the issue of display technology, the improvement methods of pinhole-type integral imaging using electroluminescent (EL) film and color filters on display panel are proposed to enhance the viewing angle, resolution, ray density and two-dimensional (2D)/3D convertibility. For large expressible depth range and 2D/3D convertibility in pinhole-type integral imaging, pinhole-type integral imaging is modified by new light source conversion layer based on EL film. The EL film has the advantage that it can operate continuously even when it is cut or punctured. Using this characteristic, the author generates an array of pinholes on an EL film to form a point light source array for reconstructing 3D images based on integral imaging. Taking advantage of the flexibility of EL films, a 2D/3D convertible integral imaging system with a wide viewing angle using a curved EL film is proposed which is extended to 360-degree viewable cylindrical 3D display system. For enhancement of ray density, resolution and expressible depth range in pinhole-type integral imaging, the system using color filter pinhole array on liquid crystal display panel with projection scheme is proposed. A color filter structure on liquid crystal display panel acts as pinhole array in integral imaging with separation of color channel. The use of color filter pinhole array and projection scheme can enlarge the region of one-elemental image and improve the resolution and ray density remarkably. In addition to the improvement of pinhole-type integral imaging, analysis and convergence of multi-view display and one-dimensional (1D) integral imaging are presented for improvement of characteristics in autostereoscopic display. For the convergence of two different autostereoscopic 3D display, multi-view display and integral imaging, light field analysis of spatio-angular distribution and its frequency domain analysis are performed. From the analysis, the convergence type of autostereoscopic 3D display based on multi-view display and 1D integral imaging is proposed by using time-multiplexing and sub-pixel multiplexing technique. On another issue in autostereoscopic 3D display, the depth resolution and accommodation response of human factors in multi-view display and integral imaging are researched. To find the effect of fundamental depth resolution and cardboard effect to the perceived depth resolution in multi-view display, the fundamental depth resolution and the cardboard effect from the synthesis process in the multi-view 3D TV broadcasting are analyzed and a subjective test is performed. In addition, the analysis and measurement of accommodation response of integral imaging with satisfying super multi-view display is performed to reveal the relation between the accommodation response of integral imaging and super multi-view condition. On the other issue of autostereoscopic 3D display, the optical information processing from elemental image and multi-view images in depth extraction and computational pickup method without pseudoscopic problem is presented. A more accurate depth extraction algorithm using optical flow from sub-images of elemental image is proposed and its applications are also presented in this dissertation.Abstract i Contents iv List of Figures ix List of Tables xvii Chapter 1 Introduction 1 1.1 Overview of autostereoscopic three-dimensional displays 1 1.2 Motivation of this dissertation 4 1.3 Scope and organization 7 Chapter 2 Enhancement of pinhole-type integral imaging using electroluminescent film and color filters on display panel 11 2.1 Integral imaging system using an electroluminescent film backlight for three-dimensional/two-dimensional convertibility and a curved structure 11 2.1.1 Introduction 11 2.1.2 Principles of 3D/2D convertible integral imaging using EL film 14 2.1.3 Experimental results 23 2.1.4 Conclusion 29 2.2 360-viewable cylindrical integral imaging system using a three-dimensionaltwo-dimensional switchable and flexible backlight 30 2.2.1 Introduction 30 2.2.2 Principles of the 360-degree viewable cylindrical integral imaging system 33 2.2.3 Analysis on the characteristic parameters and viewing zone of the 360-degree viewable cylindrical integral imaging system 36 2.2.4 Experiment 39 2.2.5 Conclusion 43 2.3 Integral imaging using color filter pinhole array on display panel 44 2.3.1 Introduction and motivation 44 2.3.2 Principles of proposed method 47 2.3.3 Experimental setup and results 54 2.3.4 Conclusion 60 Chapter 3 Analysis and convergence of multi-view display and one-dimensional integral imaging 62 3.1 Comparison of multi-view display and integral imaging 62 3.1.1 Principles of multi-view display and one-dimensional integral imaging 63 3.1.2 Principles of multi-view display and integral imaging in pickup methods 67 3.1.3 Analysis of multi-view display and integral imaging in light filed 73 3.2 Computational reacquisition of a real three-dimensional object for integral imaging without matching of pickup and display lens array 80 3.2.1 Introduction 80 3.2.2 Depth extraction and triangular mesh reconstruction from sub-images using optical flow 81 3.2.3 Conversion from point cloud to face texture information 83 3.2.4 Experimental result 85 3.2.5 Conclusion 87 3.3 Time-multiplexing and sub-pixel mapping of multi-view display and integral imaging 89 3.3.1 Design parameters of multi-view display and integral imaging 90 3.3.2 Convergence type of autostereoscopic display using time-multiplexing or sub-pixel mapping of multi-view display and one-dimensional integral imaging 91 3.3.3 Experimental result 94 Chapter 4 Perceived depth resolution and accommodation response of multi-view display and integral imaging 99 4.1 Effect of fundamental depth resolution and cardboard effect to perceived depth resolution on multi-view display 99 4.1.1 Introduction 100 4.1.2 Fundamental depth resolution from specification of slanted lenticular display 102 4.1.3 View synthesis parameters from specification of stereo pickup and multi-view display 105 4.1.4 Stereo pickup and multi-view synthesis of 3D object with varying depth resolution 109 4.1.5 Numerical comparison of synthesized view images in PSNR and NCC with varying depth resolution 113 4.1.6 Subjective test for limitation of perceived depth resolution in multi-view display 115 4.1.7 Conclusion 122 4.2 Effect of viewing region satisfying super multi-view condition in integral imaging 124 4.2.1 Introduction 124 4.2.2 Analysis of viewing region satisfying super multi-view condition in integral imaging 125 4.2.3 Expressible depth range and size of 3D object in integral imaging with super multi-view condition 128 4.2.4 Simulation 129 4.2.5 Accommodation response with super multi-view condition in integral imaging 131 4.2.6 Experimental result 132 4.2.7 Experimental result 136 Chapter 5 Conclusion 137 Bibliography 140 Appendix 148 한글 초록 149Docto

A compressive light field projection system

For about a century, researchers and experimentalists have strived to bring glasses-free 3D experiences to the big screen. Much progress has been made and light field projection systems are now commercially available. Unfortunately, available display systems usually employ dozens of devices making such setups costly, energy inefficient, and bulky. We present a compressive approach to light field synthesis with projection devices. For this purpose, we propose a novel, passive screen design that is inspired by angle-expanding Keplerian telescopes. Combined with high-speed light field projection and nonnegative light field factorization, we demonstrate that compressive light field projection is possible with a single device. We build a prototype light field projector and angle-expanding screen from scratch, evaluate the system in simulation, present a variety of results, and demonstrate that the projector can alternatively achieve super-resolved and high dynamic range 2D image display when used with a conventional screen.MIT Media Lab ConsortiumNatural Sciences and Engineering Research Council of Canada (NSERC Postdoctoral Fellowship)National Science Foundation (U.S.) (Grant NSF grant 0831281