Deformable Beamsplitters: Enhancing Perception with Wide Field of View, Varifocal Augmented Reality Displays

An augmented reality head-mounted display with full environmental awareness could present data in new ways and provide a new type of experience, allowing seamless transitions between real life and virtual content. However, creating a light-weight, optical see-through display providing both focus support and wide field of view remains a challenge. This dissertation describes a new dynamic optical element, the deformable beamsplitter, and its applications for wide field of view, varifocal, augmented reality displays. Deformable beamsplitters combine a traditional deformable membrane mirror and a beamsplitter into a single element, allowing reflected light to be manipulated by the deforming membrane mirror, while transmitted light remains unchanged. This research enables both single element optical design and correct focus while maintaining a wide field of view, as demonstrated by the description and analysis of two prototype hardware display systems which incorporate deformable beamsplitters. As a user changes the depth of their gaze when looking through these displays, the focus of virtual content can quickly be altered to match the real world by simply modulating air pressure in a chamber behind the deformable beamsplitter; thus ameliorating vergence–accommodation conflict. Two user studies verify the display prototypes’ capabilities and show the potential of the display in enhancing human performance at quickly perceiving visual stimuli. This work shows that near-eye displays built with deformable beamsplitters allow for simple optical designs that enable wide field of view and comfortable viewing experiences with the potential to enhance user perception.Doctor of Philosoph

Beaming Displays

Existing near-eye display designs struggle to balance between multiple trade-offs such as form factor, weight, computational requirements, and battery life. These design trade-offs are major obstacles on the path towards an all-day usable near-eye display. In this work, we address these trade-offs by, paradoxically, removing the display from near-eye displays. We present the beaming displays, a new type of near-eye display system that uses a projector and an all passive wearable headset. We modify an off-the-shelf projector with additional lenses. We install such a projector to the environment to beam images from a distance to a passive wearable headset. The beaming projection system tracks the current position of a wearable headset to project distortion-free images with correct perspectives. In our system, a wearable headset guides the beamed images to a user’s retina, which are then perceived as an augmented scene within a user’s field of view. In addition to providing the system design of the beaming display, we provide a physical prototype and show that the beaming display can provide resolutions as high as consumer-level near-eye displays. We also discuss the different aspects of the design space for our proposal

Roadmap on 3D integral imaging: Sensing, processing, and display

This Roadmap article on three-dimensional integral imaging provides an overview of some of the research activities in the field of integral imaging. The article discusses various aspects of the field including sensing of 3D scenes, processing of captured information, and 3D display and visualization of information. The paper consists of a series of 15 sections from the experts presenting various aspects of the field on sensing, processing, displays, augmented reality, microscopy, object recognition, and other applications. Each section represents the vision of its author to describe the progress, potential, vision, and challenging issues in this field

편광 다중화를 이용하여 향상된 기능을 제공하는 도파관 기반의 근안 디스플레이

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2021. 2. 이병호.This dissertation presents the studies on the optical design method that enhances the display performance of see-through waveguide-based near-eye displays (WNEDs) using the polarization multiplexing technique. The studies focus on the strategies to improve the crucial display performances without compromising a small form factor, the most attractive merit of the WNEDs. To achieve this goal, thin and lightweight polarization-dependent optical elements are devised and employed in the WNED structure. The polarization-dependent devices can allow multiple optical functions or optical paths depending on the polarization state of the input beam, which can break through the limitation of the waveguide system with the polarization multiplexing. To realize the function-selective eyepiece for AR applications, the proposed devices should operate as an optically transparent window for the real scene while performing specific optical functions for the virtual image. The proposed devices are manufactured in a combination structure in which polarization-dependent optical elements are stacked. The total thickness of the stacked structure is about 1 mm, and it can be attached to the waveguide surface without conspicuously increasing the form factor of the optical system. Using the proposed polarization-dependent devices, the author proposes three types of novel WNED systems with enhanced performance. First, the author suggests a compact WNED with dual focal planes. Conventional WNEDs have an inherent limitation that the focal plane of the virtual image is at an infinite distance because they extract a stream of collimated light at the out-coupler. By using the polarization-dependent eyepiece lens, an additional focal plane can be generated with the polarization multiplexing in addition to infinity depth. The proposed configuration can provide comfortable AR environments by alleviating visual fatigue caused by vergence-accommodation conflict. Second, the novel WNED configuration with extended field-of-view (FOV) is presented. In the WNEDs, the maximum allowable FOV is determined by the material properties of the diffraction optics and the substrate. By using the polarization-dependent steering combiner, the FOV can be extended up to two times, which can provide more immersive AR experiences. In addition, this dissertation demonstrates that the distortion for the real scene caused by the stacked structure cannot severely disturb the image quality, considering the acuity of human vision. Lastly, the author presents a retinal projection-based WNED with switchable viewpoints by simultaneously adopting the polarization-dependent lens and grating. The proposed system can convert the viewpoint according to the position of the eye pupil without mechanical movement. The polarization-dependent viewpoint switching can resolve the inherent problem of a narrow eyebox in retinal projection displays without employing the bulky optics for mechanical movement. In conclusion, the dissertation presents the practical optical design and detailed analysis for enhanced WNED based on the polarization multiplexing technique through various simulations and experiments. The proposed approaches are expected to be utilized as an innovative solution for compact wearable displays.본 박사학위 논문에서는 편광 다중화 기법을 이용하여 도파관 기반의 증강현실 근안 디스플레이의 성능을 향상시키는 광학 설계 및 이에 대한 분석에 대해 논의한다. 본 연구는 도파관 기반 근안 디스플레이의 가장 큰 장점인 소형 폼 팩터를 유지하면서 디스플레이 성능을 개선하는 것에 중점을 둔다. 이를 위해 기존 광학 소자에 비해 매우 가볍고 얇은 편광 의존형 결합기 광학 소자가 새롭게 제안되며, 이는 입사광의 편광 상태에 따라 독립적인 광 경로 제어를 가능케 하여 편광 다중화를 통해 향상된 성능을 제공 할 수 있다. 또한 실제 영상의 빛은 그대로 투과 시킴으로써 증강현실을 위한 소자로 활용 가능하다. 본 연구에서 제안하는 편광 의존형 결합기 광학 소자는 기하학적 위상(geometric phase, GP)에 기반하여 동작한다. GP 기반의 광학소자가 서로 직교하는 원형 편광 입사광에 대해 상보적인 기능을 수행하는 것을 이용하여, 두 개 이상의 GP 소자와 편광 제어를 위한 광학 필름들을 중첩 시킴으로써 증강현실 결합기 광학 소자를 구현할 수 있다. 이들 광학소자는 매우 얇기 때문에, 본 연구에서 제작된 편광 의존형 결합기 광학 소자의 총 두께는 1 mm 수준으로 폼 팩터 제약으로부터 자유롭다. 또한 평평한 필름 형태이므로, 평판형 도파관에 부착하기 쉽다는 이점을 지닌다. 고안된 편광 의존형 결합기 광학 소자를 사용하여 세 가지 유형의 새로운 도파관 기반의 근안 디스플레이 구조를 제안한다. 첫 번째는 입사광의 편광 상태에 따라 투명 광학 창 또는 오목 렌즈로 작동하는 편광 의존형 결합기 렌즈를 적용하여 가상 영상에 대해 이중 초점면을 제공하는 시스템이다. 제안된 구조는 기존의 도파관 기반 근안 디스플레이가 무한대 위치에 단일 초점면을 제공함으로써 발생하는 시각적 피로 및 흐릿한 증강현실 영상의 문제를 완화할 수 있다. 두 번째로는 입사광의 편광 상태에 따라 광 경로를 좌측 또는 우측으로 제어할 수 있는 편광 격자를 활용하여 가상 영상의 시야각을 기존보다 최대 2배까지 확장할 수 있는 시스템을 제안한다. 이는 단일 도파관 기반 근안 디스플레이에서 영상 결합기 (imaging combiner)로 활용되는 회절 소자의 설계 변수에 의해 제한되는 시야각 한계점을 돌파할 수 있는 구조로 컴팩트한 폼 팩터로 더욱 몰입감 있는 대화면 증강현실 영상을 제공할 수 있다. 마지막으로 위에서 제안된 두 가지 편광 의존형 광학 소자를 모두 사용하여 시점 전환이 가능한 도파관 기반의 망막 투사형 디스플레이 구조를 제안한다. 편광 다중화를 통해 다중 초점들을 선택적으로 활성화함으로써, 확장된 시청영역을 제공하는 동시에 동공 크기 변화 또는 움직임에 의한 이중 영상 문제를 완화할 수 있다. 또한 기계적 움직임 없이 시점 간의 고속 전환이 가능하다는 장점을 지니고 있다. 본 박사학위 논문에서 제시한 편광 다중화를 활용한 새로운 결합기 광학 소자 및 광학 구조들은 도파관 기반 근안 디스플레이의 향상된 성능을 제공하는 해결책 및 새로운 가능성으로 제시할 수 있을 것이라 기대된다.Abstract i Contents iii List of Tables vi List of Figures vii Chapter. 1 Introduction 1 1.1 Augmented reality near-eye display 1 1.2 Key performance parameters of near-eye displays 4 1.3 Basic scheme of waveguide-based near-eye displays 22 1.4 Motivation and purpose of this dissertation 33 1.5 Scope and organization 37 Chapter 2 Dual-focal waveguide-based near-eye display using polarization-dependent combiner lens 39 2.1 Introduction 39 2.2 Optical design for polarization-dependent combiner lens 42 2.2.1 Design and principle of polarization-dependent combiner lens 42 2.2.2 Prototype implementation 48 2.3 Waveguide-based augmented reality near-eye display with dual-focal plane using polarization-dependent combiner lens 51 2.3.1 Implementation of the prototype and experimental results 51 2.3.2 Performance analysis and discussion 57 2.4 Conclusion 69 Chapter 3 Extended-field-of-view waveguide-based near-eye display via polarization-dependent steering combiner 70 3.1 Introduction 70 3.2 Optical design for polarization-dependent steering combiner 73 3.2.1 Principle of polarization grating 73 3.2.2 Principle of polarization-dependent steering combiner 76 3.2.3 Analysis and verification experiment for real-scene distortion 77 3.3 Waveguide-based augmented reality near-eye display with extended-field-of-view 81 3.3.1 Field-of-view for volume grating based waveguide technique 81 3.3.2 Implementation of the prototype and experimental results 84 3.3.3 Performances analysis and discussion 87 3.4 Conclusion 92 Chapter 4 Viewpoint switchable retinal-projection-based near-eye display with waveguide configuration 93 4.1 Introduction 93 4.2 Polarization-dependent switchable eyebox 97 4.2.1 Optical devices for polarization-dependent switching of viewpoints 97 4.2.2 System configuration for proposed method 100 4.2.3 Design of waveguide and imaging combiner 105 4.3 Compact retinal projection-based near-eye display with switchable viewpoints via waveguide configuration 114 4.3.1 Implementation of the prototype and experimental results 114 4.3.2 Performance analysis and discussion 118 4.4 Conclusion 122 Chapter 5. Conclusion 123 Bibliography 127 Appendix 135Docto

Patterned Liquid Crystal Devices for Near-eye Displays

As a promising next-generation display, augmented reality (AR) and virtual reality (VR) have shown attractive features and attracted broad interests from both academia and industry. Currently, these near-eye displays (NEDs) have enabled numerous applications, ranging from education, medical, entertainment, to engineering, with the help of compact and functional patterned liquid crystal (LC) devices. The interplay between LC patterns and NEDs stimulates the development of novel LC devices with unique surface alignments and volume structures, which in turn feedback to achieve more compact and versatile NEDs. This dissertation will focus on the patterned LC with applications in NEDs. Firstly, we propose and explain the working principles and generation of novel patterned LC devices, including LC configurations, surface alignment mechanism, polarization field generation, and fabrication process. Secondly, we theoretically analyze the optical properties of patterned LC devices, providing the optical efficiency, devices thickness, polarization selectivity, wavelength, and angular bandwidth. Based on the dimensions of the surface pattern, the LC devices can be divided into reflector, grating, and lens, respectively. Finally, we focus on the applications of these novel patterned LC devices to address some challenges in current NEDs. More specifically, achieving a high-resolution density in NEDs, especially for VR systems is an urgent issue. To enhance the resolution without introducing any extra burden to the system, we propose an elegant method with the combination of foveated view and polarization multiplexing, based on LC reflector. For LC grating, it shows a nearly 100% efficiency with a large diffraction angle, which is a perfect candidate for the waveguide-based AR systems. We propose and demonstrate the LC grating-based waveguide AR with benchtop demo and further performance optimization. For LC lens, it can achieve controllable power and large off-axis angle while maintaining high efficiency. These unique and attractive features give LC lenses the ability to achieve a glasses-like AR architecture while maintaining high optical efficiency. Based on this LC lens, we demonstrate a novel AR system design using polarization and time multiplexing methods to simultaneously obtain a double field of view and a glasses-like form factor. The proposed patterned LC devices for NED applications are validated by both optical simulation and experiment. Multiple tabletop demos are constructed to illustrate how these patterned LC devices can significantly improve the visual experiences of these next-generation NEDs

High-dynamic-range Foveated Near-eye Display System

Wearable near-eye display has found widespread applications in education, gaming, entertainment, engineering, military training, and healthcare, just to name a few. However, the visual experience provided by current near-eye displays still falls short to what we can perceive in the real world. Three major challenges remain to be overcome: 1) limited dynamic range in display brightness and contrast, 2) inadequate angular resolution, and 3) vergence-accommodation conflict (VAC) issue. This dissertation is devoted to addressing these three critical issues from both display panel development and optical system design viewpoints. A high-dynamic-range (HDR) display requires both high peak brightness and excellent dark state. In the second and third chapters, two mainstream display technologies, namely liquid crystal display (LCD) and organic light emitting diode (OLED), are investigated to extend their dynamic range. On one hand, LCD can easily boost its peak brightness to over 1000 nits, but it is challenging to lower the dark state to \u3c 0.01 nits. To achieve HDR, we propose to use a mini-LED local dimming backlight. Based on our simulations and subjective experiments, we establish practical guidelines to correlate the device contrast ratio, viewing distance, and required local dimming zone number. On the other hand, self-emissive OLED display exhibits a true dark state, but boosting its peak brightness would unavoidably cause compromised lifetime. We propose a systematic approach to enhance OLED\u27s optical efficiency while keeping indistinguishable angular color shift. These findings will shed new light to guide future HDR display designs. In Chapter four, in order to improve angular resolution, we demonstrate a multi-resolution foveated display system with two display panels and an optical combiner. The first display panel provides wide field of view for peripheral vision, while the second panel offers ultra-high resolution for the central fovea. By an optical minifying system, both 4x and 5x enhanced resolutions are demonstrated. In addition, a Pancharatnam-Berry phase deflector is applied to actively shift the high-resolution region, in order to enable eye-tracking function. The proposed design effectively reduces the pixelation and screen-door effect in near-eye displays. The VAC issue in stereoscopic displays is believed to be the main cause of visual discomfort and fatigue when wearing VR headsets. In Chapter five, we propose a novel polarization-multiplexing approach to achieve multiplane display. A polarization-sensitive Pancharatnam-Berry phase lens and a spatial polarization modulator are employed to simultaneously create two independent focal planes. This method enables generation of two image planes without the need of temporal multiplexing. Therefore, it can effectively reduce the frame rate by one-half. In Chapter six, we briefly summarize our major accomplishments

높은 공간 대역폭을 위한 복소 진폭 이미징 및 디스플레이 시스템

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2021. 2. 이병호.빛을 파동으로 이해하면 간섭과 회절을 포함한 다양한 광학 현상을 해석 할 수 있다. 미래 기술이라 불리는 홀로그램, 3차원 이미징 및 3차원 디스플레이 시스템들은 파동의 복소진폭을 이해하고 변조함으로써 구현될 수 있다. 현존하는 광공학 장치를 넘어서는 파동 광학에 기반한 광공학 장치들을 상용화 및 발전시키기 위해 많은 연구가 진행되어왔지만, 지금껏 구현된 장치들은 공간 대역폭 (space-bandwidth product)의 제한으로 인해 그 성능이 대중의 기대에 부합하기 어려움을 겪고있다. 본 논문은 복소 진폭 이미징 및 디스플레이 시스템에서 공간 대역폭을 향상 시키는 방법을 제안한다. 복소 진폭 변조 시스템의 성능은 광학 시스템의 정보량을 나타내는 공간 대역폭에 의해 제한된다. 이 공간 대역폭을 향상시키기 위하여 저자는 다양한 다중화 기술을 적용하며, 동시에 다중화된 정보를 분리하는 알고리즘과 장치를 고안한다. 첫번째로 디지털 홀로그래피 기술에 공간 주파수를 다중화해 대역폭을 효율적으로 활용하는 방법을 고안하여 획득된 홀로그램의 촬영 영역을 증가시킨다. 두번째로, 단일 촬영 푸리에 타이코그래피 (single-shot Fourier ptychography) 기술에서는 광 조사 다중화를 사용하여 이미지 센서에 기록되는 정보의 양을 확장시킨다. 다중화 된 정보를 분해하고 복소 진폭을 획득하기 위하여 새로운 광학 시스템과 전산 알고리즘을 고안하여 해상도가 향상된 복소 진폭을 획득한다. 세번째로, 저자는 홀로그램 디스플레이에 조명 다중화 및 시분할 기술을 적용한다. 다중화 된 정보는 인간의 인지적 시간 대역폭과 제안된 시스템의 공간 필터링의 결합으로 분해된다. 구현된 홀로그래픽 디스플레이는 공간 대역폭이 확장되어 더 넓은 시야각에 삼차원 홀로그램을 제공한다. 본 논문은 작은 공간대역폭이라는 공통된 문제를 공유하는 이미징 및 디스플레이 분야의 발전에 기여할 것으로 기대된다. 저자는 본 연구에서 제안된 방법이 다양한 복소 진폭 변조 시스템의 성능 향상에 영감을 주며, 나아가 삼차원 계측, 홀로그래피, 가상 및 증강현실을 포함한 다양한 미래 산업에 발전에 기여할 수 있기를 기대한다.Understanding light as a wave makes it possible to interpret a variety of phenomena, including interference and diffraction. By modulating the complex amplitude of the wave, hologram, three-dimensional imaging, and three-dimensional display system, called future technologies, can be implemented that surpass the currently commercialized optical engineering devices. Although a lot of research has been conducted to develop and commercialize the wave optical system, state-of-the-art devices are still far from the performance expected by the public due to the limited space-bandwidth product (SBP). This dissertation presents the studies on high SBP for complex amplitude imaging and display systems. The performance of a complex amplitude modulating system is limited by the SBP, which represents the amount of information in the optical system. To improve the SBP of the complex amplitude in a limited amount of information, the author applies various multiplexing techniques suitable for the implemented system. In practice, the spatial frequency multiplexed digital holography is devised by efficiently allocating frequency bandwidth with dual-wavelength light sources. The author also applies illumination multiplexing to the single-shot Fourier ptychography to expand the amount of information recorded in the image sensor. Computational reconstruction algorithm combined with novel optical design allows the acquired multiplexed information to be decomposed in the imaging system, leading to improvement of size of the image or resolution. Furthermore, the author applied illumination multiplexing and temporal multiplexing techniques to holographic displays. The multiplexed information is decomposed by a combination of human perceptual temporal bandwidth and spatial filtering. The SBP enhanced holographic display is implemented, providing a more wide viewing angle. It is expected that this thesis will contribute to the development of the imaging and display fields, which share a common problem of small SBP. The author hopes that the proposed methods will inspire various researchers to approach the implementation of complex amplitude modulating systems, and various future industries, including 3-D inspection, holography, virtual reality, and augmented reality will be realized with high-performance.Abstract i Contents iii List of Tables vi List of Figures vii 1 Introduction 1 1.1 Complex Amplitude of Wave 1 1.2 Complex Amplitude Optical System 3 1.3 Motivation and Purpose of the Dissertation 5 1.4 Scope and Organization 8 2 Space-Bandwidth Product 10 2.1 Overview of Space-Bandwidth Product 10 2.2 Space-Bandwidth Product of Complex Amplitude Imaging Systems 11 2.3 Space-Bandwidth Product of Complex Amplitude Display Systems 13 3 Double Size Complex Amplitude Imaging via Digital Holography 15 3.1 Introduction 15 3.1.1 Digital Holography 16 3.1.2 Frequency Multiplexed Digital Holography 22 3.1.3 Adaptation of Diffractive Grating for Simple Interferometer 24 3.2 Principle 26 3.2.1 Single Diffraction Grating Off-Axis Digital Holography 26 3.2.2 Double Size Implementation with Multiplexed Illumination 29 3.3 Implementation 32 3.4 Experimental Results 34 3.4.1 Resolution Assessment 34 3.4.2 Imaging Result 36 3.4.3 Quantitative 3-D Measurement 38 3.5 Conclusion 42 4 High-Resolution Complex Amplitude Imaging via Fourier Ptychographic Microscopy 43 4.1 Introduction 43 4.1.1 Phase Retrieval 45 4.1.2 Fourier Ptychographic Microscopy 47 4.2 Principle 52 4.2.1 Imaging System for Single-Shot Fourier Ptychographic Microscopy 52 4.2.2 Multiplexed Illumination 55 4.2.3 Reconstruction Algorithm 58 4.3 Implementation 60 4.3.1 Numerical Simulation 60 4.3.2 System Design 64 4.4 Results and Assessment 65 4.4.1 Resolution 65 4.4.2 Phase Retrieval of Biological Specimen 68 4.5 Discussion 71 4.6 Conclusion 73 5 Viewing Angle Enhancement for Holographic Display 74 5.1 Introduction 74 5.1.1 Complex Amplitude Representation 76 5.1.2 DMD Holographic Displays 79 5.2 Principle 81 5.2.1 Structured Illumination 81 5.2.2 TM with Array System 83 5.2.3 Time Domain Design 84 5.3 Implementation 85 5.3.1 Hardware Design 85 5.3.2 Frequency Domain Design 85 5.3.3 Aberration Correction 87 5.4 Results 88 5.5 Discussion 92 5.5.1 Speckle 92 5.5.2 Applications for Near-eye Displays 94 5.6 Conclusion 99 6 Conclusion 100 Appendix 116 Abstract (In Korean) 117Docto