A Systematic Review of Extended Reality (XR) for Understanding and Augmenting Vision Loss

Over the past decade, extended reality (XR) has emerged as an assistive technology not only to augment residual vision of people losing their sight but also to study the rudimentary vision restored to blind people by a visual neuroprosthesis. To make the best use of these emerging technologies, it is valuable and timely to understand the state of this research and identify any shortcomings that are present. Here we present a systematic literature review of 227 publications from 106 different venues assessing the potential of XR technology to further visual accessibility. In contrast to other reviews, we sample studies from multiple scientific disciplines, focus on augmentation of a person's residual vision, and require studies to feature a quantitative evaluation with appropriate end users. We summarize prominent findings from different XR research areas, show how the landscape has changed over the last decade, and identify scientific gaps in the literature. Specifically, we highlight the need for real-world validation, the broadening of end-user participation, and a more nuanced understanding of the suitability and usability of different XR-based accessibility aids. By broadening end-user participation to early stages of the design process and shifting the focus from behavioral performance to qualitative assessments of usability, future research has the potential to develop XR technologies that may not only allow for studying vision loss, but also enable novel visual accessibility aids with the potential to impact the lives of millions of people living with vision loss

Size and shape constancy in consumer virtual reality

With the increase in popularity of consumer virtual reality headsets, for research and other applications, it is important to understand the accuracy of 3D perception in VR. We investigated the perceptual accuracy of near-field virtual distances using a size and shape constancy task, in two commercially available devices. Participants wore either the HTC Vive or the Oculus Rift and adjusted the size of a virtual stimulus to match the geometric qualities (size and depth) of a physical stimulus they were able to refer to haptically. The judgments participants made allowed for an indirect measure of their perception of the egocentric, virtual distance to the stimuli. The data show under-constancy and are consistent with research from carefully calibrated psychophysical techniques. There was no difference in the degree of constancy found in the two headsets. We conclude that consumer virtual reality headsets provide a sufficiently high degree of accuracy in distance perception, to allow them to be used confidently in future experimental vision science, and other research applications in psychology

Perception-driven approaches to real-time remote immersive visualization

In remote immersive visualization systems, real-time 3D perception through RGB-D cameras, combined with modern Virtual Reality (VR) interfaces, enhances the user’s sense of presence in a remote scene through 3D reconstruction rendered in a remote immersive visualization system. Particularly, in situations when there is a need to visualize, explore and perform tasks in inaccessible environments, too hazardous or distant. However, a remote visualization system requires the entire pipeline from 3D data acquisition to VR rendering satisfies the speed, throughput, and high visual realism. Mainly when using point-cloud, there is a fundamental quality difference between the acquired data of the physical world and the displayed data because of network latency and throughput limitations that negatively impact the sense of presence and provoke cybersickness. This thesis presents state-of-the-art research to address these problems by taking the human visual system as inspiration, from sensor data acquisition to VR rendering. The human visual system does not have a uniform vision across the field of view; It has the sharpest visual acuity at the center of the field of view. The acuity falls off towards the periphery. The peripheral vision provides lower resolution to guide the eye movements so that the central vision visits all the interesting crucial parts. As a first contribution, the thesis developed remote visualization strategies that utilize the acuity fall-off to facilitate the processing, transmission, buffering, and rendering in VR of 3D reconstructed scenes while simultaneously reducing throughput requirements and latency. As a second contribution, the thesis looked into attentional mechanisms to select and draw user engagement to specific information from the dynamic spatio-temporal environment. It proposed a strategy to analyze the remote scene concerning the 3D structure of the scene, its layout, and the spatial, functional, and semantic relationships between objects in the scene. The strategy primarily focuses on analyzing the scene with models the human visual perception uses. It sets a more significant proportion of computational resources on objects of interest and creates a more realistic visualization. As a supplementary contribution, A new volumetric point-cloud density-based Peak Signal-to-Noise Ratio (PSNR) metric is proposed to evaluate the introduced techniques. An in-depth evaluation of the presented systems, comparative examination of the proposed point cloud metric, user studies, and experiments demonstrated that the methods introduced in this thesis are visually superior while significantly reducing latency and throughput

편광 다중화를 이용하여 향상된 기능을 제공하는 도파관 기반의 근안 디스플레이

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2021. 2. 이병호.This dissertation presents the studies on the optical design method that enhances the display performance of see-through waveguide-based near-eye displays (WNEDs) using the polarization multiplexing technique. The studies focus on the strategies to improve the crucial display performances without compromising a small form factor, the most attractive merit of the WNEDs. To achieve this goal, thin and lightweight polarization-dependent optical elements are devised and employed in the WNED structure. The polarization-dependent devices can allow multiple optical functions or optical paths depending on the polarization state of the input beam, which can break through the limitation of the waveguide system with the polarization multiplexing. To realize the function-selective eyepiece for AR applications, the proposed devices should operate as an optically transparent window for the real scene while performing specific optical functions for the virtual image. The proposed devices are manufactured in a combination structure in which polarization-dependent optical elements are stacked. The total thickness of the stacked structure is about 1 mm, and it can be attached to the waveguide surface without conspicuously increasing the form factor of the optical system. Using the proposed polarization-dependent devices, the author proposes three types of novel WNED systems with enhanced performance. First, the author suggests a compact WNED with dual focal planes. Conventional WNEDs have an inherent limitation that the focal plane of the virtual image is at an infinite distance because they extract a stream of collimated light at the out-coupler. By using the polarization-dependent eyepiece lens, an additional focal plane can be generated with the polarization multiplexing in addition to infinity depth. The proposed configuration can provide comfortable AR environments by alleviating visual fatigue caused by vergence-accommodation conflict. Second, the novel WNED configuration with extended field-of-view (FOV) is presented. In the WNEDs, the maximum allowable FOV is determined by the material properties of the diffraction optics and the substrate. By using the polarization-dependent steering combiner, the FOV can be extended up to two times, which can provide more immersive AR experiences. In addition, this dissertation demonstrates that the distortion for the real scene caused by the stacked structure cannot severely disturb the image quality, considering the acuity of human vision. Lastly, the author presents a retinal projection-based WNED with switchable viewpoints by simultaneously adopting the polarization-dependent lens and grating. The proposed system can convert the viewpoint according to the position of the eye pupil without mechanical movement. The polarization-dependent viewpoint switching can resolve the inherent problem of a narrow eyebox in retinal projection displays without employing the bulky optics for mechanical movement. In conclusion, the dissertation presents the practical optical design and detailed analysis for enhanced WNED based on the polarization multiplexing technique through various simulations and experiments. The proposed approaches are expected to be utilized as an innovative solution for compact wearable displays.본 박사학위 논문에서는 편광 다중화 기법을 이용하여 도파관 기반의 증강현실 근안 디스플레이의 성능을 향상시키는 광학 설계 및 이에 대한 분석에 대해 논의한다. 본 연구는 도파관 기반 근안 디스플레이의 가장 큰 장점인 소형 폼 팩터를 유지하면서 디스플레이 성능을 개선하는 것에 중점을 둔다. 이를 위해 기존 광학 소자에 비해 매우 가볍고 얇은 편광 의존형 결합기 광학 소자가 새롭게 제안되며, 이는 입사광의 편광 상태에 따라 독립적인 광 경로 제어를 가능케 하여 편광 다중화를 통해 향상된 성능을 제공 할 수 있다. 또한 실제 영상의 빛은 그대로 투과 시킴으로써 증강현실을 위한 소자로 활용 가능하다. 본 연구에서 제안하는 편광 의존형 결합기 광학 소자는 기하학적 위상(geometric phase, GP)에 기반하여 동작한다. GP 기반의 광학소자가 서로 직교하는 원형 편광 입사광에 대해 상보적인 기능을 수행하는 것을 이용하여, 두 개 이상의 GP 소자와 편광 제어를 위한 광학 필름들을 중첩 시킴으로써 증강현실 결합기 광학 소자를 구현할 수 있다. 이들 광학소자는 매우 얇기 때문에, 본 연구에서 제작된 편광 의존형 결합기 광학 소자의 총 두께는 1 mm 수준으로 폼 팩터 제약으로부터 자유롭다. 또한 평평한 필름 형태이므로, 평판형 도파관에 부착하기 쉽다는 이점을 지닌다. 고안된 편광 의존형 결합기 광학 소자를 사용하여 세 가지 유형의 새로운 도파관 기반의 근안 디스플레이 구조를 제안한다. 첫 번째는 입사광의 편광 상태에 따라 투명 광학 창 또는 오목 렌즈로 작동하는 편광 의존형 결합기 렌즈를 적용하여 가상 영상에 대해 이중 초점면을 제공하는 시스템이다. 제안된 구조는 기존의 도파관 기반 근안 디스플레이가 무한대 위치에 단일 초점면을 제공함으로써 발생하는 시각적 피로 및 흐릿한 증강현실 영상의 문제를 완화할 수 있다. 두 번째로는 입사광의 편광 상태에 따라 광 경로를 좌측 또는 우측으로 제어할 수 있는 편광 격자를 활용하여 가상 영상의 시야각을 기존보다 최대 2배까지 확장할 수 있는 시스템을 제안한다. 이는 단일 도파관 기반 근안 디스플레이에서 영상 결합기 (imaging combiner)로 활용되는 회절 소자의 설계 변수에 의해 제한되는 시야각 한계점을 돌파할 수 있는 구조로 컴팩트한 폼 팩터로 더욱 몰입감 있는 대화면 증강현실 영상을 제공할 수 있다. 마지막으로 위에서 제안된 두 가지 편광 의존형 광학 소자를 모두 사용하여 시점 전환이 가능한 도파관 기반의 망막 투사형 디스플레이 구조를 제안한다. 편광 다중화를 통해 다중 초점들을 선택적으로 활성화함으로써, 확장된 시청영역을 제공하는 동시에 동공 크기 변화 또는 움직임에 의한 이중 영상 문제를 완화할 수 있다. 또한 기계적 움직임 없이 시점 간의 고속 전환이 가능하다는 장점을 지니고 있다. 본 박사학위 논문에서 제시한 편광 다중화를 활용한 새로운 결합기 광학 소자 및 광학 구조들은 도파관 기반 근안 디스플레이의 향상된 성능을 제공하는 해결책 및 새로운 가능성으로 제시할 수 있을 것이라 기대된다.Abstract i Contents iii List of Tables vi List of Figures vii Chapter. 1 Introduction 1 1.1 Augmented reality near-eye display 1 1.2 Key performance parameters of near-eye displays 4 1.3 Basic scheme of waveguide-based near-eye displays 22 1.4 Motivation and purpose of this dissertation 33 1.5 Scope and organization 37 Chapter 2 Dual-focal waveguide-based near-eye display using polarization-dependent combiner lens 39 2.1 Introduction 39 2.2 Optical design for polarization-dependent combiner lens 42 2.2.1 Design and principle of polarization-dependent combiner lens 42 2.2.2 Prototype implementation 48 2.3 Waveguide-based augmented reality near-eye display with dual-focal plane using polarization-dependent combiner lens 51 2.3.1 Implementation of the prototype and experimental results 51 2.3.2 Performance analysis and discussion 57 2.4 Conclusion 69 Chapter 3 Extended-field-of-view waveguide-based near-eye display via polarization-dependent steering combiner 70 3.1 Introduction 70 3.2 Optical design for polarization-dependent steering combiner 73 3.2.1 Principle of polarization grating 73 3.2.2 Principle of polarization-dependent steering combiner 76 3.2.3 Analysis and verification experiment for real-scene distortion 77 3.3 Waveguide-based augmented reality near-eye display with extended-field-of-view 81 3.3.1 Field-of-view for volume grating based waveguide technique 81 3.3.2 Implementation of the prototype and experimental results 84 3.3.3 Performances analysis and discussion 87 3.4 Conclusion 92 Chapter 4 Viewpoint switchable retinal-projection-based near-eye display with waveguide configuration 93 4.1 Introduction 93 4.2 Polarization-dependent switchable eyebox 97 4.2.1 Optical devices for polarization-dependent switching of viewpoints 97 4.2.2 System configuration for proposed method 100 4.2.3 Design of waveguide and imaging combiner 105 4.3 Compact retinal projection-based near-eye display with switchable viewpoints via waveguide configuration 114 4.3.1 Implementation of the prototype and experimental results 114 4.3.2 Performance analysis and discussion 118 4.4 Conclusion 122 Chapter 5. Conclusion 123 Bibliography 127 Appendix 135Docto

Enhancing Quality Assurance using Virtual Design Engineering: Case Study of Space Shuttle Challenger

Virtual Design Engineering is an emerging method of increasing quality of systems. Including Virtual Design as a part of the traditional established Failure Mode, Effects, and Criticality Analysis process greatly enhances hazard and risk analysis while reducing overall costs. In this study these enhancements are explored and expanded upon to discover how overall system quality could be increased and all stakeholders could more accurately understand the hazards involved. Stakeholder misunderstanding or misapplication of hazards is of great importance to complex systems. An illustrative example of how these factors could have changed the outcome of a real-world engineering failure is provided

How can Extended Reality Help Individuals with Depth Misperception?

Despite the recent actual uses of Extended Reality (XR) in treatment of patients, some areas are less explored. One gap in research is how XR can improve depth perception for patients. Accordingly, the depth perception process in XR settings and in human vision are explored and trackers, visual sensors, and displays as assistive tools of XR settings are scrutinized to extract their potentials in influencing users’ depth perception experience. Depth perception enhancement is relying not only on depth perception algorithms, but also on visualization algorithms, display new technologies, computation power enhancements, and vision apparatus neural mechanism knowledge advancements. Finally, it is discussed that XR holds assistive features not only for the improvement of vision impairments but also for the diagnosis part. Although, each specific patient requires a specific set of XR setting due to different neural or cognition reactions in different individuals with same the disease

Improving Depth Perception in Immersive Media Devices by Addressing Vergence-Accommodation Conflict

: Recently, immersive media devices have seen a boost in popularity. However, many problems still remain. Depth perception is a crucial part of how humans behave and interact with their environment. Convergence and accommodation are two physiological mechanisms that provide important depth cues. However, when humans are immersed in virtual environments, they experience a mismatch between these cues. This mismatch causes users to feel discomfort while also hindering their ability to fully perceive object distances. To address the conflict, we have developed a technique that encompasses inverse blurring into immersive media devices. For the inverse blurring, we utilize the classical Wiener deconvolution approach by proposing a novel technique that is applied without the need for an eye-tracker and implemented in a commercial immersive media device. The technique's ability to compensate for the vergence-accommodation conflict was verified through two user studies aimed at reaching and spatial awareness, respectively. The two studies yielded a statistically significant 36% and 48% error reduction in user performance to estimate distances, respectively. Overall, the work done demonstrates how visual stimuli can be modified to allow users to achieve a more natural perception and interaction with the virtual environment