On the Interplay of Foveated Rendering and Video Encoding

Publisher Copyright: © 2020 Owner/Author.Humans have sharp central vision but low peripheral visual acuity. Prior work has taken advantage of this phenomenon in two ways: foveated rendering (FR) reduces the computational workload of rendering by producing lower visual quality for peripheral regions and foveated video encoding (FVE) reduces the bitrate of streamed video through heavier compression of peripheral regions. Remote rendering systems require both rendering and video encoding and the two techniques can be combined to reduce both computing and bandwidth consumption. We report early results from such a combination with remote VR rendering. The results highlight that FR causes large bitrate overhead when combined with normal video encoding but combining it with FVE can mitigate it.Peer reviewe

Accelerated Foveated Rendering based on Adaptive Tessellation

We propose an optimization method for adaptive geometric tessellation, involving the saccadic motion of the human eye and foveated rendering. Increased demands on computational resources, especially in the field of head-mounted devices with gaze contingency make optimization schemes pertinent for a seamless user experience. For implementing foveated rendering, our algorithm tessellates a 3D model in real-time based on the location of the user's gaze, substituted with a mouse cursor in this project as a proof of concept. Saccades and fixations of the human eye are simulated by delaying the process of tessellation and rendering by the minimum time taken to complete a saccade. Calculations required for tessellation and rendering the changes on the screen are stalled as and when the eye fixates after a saccade. The paper walks through our contribution by describing the theory, the application method, and results from our user study evaluating our method.<br/

Real-time Human Eye Resolution Ray Tracing in Mixed Reality

Mixed reality applications require natural visualizations. Ray tracing is one of the candidates for this purpose. Real-time ray tracing is slowly becoming a reality in consumer market mixed and virtual reality. This is happening due to development in display technologies and computer hardware. Some of these examples are foveated rendering enabled high resolution displays, like Varjo mixed reality headset, and parallel computing enablers, like GPUs getting ray tracing hardware acceleration enablers, such as for example Nvidia's RTX. Currently, the challenge in ray tracing is resource need especially in mixed reality where low latency is wanted and with human eye resolution where high resolution needs are obvious. In this paper, we design and implement a novel foveated ray tracing solution called Human Eye Resolution Ray Tracer (HERR) that achieves real-time frame rates in human eye resolution in mixed reality.Peer reviewe

Event Guided Depth Sensing

Active depth sensors like structured light, lidar, and time-of-flight systems sample the depth of the entire scene uniformly at a fixed scan rate. This leads to limited spatiotemporal resolution where redundant static information is over-sampled and precious motion information might be under-sampled. In this paper, we present an efficient bio-inspired event-camera-driven depth estimation algorithm. In our approach, we dynamically illuminate areas of interest densely, depending on the scene activity detected by the event camera, and sparsely illuminate areas in the field of view with no motion. The depth estimation is achieved by an event-based structured light system consisting of a laser point projector coupled with a second event-based sensor tuned to detect the reflection of the laser from the scene. We show the feasibility of our approach in a simulated autonomous driving scenario and real indoor sequences using our prototype. We show that, in natural scenes like autonomous driving and indoor environments, moving edges correspond to less than 10% of the scene on average. Thus our setup requires the sensor to scan only 10% of the scene, which could lead to almost 90% less power consumption by the illumination source. While we present the evaluation and proof-of-concept for an event-based structured-light system, the ideas presented here are applicable for a wide range of depth sensing modalities like LIDAR, time-of-flight, and standard stereo

Event Guided Depth Sensing

Active depth sensors like structured light, lidar, and time-of-flight systems sample the depth of the entire scene uniformly at a fixed scan rate. This leads to limited spatiotemporal resolution where redundant static information is over-sampled and precious motion information might be under-sampled. In this paper, we present an efficient bio-inspired event-camera-driven depth estimation algorithm. In our approach, we dynamically illuminate areas of interest densely, depending on the scene activity detected by the event camera, and sparsely illuminate areas in the field of view with no motion. The depth estimation is achieved by an event-based structured light system consisting of a laser point projector coupled with a second event-based sensor tuned to detect the reflection of the laser from the scene. We show the feasibility of our approach in a simulated autonomous driving scenario and real indoor sequences using our prototype. We show that, in natural scenes like autonomous driving and indoor environments, moving edges correspond to less than 10% of the scene on average. Thus our setup requires the sensor to scan only 10% of the scene, which could lead to almost 90% less power consumption by the illumination source. While we present the evaluation and proof-of-concept for an event-based structured-light system, the ideas presented here are applicable for a wide range of depth sensing modalities like LIDAR, time-of-flight, and standard stereo