Foveated Video Streaming for Cloud Gaming

Good user experience with interactive cloud-based multimedia applications, such as cloud gaming and cloud-based VR, requires low end-to-end latency and large amounts of downstream network bandwidth at the same time. In this paper, we present a foveated video streaming system for cloud gaming. The system adapts video stream quality by adjusting the encoding parameters on the fly to match the player's gaze position. We conduct measurements with a prototype that we developed for a cloud gaming system in conjunction with eye tracker hardware. Evaluation results suggest that such foveated streaming can reduce bandwidth requirements by even more than 50% depending on parametrization of the foveated video coding and that it is feasible from the latency perspective.Comment: Submitted to: IEEE 19th International Workshop on Multimedia Signal Processin

FoVR: Attention-based VR Streaming through Bandwidth-limited Wireless Networks

Consumer Virtual Reality (VR) has been widely used in various application areas, such as entertainment and medicine. In spite of the superb immersion experience, to enable high-quality VR on untethered mobile devices remains an extremely challenging task. The high bandwidth demands of VR streaming generally overburden a conventional wireless connection, which affects the user experience and in turn limits the usability of VR in practice. In this paper, we propose FoVR, attention-based hierarchical VR streaming through bandwidth-limited wireless networks. The design of FoVR stems from the insight that human's vision is hierarchical, so that different areas in the field of view (FoV) can be served with VR content of different qualities. By exploiting the gaze tracking capacity of the VR devices, FoVR is able to accurately predict the user's attention so that the streaming of hierarchical VR can be appropriately scheduled. In this way, FoVR significantly reduces the bandwidth cost and computing cost while keeping high quality of user experience. We implement FoVR on a commercial VR device and evaluate its performance in various scenarios. The experiment results show that FoVR reduces the bandwidth cost by 88.9% and 76.2%, respectively compared to the original VR streaming and the state-of-the-art approach

D-SAV360: A Dataset of Gaze Scanpaths on 360° Ambisonic Videos

Understanding human visual behavior within virtual reality environments is crucial to fully leverage their potential. While previous research has provided rich visual data from human observers, existing gaze datasets often suffer from the absence of multimodal stimuli. Moreover, no dataset has yet gathered eye gaze trajectories (i.e., scanpaths) for dynamic content with directional ambisonic sound, which is a critical aspect of sound perception by humans. To address this gap, we introduce D-SAV360, a dataset of 4,609 head and eye scanpaths for 360° videos with first-order ambisonics. This dataset enables a more comprehensive study of multimodal interaction on visual behavior in virtual reality environments. We analyze our collected scanpaths from a total of 87 participants viewing 85 different videos and show that various factors such as viewing mode, content type, and gender significantly impact eye movement statistics. We demonstrate the potential of D-SAV360 as a benchmarking resource for state-of-the-art attention prediction models and discuss its possible applications in further research. By providing a comprehensive dataset of eye movement data for dynamic, multimodal virtual environments, our work can facilitate future investigations of visual behavior and attention in virtual reality

Remote Visual Observation of Real Places Through Virtual Reality Headsets

Virtual Reality has always represented a fascinating yet powerful opportunity that has attracted studies and technology developments, especially since the latest release on the market of powerful high-resolution and wide field-of-view VR headsets. While the great potential of such VR systems is common and accepted knowledge, issues remain related to how to design systems and setups capable of fully exploiting the latest hardware advances. The aim of the proposed research is to study and understand how to increase the perceived level of realism and sense of presence when remotely observing real places through VR headset displays. Hence, to produce a set of guidelines that give directions to system designers about how to optimize the display-camera setup to enhance performance, focusing on remote visual observation of real places. The outcome of this investigation represents unique knowledge that is believed to be very beneficial for better VR headset designs towards improved remote observation systems. To achieve the proposed goal, this thesis presents a thorough investigation of existing literature and previous researches, which is carried out systematically to identify the most important factors ruling realism, depth perception, comfort, and sense of presence in VR headset observation. Once identified, these factors are further discussed and assessed through a series of experiments and usability studies, based on a predefined set of research questions. More specifically, the role of familiarity with the observed place, the role of the environment characteristics shown to the viewer, and the role of the display used for the remote observation of the virtual environment are further investigated. To gain more insights, two usability studies are proposed with the aim of defining guidelines and best practices. The main outcomes from the two studies demonstrate that test users can experience an enhanced realistic observation when natural features, higher resolution displays, natural illumination, and high image contrast are used in Mobile VR. In terms of comfort, simple scene layouts and relaxing environments are considered ideal to reduce visual fatigue and eye strain. Furthermore, sense of presence increases when observed environments induce strong emotions, and depth perception improves in VR when several monocular cues such as lights and shadows are combined with binocular depth cues. Based on these results, this investigation then presents a focused evaluation on the outcomes and introduces an innovative eye-adapted High Dynamic Range (HDR) approach, which the author believes to be of great improvement in the context of remote observation when combined with eye-tracked VR headsets. Within this purpose, a third user study is proposed to compare static HDR and eye-adapted HDR observation in VR, to assess that the latter can improve realism, depth perception, sense of presence, and in certain cases even comfort. Results from this last study confirmed the author expectations, proving that eye-adapted HDR and eye tracking should be used to achieve best visual performances for remote observation in modern VR systems

Recording, Analysis and Playback of Spatial Sound Field using Novel Design Methods of Transducer Arrays

Nowadays, a growing interest in the recording and reproduction of spatial audio has been observed. With virtual and augmented reality technologies spreading fast thanks to entertainment and video game industries, also the professional opportunities in the field of engineering are evolving. However, despite many microphone arrays are reaching the market, most of them is not optimized for engineering or diagnostic use and remains mainly confined to voice and music recordings. In this thesis, the design of two new systems for recording and analysing the spatial distribution of sound energy, employing arrays of transducers and cameras, is discussed. Both acoustic and visual spatial information is recorded and combined together to produce static and dynamic colour maps, with a specially designed software and employing Ambisonics and Spatial PCM Sampling (SPS), two common spatial audio formats, for signals processing. The first solution consists in a microphone array made of 32 capsules and a circular array of eight cameras, optimized for low frequencies. The size of the array is designed accordingly to the frequency range of interest for automotive Noise, Vibration & Harshness (NVH) applications. The second system is an underwater probe with four hydrophones and a panoramic camera, with which it is possible to monitor the effects of underwater noise produced by human activities on marine species. Finite Elements Method (FEM) simulations have been used to calculate the array response, thus deriving the filtering matrix and performing theoretical evaluation of the spatial performance. Field tests of the proposed solutions are presented in comparison with the current state-of-the-art equipment. The faithful reproduction of the spatial sound field arouses equally interest. Hence, a method to playback panoramic video with spatial audio is presented, making use of Virtual Reality (VR) technology, spatial audio, individualized Head Related Transfer Functions (HRTFs) and personalized headphones equalization. The work in its entirety presents a complete methodology for recording, analysing and reproducing the spatial information of soundscapes