Autopoiesis through agency in virtual reality nonfiction

Documentary filmmakers are gradually embracing immersive media to create novel Virtual Reality Nonfiction (VRNF) content. Over the past twenty years initial experimentation in this new medium has brought forward numerous linearly structured 360° documentaries that maintain a close link to traditional documentary modes. More recently, we have observed a shift from the relatively passive 360° cinema towards more open-world, non-linear, game-like interactive experiences that challenge traditional definitions of the documentary genre. Volumetric world-building techniques provide nonfiction creators with additional tools that afford ‘viewer-users’ spatial and interactive agency, leading to a heightened autopoietic realisation of the storyworld. VRNF creators have the potential to allow their viewer-users enhanced control over framing, temporal ordering of the plot and spatial unfolding of the diegetic world, thus inviting them to become actual co-creators of a deeply personal and personalized experience. This article addresses how VRNF may go beyond the mere ‘documentation’ of people, places or past events that existed in a pre-filmic reality and provide viewer-users through augmented agency a unique present-tense autopoietic experience that pushes the boundaries of traditional 2D documentary.<br/

Space Time Exploration of Musical Instruments

Musical instruments are tools used to generate sounds for musical expression. Virtual Reality (VR) and Augmented Reality (AR) musical instruments create sounds that may be spatially disjointed from the instrument controls. Spatial audio processing can be used to position the Extended Reality (XR) musical instruments and their corresponding sounds in the same space. This dissertation investigates novel ways of combining spatial reverb models to improve the naturalness of XR musical instruments. Seven spatial reverb systems, combinations of a shoebox spatial reverb model, a raytracing spatial reverb model, and measured directional room impulse response convolution reverb, were compared in a pilot study. A novel hybrid system of synthetic early reflections and directional room impulse responses was preferred for naturalness when tested over headphones with three instruments created by the author: AR electric guitar, AR drumset, and VR Singing Kite. This research culminated in a concert, Spherical Sound Search, which showcased the preferred hybrid system, the three XR musical instruments, and four re-contextualized spatial audio effects (spatial looping, spatial delay, spatial feedback, and spatial compression). The three pieces in the concert explored different aspects of XR modalities and presented the novel system with spatial audio effects to a larger audience by rendering to an octophonic loudspeaker layout

Capture, Reconstruction, and Representation of the Visual Real World for Virtual Reality

We provide an overview of the concerns, current practice, and limitations for capturing, reconstructing, and representing the real world visually within virtual reality. Given that our goals are to capture, transmit, and depict complex real-world phenomena to humans, these challenges cover the opto-electro-mechanical, computational, informational, and perceptual fields. Practically producing a system for real-world VR capture requires navigating a complex design space and pushing the state of the art in each of these areas. As such, we outline several promising directions for future work to improve the quality and flexibility of real-world VR capture systems

Enhancing the broadcasted TV consumption experience with broadband omnidirectional video content

[EN] The current wide range of heterogeneous consumption devices and delivery technologies, offers the opportunity to provide related contents in order to enhance and enrich the TV consumption experience. This paper describes a solution to handle the delivery and synchronous consumption of traditional broadcast TV content and related broadband omnidirectional video content. The solution is intended to support both hybrid (broadcast/broadband) delivery technologies and has been designed to be compatible with the Hybrid Broadcast Broadband TV (HbbTV) standard. In particular, some specifications of HbbTV, such as the use of global timestamps or discovery mechanisms, have been adopted. However, additional functionalities have been designed to achieve accurate synchronization and to support the playout of omnidirectional video content in current consumption devices. In order to prove that commercial hybrid environments could be immediately enhanced with this type of content, the proposed solution has been included in a testbed, and objectively and subjectively evaluated. Regarding the omnidirectional video content, the two most common types of projections are supported: equirectangular and cube map. The results of the objective assessment show that the playout of broadband delivered omnidirectional video content in companion devices can be accurately synchronized with the playout on TV of traditional broadcast 2D content. The results of the subjective assessment show the high interest of users in this type of new enriched and immersive experience that contributes to enhance their Quality of Experience (QoE) and engagement.This work was supported by the Generalitat Valenciana, Investigacion Competitiva Proyectos, through the Research and Development Program Grants for Research Groups to be Consolidated, under Grant AICO/2017/059 and Grant AICO/2017Marfil-Reguero, D.; Boronat, F.; López, J.; Vidal Meló, A. (2019). Enhancing the broadcasted TV consumption experience with broadband omnidirectional video content. IEEE Access. 7:171864-171883. https://doi.org/10.1109/ACCESS.2019.2956084S171864171883

Virtual reality-based interventions for patients with paranoia:A systematic review

Background and objective: Paranoia is an important psychiatric symptom with a remarkable effect on daily life. Virtual reality (VR)-based treatments are influential and safe for patients with paranoia. This study aimed to evaluate the effectiveness, and define the clinical and technical characteristics of available VR strategies for the treatment of patients with paranoia. Materials and methods: Studies published up to 25/11/2021 reporting VR-based interventions for the treatment of patients with paranoia were reviewed in five databases, including PubMed, Embase, Web of Science, PsycINFO, and Scopus. Results: Out of 302 initial search results, eight were included in the present study based on the inclusion criteria. Six studies were randomized clinical trials with the interventions in the experimental group being based on VR, compared to routine interventions as controls. Two were before-after studies. The most commonly used hardware and software were head-mounted display and Unity3D, respectively. Interventions had a range of 1-16 sessions with follow-up durations of 0-6 months. All investigations showed positive results in the main target, including improved social participation, reduced level of anxiety, as well as diminished suspicious ideas and paranoid symptoms. Conclusions: Our findings demonstrated that VR-based interventions are effective treatments. Although the use of VR technology is limited for a variety of reasons, such as cost, it improves symptoms in patients with paranoia

From Capture to Display: A Survey on Volumetric Video

Volumetric video, which offers immersive viewing experiences, is gaining increasing prominence. With its six degrees of freedom, it provides viewers with greater immersion and interactivity compared to traditional videos. Despite their potential, volumetric video services poses significant challenges. This survey conducts a comprehensive review of the existing literature on volumetric video. We firstly provide a general framework of volumetric video services, followed by a discussion on prerequisites for volumetric video, encompassing representations, open datasets, and quality assessment metrics. Then we delve into the current methodologies for each stage of the volumetric video service pipeline, detailing capturing, compression, transmission, rendering, and display techniques. Lastly, we explore various applications enabled by this pioneering technology and we present an array of research challenges and opportunities in the domain of volumetric video services. This survey aspires to provide a holistic understanding of this burgeoning field and shed light on potential future research trajectories, aiming to bring the vision of volumetric video to fruition.Comment: Submitte

VR-based Soundscape Evaluation: Auralising the Sound from Audio Rendering, Reflection Modelling to Source Synthesis in the Acoustic Environment

Soundscape has been growing as a research field associated with acoustics, urban planning, environmental psychology and other disciplines since it was first introduced in the 1960s. To assess soundscapes, subjective validation is frequently integrated with soundscape reproduction. However, the existing soundscape standards do not give clear reproduction specifications to recreate a virtual sound environment. Selecting appropriate audio rendering methods, simulating sound propagation, and synthesising non-point sound sources remain major challenges for researchers. This thesis therefore attempts to give alternative or simplified strategies to reproduce a virtual sound environment by suggesting binaural or monaural audio renderings, reflection modelling during sound propagation, and less synthesis points of non-point sources. To solve these unclear issues, a systematic review of original studies first examines the ecological validity of immersive virtual reality in soundscape evaluation. Through recording and reproducing audio-visual stimuli of sound environments, participants give their subjective responses according to the structured questionnaires. Thus, different audio rendering, reflection modelling, and source synthesis methods are validated by subjective evaluation. The results of this thesis reveal that a rational setup of VR techniques and evaluation methods will be a solid foundation for soundscape evaluation with reliable ecological validity. For soundscape audio rendering, the binaural rendering still dominates the soundscape evaluation compared with the monaural. For sound propagation with consideration of different reflection conditions, fewer orders can be employed during sound reflection to assess different kinds of sounds in outdoor sound environments through VR experiences. The VR experience combining both HMDs and Ambisonics will significantly strengthen our immersion at low orders. For non-point source synthesis, especially line sources, when adequate synthesis points reach the threshold of the minimum audible angle, human ears cannot distinguish the location of the synthesised sound sources in the horizontal plane, thus increasing immersion significantly. These minimum specifications and simplifications refine the understanding of soundscape reproduction, and the findings will be beneficial for researchers and engineers in determining appropriate audio rendering, sound propagation modelling, and non-point source synthesis strategies