Real-Time Viewport-Aware Optical Flow Estimation in 360-degree Videos for Visually-Induced Motion Sickness Mitigation

Visually-induced motion sickness (VIMS), a side effect of perceived motion caused by visual stimulation, is a major obstacle to the widespread use of Virtual Reality (VR). Along with scene object information, visual stimulation can be primarily indicated by optical flow, which characterizes the motion pattern, such as the intensity and direction of the moving image. We estimated the real time optical flow in 360-degree videos targeted at immersive user interactive visualization based on the user's current viewport. The proposed method allows the estimation of customized visual flow for each experience of dynamic 360-degree videos and is an improvement over previous methods that consider a single optical flow value for the entire equirectangular frame. We applied our method to modulate the opacity of granulated rest frames (GRFs), a technique consisting of visual noise-like randomly distributed visual references that are stable to the user's body during immersive pre-recorded 360-degree video experience. We report the results of a pilot one-session between-subject study with 18 participants, where users watched a 2-minute high-intensity 360-degree video. The results show that our proposed method successfully estimates optical flow, with pilot data showing that GRFs combined with real-time optical flow estimation may improve user comfort when watching 360-degree videos. However, more data are needed for statistically significant results

Real vs Simulated Foveated Rendering to Reduce Visual Discomfort in Virtual Reality

In this paper, a study aimed at investigating the effects of real (using eye tracking to determine the fixation) and simulated foveated blurring in immersive Virtual Reality is presented. Techniques to reduce the optical flow perceived at the visual field margins are often employed in immersive Virtual Reality environments to alleviate discomfort experienced when the visual motion perception does not correspond to the body's acceleration. Although still preliminary, our results suggest that for participants with higher self-declared sensitivity to sickness, there might be an improvement for nausea when using blurring. The (perceived) difficulty of the task seems to improve when the real foveated method is used.Comment: 9 pages, 2 figures, 1 table, to be published in proceedings of the 18th International Conference promoted by the IFIP Technical Committee 13 on Human Computer Interaction, INTERACT 2021. August 30th September 3rd, 2021, Bari, Ital

Physiological Targets of Artificial Gravity: The Sensory-Motor System

This chapter describes the pros and cons of artificial gravity applications in relation to human sensory-motor functioning in space. Spaceflight creates a challenge for sensory-motor functions that depend on gravity, which include postural balance, locomotion, eye-hand coordination, and spatial orientation. The sensory systems, and in particular the vestibular system, must adapt to weightlessness on entering orbit, and again to normal gravity upon return to Earth. During this period of adaptation, which persists beyond the actual gravity-level transition itself the sensory-motor systems are disturbed. Although artificial gravity may prove to be beneficial for the musculoskeletal and cardiovascular systems, it may well have negative side effects for the neurovestibular system, such as spatial disorientation, malcoordination, and nausea

Методи пом'якшення

The section describes mitigation methods.У розділі описано методи помякшення

The effects of expectancy and control on the perception of ego-motion in space: a combined postural and electrophysiological study

In the beginning of this work was the scientific question: does the amount of control over visual self-motion cues influence their processing and/or perception? In Experiment 1, we tried to explore the possibility to use optic flow as a visual motion cue and see whether we can observe a sensory attenuation or modulation on the behavioural level in trials in which the optic flow was self-initiated using putative different levels of control by instructed or uninstructed button-presses compared to passive flow. This experiment, while not able to demonstrate a sensory modulation and with several important limitations (see below), was however an important basis for the planning of Experiment 2 and a proof-of-concept that this method has the potential to address our research question and is feasible given our facilities. In Experiment 2, we tried to overcome some of the limitations, further improved the re-producibility (e.g. stimuli and instructions) and extended our methodology to the measurement of neurophysiological and postural data to enquire about not only the behavioural level but also the processing on the physiological level. This experiment presented evidence that self-motion cues with the same physical properties are somehow processed differently at the cortical level depending on whether they are self-initiated or not. In addition to overcoming certain limitations in Experiment 3 (e.g. having a no optic flow control condition and using the standard EEG setup besides the mobile setup from Experiment 2), we were able to reproduce our findings in different subjects, a larger population and under a different posture. We were also able to show that our results are highly robust (e.g. removal of half the participants from the analysis did not change the pattern). Further outcomes from our study are that the scientific community can put more trust into mobile EEG setups given robust effects and diligent artifact removal. Additionally, we contributed findings on the relationship of vection and VIMS and tried to bridge the gap between the highly relevant fields of research on visual motion perception and sense of agency. This might act as an exploratory foundation for further research which will be essential for the economical and medical applicability of VR devices and for a deeper understanding of locomotion and navigation per se. The ability to perceive self-motion cues and dissociate them from cues for motion in the environment is fundamental for being able to take actions in the complex, dynamic environments which are our daily lives. In fact, it could be seen as a classical example of the dynamic coupling of action and perception to reach goals which is one of the most fundamental abilities not only for humans, but throughout the animal kingdom which may have lain the evolutionary basis for the later development of the human brain with its complexity as we see it nowadays (Godfrey-Smith 2016).Den Grundstein für diese Arbeit legte die Frage: spielt es für die Wahrnehmung und Verarbeitung von visuellem Feedback, das in Folge von Eigenbewegung im Raum entsteht, eine Rolle wie viel Kontrolle wir über die Bewegung haben? Wird das Feedback von aktiven Bewegungen anders verarbeitet als das von passiven? Im ersten Experiment explorierten wir die Möglichkeit uns dieser Fragestellung mit optic flow als visuellem Stimulus zu nähern. Wir haben dazu ein Experiment entwickelt bei dem gesunde Proband:innen unterschiedlich viel Kontrolle über den optic flow haben und sie anschließend zu ihrem Bewegungsempfinden (Vection) befragt. Während dieses Experiment keine relevante Modulation nachweisen konnte, so stellte es doch eine wichtige methodologische Grundlage für die Entwicklung der weiteren Experimente dar. Die wichtigsten Änderungen in Experiment 2 umfassten zum einen Modifikationen an den Stimuli und eine ausgeprägtere Formalisierung der Instruktionen, zum anderen die zusätzliche Erhebung von neurophysiologischen und posturalen Daten. Diese Änderungen erlaubten uns nicht nur explizite Unterschiede in der Intensität der Wahrnehmung von Vection zu erfassen, sondern auch eventuelle Modifikationen in der Verarbeitung der Stimuli messbar zu machen. Dieses Experiment lieferte Hinweise darauf, dass Stimuli mit denselben physikalischen Eigenschaften auf kortikaler Ebene anders verarbeitet werden, je nachdem ob sie selbst initiiert oder Computer-generiert sind. In Experiment 3 führten wir klassische Kontrollbedingungen wie zum Beispiel Versuche mit statischen Stimuli ein. Wir veränderten weiterhin die Körperposition, so dass Proband:innen nun saßen und die Hälfte der Versuche mit einer Kinnstütze stattfand. Damit konnten wir das Risiko, das unsere neurophysiologischen Effekte Bewegungsartefakte sind, minimieren. Insgesamt waren wir dazu in der Lage die Haupteffekte von Experiment 2 (agency-abhängige Modulation der evozierten Desynchronisation und der Amplitude der evozierten Potentiale) in Experiment 3 zu reproduzieren, obwohl wir hier eine deutlich größere Kohorte sowie andere Pro-band:innen in einer anderen Körperhaltung testeten. Diese Resultate sind sehr robust, so dass sie weiterhin deutlich erkennbar sind, auch nachdem wir ver-suchsweise die Hälfte der Proband:innen aus der Analyse ausgeschlossen hatten. Zusätzlich zu unserer ursprünglichen Fragestellung zeigten unsere Experimente, dass die wissenschaftliche Community mehr auf die Ergebnisse von Studien, die ein mobiles EEG-Setup verwenden, vertrauen kann, solange es sich um robuste Effekte handelt und ausreichend auf die Identifikation und Entfernung von Bewegungsartefakten geachtet wird. Außerdem konnten wir mit unseren Daten dazu beitragen die Zusammenhänge zwischen Vection und visuell-induzierter Bewegungskrankheit besser zu verstehen. Unsere Experimente versuchen die Brücke zu schlagen zwischen den jeweils für sich gesehen hoch relevanten Forschungsfeldern rund um die visuelle Bewegung-swahrnehmung und den Sense of Agency. Diese Felder zusammenzubringen wird eine essenzielle Rolle spielen, sowohl um das volle Potential von VR-Applikationen zu entfalten als auch um Lokomotion und Navigation umfassender zu begreifen. Die Fähigkeit Eigenbewegung von Bewegungen in der Umgebung anhand von visuellen Informationen zu unterscheiden, ist entscheidend um in der komplexen, dynamischen Umwelt unseres täglichen Lebens erfolgreich agieren und navigieren zu können. Diese Fähigkeit ist ein schönes Beispiel für die dynamische Koppelung von Handlung und Wahrnehmung zum Erreichen unserer Ziele und vermutlich eine der fundamentalsten Fähigkeiten nicht nur für Menschen, sondern auch im übrigen Tierreich. Möglicherweise so fundamental, dass sie die evolutionäre Basis für die spätere Entwicklung des menschlichen Gehirns in all seiner Komplexität und Schönheit, gelegt haben könnte (Godfrey-Smith 2016)

Exploring Adaptation-Based Techniques to Create Comfortable Virtual Reality Experiences

Virtual reality (VR) is transitioning from research to widespread consumer use. However, VR sickness - a type of motion sickness associated with VR usage - is believed to be a major impediment to the mass adoption of VR and it is estimatedto affect more than two-thirds of VR users. Previous research also shows that VR sickness affects some vulnerable groups, such as women, more severely than other groups. Although several strategies have been developed to mitigate VR sickness, most of them are not equally effective for all users since the effectiveness of any particular strategy varies across individuals. There are also concerns that some widely used VR sickness mitigation strategies, such as field-of-view (FOV) restriction, may have negative consequences on women. This thesis aims: 1) to provide theoretical understanding of the aspects of VR systems that cause VR sickness to affect some user more than others, with focus on sex differences, and 2) to develop adaptation-based strategies that could mitigate VR sickness for all VR users irrespective of their differences.Towards these goals, I first investigate the effectiveness of FOV restriction in reducing VR sickness across genders, and it’s effects on women’s spatial navigation ability. Then, based on findings from the first set of studies, I develop andempirically evaluate a novel adaptive eye gaze-contingent FOV restrictor that allows users to have a wider visual field while blocking their peripheral FOV. The wider visual field would be beneficial for women’s spatial navigation performance in virtual environments. Finally, I provide a novel standardized adaptation-based training paradigm that supplements existing VR sickness mitigation techniques by allowing the user to best prepare themselves for continued VR use. Evaluation of this strategy suggests that it could reduce and even eliminate VR sickness in susceptible individuals irrespective of their individual differences

Leveraging eXtented Reality & Human-Computer Interaction for User Experi- ence in 360◦ Video

EXtended Reality systems have resurged as a medium for work and entertainment. While 360o video has been characterized as less immersive than computer-generated VR, its realism, ease of use and affordability mean it is in widespread commercial use. Based on the prevalence and potential of the 360o video format, this research is focused on improving and augmenting the user experience of watching 360o video. By leveraging knowledge from Extented Reality (XR) systems and Human-Computer Interaction (HCI), this research addresses two issues affecting user experience in 360o video: Attention Guidance and Visually Induced Motion Sickness (VIMS). This research work relies on the construction of multiple artifacts to answer the de- fined research questions: (1) IVRUX, a tool for analysis of immersive VR narrative expe- riences; (2) Cue Control, a tool for creation of spatial audio soundtracks for 360o video, as well as enabling the collection and analysis of captured metrics emerging from the user experience; and (3) VIMS mitigation pipeline, a linear sequence of modules (including optical flow and visual SLAM among others) that control parameters for visual modi- fications such as a restricted Field of View (FoV). These artifacts are accompanied by evaluation studies targeting the defined research questions. Through Cue Control, this research shows that non-diegetic music can be spatialized to act as orientation for users. A partial spatialization of music was deemed ineffective when used for orientation. Addi- tionally, our results also demonstrate that diegetic sounds are used for notification rather than orientation. Through VIMS mitigation pipeline, this research shows that dynamic restricted FoV is statistically significant in mitigating VIMS, while mantaining desired levels of Presence. Both Cue Control and the VIMS mitigation pipeline emerged from a Research through Design (RtD) approach, where the IVRUX artifact is the product of de- sign knowledge and gave direction to research. The research presented in this thesis is of interest to practitioners and researchers working on 360o video and helps delineate future directions in making 360o video a rich design space for interaction and narrative.Sistemas de Realidade EXtendida ressurgiram como um meio de comunicação para o tra- balho e entretenimento. Enquanto que o vídeo 360o tem sido caracterizado como sendo menos imersivo que a Realidade Virtual gerada por computador, o seu realismo, facili- dade de uso e acessibilidade significa que tem uso comercial generalizado. Baseado na prevalência e potencial do formato de vídeo 360o, esta pesquisa está focada em melhorar e aumentar a experiência de utilizador ao ver vídeos 360o. Impulsionado por conhecimento de sistemas de Realidade eXtendida (XR) e Interacção Humano-Computador (HCI), esta pesquisa aborda dois problemas que afetam a experiência de utilizador em vídeo 360o: Orientação de Atenção e Enjoo de Movimento Induzido Visualmente (VIMS). Este trabalho de pesquisa é apoiado na construção de múltiplos artefactos para res- ponder as perguntas de pesquisa definidas: (1) IVRUX, uma ferramenta para análise de experiências narrativas imersivas em VR; (2) Cue Control, uma ferramenta para a criação de bandas sonoras de áudio espacial, enquanto permite a recolha e análise de métricas capturadas emergentes da experiencia de utilizador; e (3) canal para a mitigação de VIMS, uma sequência linear de módulos (incluindo fluxo ótico e SLAM visual entre outros) que controla parâmetros para modificações visuais como o campo de visão restringido. Estes artefactos estão acompanhados por estudos de avaliação direcionados para às perguntas de pesquisa definidas. Através do Cue Control, esta pesquisa mostra que música não- diegética pode ser espacializada para servir como orientação para os utilizadores. Uma espacialização parcial da música foi considerada ineficaz quando usada para a orientação. Adicionalmente, os nossos resultados demonstram que sons diegéticos são usados para notificação em vez de orientação. Através do canal para a mitigação de VIMS, esta pesquisa mostra que o campo de visão restrito e dinâmico é estatisticamente significante ao mitigar VIMS, enquanto mantem níveis desejados de Presença. Ambos Cue Control e o canal para a mitigação de VIMS emergiram de uma abordagem de Pesquisa através do Design (RtD), onde o artefacto IVRUX é o produto de conhecimento de design e deu direcção à pesquisa. A pesquisa apresentada nesta tese é de interesse para profissionais e investigadores tra- balhando em vídeo 360o e ajuda a delinear futuras direções em tornar o vídeo 360o um espaço de design rico para a interação e narrativa

The Effects of Primary and Secondary Task Workloads on Cybersickness in Immersive Virtual Active Exploration Experiences

Virtual reality (VR) technology promises to transform humanity. The technology enables users to explore and interact with computer-generated environments that can be simulated to approximate or deviate from reality. This creates an endless number of ways to propitiously apply the technology in our lives. It follows that large technological conglomerates are pushing for the widespread adoption of VR, financing the creation of the Metaverse - a hypothetical representation of the next iteration of the internet. Even with VR technology\u27s continuous growth, its widespread adoption remains long overdue. This can largely be attributed to an affliction called cybersickness, an analog to motion sickness, which often manifests in users as an undesirable side-effect of VR experiences, inhibiting its sustained usage. This makes it highly important to study factors related to the malady. The tasks performed in a simulated environment provide context, purpose, and meaning to the experience. Active exploration experiences afford users control over their motion, primarily allowing them to navigate through an environment. While navigating, users may also have to engage in secondary tasks that can be distracting. These navigation and distraction tasks differ in terms of the source and magnitude of attentional demands involved, potentially influencing how cyber-sickening a simulation can be. Given the sparse literature in this area, this dissertation sets out to investigate how the interplay between these factors impacts the onset and severity of sickness, thereby contributing to the knowledge base on how the attentional demands associated with the tasks performed during navigation affect cybersickness in virtual reality