Bimodal perception of audio-visual material properties for virtual environments

INRIA Research Report 6687High-quality rendering of both audio and visual material properties is very important in interactive virtual environments, since convincingly rendered materials increase realism and the sense of immersion. We studied how the level of detail of auditory and visual stimuli interact in the perception of audio-visual material rendering quality. Our study is based on perception of material discrimination, when varying the levels of detail of modal synthesis for sound, and Bidirectional Reflectance Distribution Functions for graphics. We performed an experiment for two different models (a Dragon and a Bunny model) and two material types (Plastic and Gold). The results show a significant interaction between auditory and visual level of detail in the perception of material similarity, when comparing approximate levels of detail to a high-quality audio-visual reference rendering. We show how this result can contribute to significant savings in computation time in an interactive audio-visual rendering system. To our knowledge this is the first study which shows interaction of audio and graphics representation in a material perception task

Bimodal perception of audio-visual material properties for virtual environments

International audienceHigh-quality rendering of both audio and visual material properties is very important in interac- tive virtual environments, since convicingly rendered materials increase realism and the sense of immersion. We studied how the level of detail of auditory and visual stimuli interact in the per- ception of audio-visual material rendering quality. Our study is based on perception of material discrimination, when varying the levels of detail of modal synthesis for sound, and Bidirectional Re ectance Distribution Functions for graphics. We performed an experiment for two di erent models (a Dragon and a Bunny model) and two material types (Plastic and Gold). The results show a signi cant interaction between auditory and visual level of detail in the perception of material similarity, when comparing approximate levels of detail to a high-quality audio-visual reference rendering. We show how this result can contribute to signi cant savings in computation time in an interactive audio-visual rendering system. To our knowledge this is the rst study which shows interaction of audio and graphics representation in a material perception task

A psychology literature study on modality related issues for multimodal presentation in crisis management

The motivation of this psychology literature study is to obtain modality related guidelines for real-time information presentation in crisis management environment. The crisis management task is usually companied by time urgency, risk, uncertainty, and high information density. Decision makers (crisis managers) might undergo cognitive overload and tend to show biases in their performances. Therefore, the on-going crisis event needs to be presented in a manner that enhances perception, assists diagnosis, and prevents cognitive overload. To this end, this study looked into the modality effects on perception, cognitive load, working memory, learning, and attention. Selected topics include working memory, dual-coding theory, cognitive load theory, multimedia learning, and attention. The findings are several modality usage guidelines which may lead to more efficient use of the user’s cognitive capacity and enhance the information perception

Embodied Cognition and Perception: Dewey, Science and Skepticism

This article examines how Modern theories of mind remain even in some materialistic and hence ontologically anti-dualistic views; and shows how Dewey, anticipating Merleau-Ponty and 4E cognitive scientists, repudiates these theories. Throughout I place Dewey’s thought in the context of scientific inquiry, both recent and historical and including the cognitive as well as traditional sciences; and I show how he incorporated sciences of his day into his thought, while also anticipating enactive cognitive science. While emphasizing Dewey’s continued relevance, my main goal is to show how his scientifically informed account of perception and cognition combats skepticism propagated by certain scientific visions, exacerbated by commonplace notions about mind, that jointly suggest that human beings lack genuine access to reality

Interoceptive Ingredients of Body Ownership: Affective Touch and Cardiac Awareness in the Rubber Hand Illusion

This document is the Accepted Manuscript version of the following article: Laura Crucianelli, Charlotte Krahe, Paul M. Jenkinson, Aikaterini (Katerina) Fotopoulou, 'Interoceptive Ingredients of Body Ownership: Affective Touch and Cardiac Awareness in the Rubber Hand Illusion', Cortex, first published online 1 May 2017, available at doi: https://doi.org/10.1016/j.cortex.2017.04.018. © 2017 Elsevier Ltd. All rights reserved.The sense of body ownership represents a fundamental aspect of bodily self-consciousness. Using multisensory integration paradigms, recent studies have shown that both exteroceptive and interoceptive information contribute to our sense of body ownership. Interoception refers to the physiological sense of the condition of the body, including afferent signals that originate inside the body and outside the body. However, it remains unclear whether individual sensitivity to interoceptive modalities is unitary or differs between modalities. It is also unclear whether the effect of interoceptive information on body ownership is caused by exteroceptive ‘visual capture’ of these modalities, or by bottom-up processing of interoceptive information. This study aimed to test these questions in two separate samples. In the first experiment (N = 76), we examined the relationship between two different interoceptive modalities, namely cardiac awareness based on a heartbeat counting task, and affective touch perception based on stimulation of a specialized C tactile (CT) afferent system. This is an interoceptive modality of affective and social significance. In a second experiment (N = 63), we explored whether ‘off-line’ trait interoceptive sensitivity based on a heartbeat counting task would modulate the extent to which CT affective touch influences the multisensory process during the rubber hand illusion (RHI). We found that affective touch enhanced the subjective experience of body ownership during the RHI. Nevertheless, interoceptive sensitivity, as measured by a heartbeat counting task, did not modulate this effect, nor did it relate to the perception of ownership or of CT-optimal affective touch more generally. By contrast, this trait measure of interoceptive sensitivity appeared most relevant when the multisensory context of interoception was ambiguous, suggesting that the perception of interoceptive signals and their effects on body ownership may depend on individual abilities to regulate the balance of interoception and exteroception in given contexts.Peer reviewedFinal Accepted Versio

The Recurrent Model of Bodily Spatial Phenomenology

In this paper, we introduce and defend the recurrent model for understanding bodily spatial phenomenology. While Longo, Azañón and Haggard (2010) propose a bottom-up model, Bermúdez (2017) emphasizes the top-down aspect of the information processing loop. We argue that both are only half of the story. Section 1 intro- duces what the issues are. Section 2 starts by explaining why the top- down, descending direction is necessary with the illustration from the ‘body-based tactile rescaling’ paradigm (de Vignemont, Ehrsson and Haggard, 2005). It then argues that the bottom-up, ascending direction is also necessary, and substantiates this view with recent research on skin space and tactile field (Haggard et al., 2017). Section 3 discusses the model’s application to body ownership and bodily self-representation. Implications also extend to topics such as sense modality individuation (Macpherson, 2011), the constancy- based view of perception (Burge, 2010), and the perception/cognition divide (Firestone and Scholl, 2016)

The Sound Manifesto

Computing practice today depends on visual output to drive almost all user interaction. Other senses, such as audition, may be totally neglected, or used tangentially, or used in highly restricted specialized ways. We have excellent audio rendering through D-A conversion, but we lack rich general facilities for modeling and manipulating sound comparable in quality and flexibility to graphics. We need co-ordinated research in several disciplines to improve the use of sound as an interactive information channel. Incremental and separate improvements in synthesis, analysis, speech processing, audiology, acoustics, music, etc. will not alone produce the radical progress that we seek in sonic practice. We also need to create a new central topic of study in digital audio research. The new topic will assimilate the contributions of different disciplines on a common foundation. The key central concept that we lack is sound as a general-purpose information channel. We must investigate the structure of this information channel, which is driven by the co-operative development of auditory perception and physical sound production. Particular audible encodings, such as speech and music, illuminate sonic information by example, but they are no more sufficient for a characterization than typography is sufficient for a characterization of visual information.Comment: To appear in the conference on Critical Technologies for the Future of Computing, part of SPIE's International Symposium on Optical Science and Technology, 30 July to 4 August 2000, San Diego, C

A Unified Quantitative Model of Vision and Audition

We have put forwards a unified quantitative framework of vision and audition, based on existing data and theories. According to this model, the retina is a feedforward network self-adaptive to inputs in a specific period. After fully grown, cells become specialized detectors based on statistics of stimulus history. This model has provided explanations for perception mechanisms of colour, shape, depth and motion. Moreover, based on this ground we have put forwards a bold conjecture that single ear can detect sound direction. This is complementary to existing theories and has provided better explanations for sound localization.Comment: 7 pages, 3 figure

Impact of model fidelity in factory layout assessment using immersive discrete event simulation

Discrete Event Simulation (DES) can help speed up the layout design process. It offers further benefits when combined with Virtual Reality (VR). The latest technology, Immersive Virtual Reality (IVR), immerses users in virtual prototypes of their manufacturing plants to-be, potentially helping decision-making. This work seeks to evaluate the impact of visual fidelity, which refers to the degree to which objects in VR conforms to the real world, using an IVR visualisation of the DES model of an actual shop floor. User studies are performed using scenarios populated with low- and high-fidelity models. Study participant carried out four tasks representative of layout decision-making. Limitations of existing IVR technology was found to cause motion sickness. The results indicate with the particular group of naïve modellers used that there is no significant difference in benefits between low and high fidelity, suggesting that low fidelity VR models may be more cost-effective for this group