How Do We Experience Crossmodal Correspondent Mulsemedia Content?

Sensory studies emerged as a significant influence upon Human Computer Interaction and traditional multimedia. Mulsemedia is an area that extends multimedia addressing issues of multisensorial response through the combination of at least three media, typically a non-traditional media with traditional audio-visual content. In this paper, we explore the concepts of Quality of Experience and crossmodal correspondences through a case study of different types of mulsemedia setups. The content is designed following principles of crossmodal correspondence between different sensory dimensions and delivered through olfactory, auditory and vibrotactile displays. The Quality of Experience is evaluated through both subjective (questionnaire) and objective means (eye gaze and heart rate). Results show that the auditory experience has an influence on the olfactory sensorial responses and lessens the perception of lingering odor. Heat maps of the eye gazes suggest that the crossmodality between olfactory and visual content leads to an increased visual attention on the factors of the employed crossmodal correspondence (e.g., color, brightness, shape)

On the influence of individual differences in cross-modal Mulsemedia QoE

Quality of Experience (QoE) is inextricably linked to the human side of the multimedia experience. Whilst there has been a considerable amount of research undertaken to explore the various dimensions of QoE, one facet which been relatively unexplored is the role of individual differences in determining an individual’s QoE. Whereas this is certainly true of multimedia applications, when it comes to mulsemedia (multiple media engaging three or more human senses) this is even more so, given its emerging and novel nature. Accordingly, in this paper we report the results of a study which investigated the role that individual differences (such as age, gender, education, and smell sensitivity) have on QoE, when mulsemedia incorporating olfactory and haptic stimuli is experienced in cross-modal environments. Our results reveal that whilst users had a satisfying overall mulsemedia experience the specific use of cross modally matched odours did not result in significantly higher QoE levels than when a control scent (rosemary) was employed. However, aspects of QoE are impacted upon by all individual differences dimensions considered in our study

A QoE Model for Mulsemedia TV in a Smart Home Environment

The provision to the users of realistic media contents is one of the main goals of future media services. The sense of reality perceived by the user can be enhanced by adding various sensorial effects to the conventional audio-visual content, through the stimulation of the five senses stimulation (sight, hearing, touch, smell and taste), the so-called multi-sensorial media (mulsemedia). To deliver the additional effects within a smart home (SH) environment, custom devices (e.g., air conditioning, lights) providing opportune smart features, are preferred to ad-hoc devices, often deployed in a specific context such as for example in gaming consoles. In the present study, a prototype for a mulsemedia TV application, implemented in a real smart home scenario, allowed the authors to assess the user's Quality of Experience (QoE) through test measurement campaign. The impact of specific sensory effects (i.e., light, airflow, vibration) on the user experience regarding the enhancement of sense of reality, annoyance, and intensity of the effects was investigated through subjective assessment. The need for multi sensorial QoE models is an important challenge for future research in this field, considering the time and cost of subjective quality assessments. Therefore, based on the subjective assessment results, this paper instantiates and validates a parametric QoE model for multi-sensorial TV in a SH scenario which indicates the relationship between the quality of audiovisual contents and user-perceived QoE for sensory effects applications

Integration of multi-sensorial effects in synchronised immersive hybrid TV scenarios

[EN] Traditionally, TV media content has exclusively involved 2D or 3D audiovisual streams consumed by using a simple TV device. However, in order to generate more immersive media consumption experiences, other new types of content (e.g., omnidirectional video), consumption devices (e.g., Head Mounted Displays or HMD) and solutions to stimulate other senses beyond the traditional ones of sight and hearing, can be used. Multi-sensorial media content (a.k.a. mulsemedia) facilitates additional sensory effects that stimulate other senses during the media consumption, with the aim of providing the consumers with a more immersive and realistic experience. They provide the users with a greater degree of realism and immersion, but can also provide greater social integration (e.g., people with AV deficiencies or attention span problems) and even contribute to creating better educational programs (e.g., for learning through the senses in educational content or scientific divulgation). Examples of sensory effects that can be used are olfactory effects (scents), tactile effects (e.g., vibration, wind or pressure effects), and ambient effects (e.g., temperature or lighting). In this paper, a solution for providing multi-sensorial and immersive hybrid (broadcast/broadband) TV content consumption experiences, including omnidirectional video and sensory effects, is presented. It has been designed, implemented, and subjectively evaluated (by 32 participants) in an end-to-end platform for hybrid content generation, delivery and synchronised consumption. The satisfactory results which were obtained regarding the perception of fine synchronisation between sensory effects and multimedia content, and regarding the users' perceived QoE, are summarised and discussed.This work was supported in part by the "Vicerrectorado de Investigacion de la Universitat Politecnica de Valencia'' under Project PAID-11-21 and Project PAID-12-21.Marfil, D.; Boronat, F.; González-Salinas, J.; Sapena Piera, A. (2022). Integration of multi-sensorial effects in synchronised immersive hybrid TV scenarios. IEEE Access. 10:79071-79089. https://doi.org/10.1109/ACCESS.2022.319417079071790891

Virtual Synaesthesia: Crossmodal Correspondences and Synesthetic Experiences

As technology develops to allow for the integration of additional senses into interactive experiences, there is a need to bridge the divide between the real and the virtual in a manner that stimulates the five senses consistently and in harmony with the sensory expectations of the user. Applying the philosophy of a neurological condition known as synaesthesia and crossmodal correspondences, defined as the coupling of the senses, can provide numerous cognitive benefits and offers an insight into which senses are most likely to be ‘bound’ together. This thesis aims to present a design paradigm called ‘virtual synaesthesia’ the goal of the paradigm is to make multisensory experiences more human-orientated by considering how the brain combines senses in both the general population (crossmodal correspondences) and within a select few individuals (natural synaesthesia). Towards this aim, a literature review is conducted covering the related areas of research umbrellaed by the concept of ‘virtual synaesthesia’. Its research areas are natural synaesthesia, crossmodal correspondences, multisensory experiences, and sensory substitution/augmentation. This thesis examines augmenting interactive and multisensory experiences with strong (natural synaesthesia) and weak (crossmodal correspondences) synaesthesia. This thesis answers the following research questions: Is it possible to replicate the underlying cognitive benefits of odour-vision synaesthesia? Do people have consistent correspondences between olfaction and an aggregate of different sensory modalities? What is the nature and origin of these correspondences? And Is it possible to predict the crossmodal correspondences attributed to odours? The benefits of augmenting a human-machine interface using an artificial form of odour-vision synaesthesia are explored to answer these questions. This concept is exemplified by transforming odours transduced using a custom-made electronic nose and transforming an odour's ‘chemical footprint’ into a 2D abstract shape representing the current odour. Electronic noses can transform odours in the vapour phase generating a series of electrical signals that represent the current odour source. Weak synaesthesia (crossmodal correspondences) is then investigated to determine if people have consistent correspondences between odours and the angularity of shapes, the smoothness of texture, perceived pleasantness, pitch, musical, and emotional dimensions. Following on from this research, the nature and origin of these correspondences were explored using the underlying hedonic (values relating to pleasantness), semantic (knowledge of the identity of the odour) and physicochemical (the physical and chemical characteristics of the odour) dependencies. The final research chapter investigates the possibility of removing the bottleneck of conducting extensive human trials by determining what the crossmodal correspondences towards specific odours are by developing machine learning models to predict the crossmodal perception of odours using their underlying physicochemical features. The work presented in this thesis provides some insight and evidence of the benefits of incorporating the concept ‘virtual synaesthesia’ into human-machine interfaces and research into the methodology embodied by ‘virtual synaesthesia’, namely crossmodal correspondences. Overall, the work presented in this thesis shows potential for augmenting multisensory experiences with more refined capabilities leading to more enriched experiences, better designs, and a more intuitive way to convey information crossmodally