Multi-sensory media experiences

The way we experience the world is based on our five senses, which allow us unique and often surprising sensations of our environment. Interactive technologies are mainly stimulating our senses of vision and hearing, partly our sense of touch, and the sense of taste and smell are widely under-exploited. There is however a growing international interest of the film, video, and game industries in more immersive viewing and gaming experiences. In the 20th century there was a demand for a controllable way to describe colours that initiated intense research on the descriptions of colours and substantially contributed to advances in computer graphics, image processing, photography and cinematography. Similarly, the 21st century now demands an investigation of touch, taste, and smell as sensory interaction modalities to enhance media experiences

MediaSync: Handbook on Multimedia Synchronization

This book provides an approachable overview of the most recent advances in the fascinating field of media synchronization (mediasync), gathering contributions from the most representative and influential experts. Understanding the challenges of this field in the current multi-sensory, multi-device, and multi-protocol world is not an easy task. The book revisits the foundations of mediasync, including theoretical frameworks and models, highlights ongoing research efforts, like hybrid broadband broadcast (HBB) delivery and users' perception modeling (i.e., Quality of Experience or QoE), and paves the way for the future (e.g., towards the deployment of multi-sensory and ultra-realistic experiences). Although many advances around mediasync have been devised and deployed, this area of research is getting renewed attention to overcome remaining challenges in the next-generation (heterogeneous and ubiquitous) media ecosystem. Given the significant advances in this research area, its current relevance and the multiple disciplines it involves, the availability of a reference book on mediasync becomes necessary. This book fills the gap in this context. In particular, it addresses key aspects and reviews the most relevant contributions within the mediasync research space, from different perspectives. Mediasync: Handbook on Multimedia Synchronization is the perfect companion for scholars and practitioners that want to acquire strong knowledge about this research area, and also approach the challenges behind ensuring the best mediated experiences, by providing the adequate synchronization between the media elements that constitute these experiences

Deep Reminding

Deep Reminding is an immersive multi-sensory experience comprised of sound, light and digital painting. My interest in the existential and mystical qualities of sound and light, as well as their power to engage, alter, and expand the consciousness of the audience, forms the basis of this body of work. These multi-sensory experiences open us to feelings and information that are not accessible through language and other media; they create an internal dialogue between the mysterious nature of the exhibition space and our perception of being. Conscious feelings and thoughts are suspended, hypnotized through listening and seeing

Integration of multi-sensorial effects in synchronised immersive hybrid TV scenarios

[EN] Traditionally, TV media content has exclusively involved 2D or 3D audiovisual streams consumed by using a simple TV device. However, in order to generate more immersive media consumption experiences, other new types of content (e.g., omnidirectional video), consumption devices (e.g., Head Mounted Displays or HMD) and solutions to stimulate other senses beyond the traditional ones of sight and hearing, can be used. Multi-sensorial media content (a.k.a. mulsemedia) facilitates additional sensory effects that stimulate other senses during the media consumption, with the aim of providing the consumers with a more immersive and realistic experience. They provide the users with a greater degree of realism and immersion, but can also provide greater social integration (e.g., people with AV deficiencies or attention span problems) and even contribute to creating better educational programs (e.g., for learning through the senses in educational content or scientific divulgation). Examples of sensory effects that can be used are olfactory effects (scents), tactile effects (e.g., vibration, wind or pressure effects), and ambient effects (e.g., temperature or lighting). In this paper, a solution for providing multi-sensorial and immersive hybrid (broadcast/broadband) TV content consumption experiences, including omnidirectional video and sensory effects, is presented. It has been designed, implemented, and subjectively evaluated (by 32 participants) in an end-to-end platform for hybrid content generation, delivery and synchronised consumption. The satisfactory results which were obtained regarding the perception of fine synchronisation between sensory effects and multimedia content, and regarding the users' perceived QoE, are summarised and discussed.This work was supported in part by the "Vicerrectorado de Investigacion de la Universitat Politecnica de Valencia'' under Project PAID-11-21 and Project PAID-12-21.Marfil, D.; Boronat, F.; González-Salinas, J.; Sapena Piera, A. (2022). Integration of multi-sensorial effects in synchronised immersive hybrid TV scenarios. IEEE Access. 10:79071-79089. https://doi.org/10.1109/ACCESS.2022.319417079071790891

The Shape of Poetry: A Typographic Exploration of Poetry and Synesthesia

The Shape of Poetry is a thesis which explores the relationships of synesthesia, typography and poetry. Synesthesia is a neurologically-based condition in which stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway. This phenomenon is common in poetry as a rhetorical device. In graphic design, we uses typography and other visual elements to make cross-sensory metaphors as well. The similarity among these three provides a new field of research possibilities. The goal of the thesis is to visualize poetry by exploring typographic variations and implementing synesthesia findings. The final result is a campaign to promote William Shakespeare’s sonnets, in both print and digital applications. Overall the findings of this thesis aim at giving designers another angle when designing for multi-media platforms, and inspire those who have the similar interests in the relationship of sensory perception and design

What can be learned about the relationship with food in food blogs?:A netnographic analysis of Finnish food bloggers’ posts on social media

Social media provide numerous platforms to build relationships with food. In this research, we focus on Finnish food bloggers with the purpose to analyze how the relationship with food appears in food blogs. To answer this question, we conducted a netnographic research by visiting food blogs. The analysis followed qualitative content analysis. The main categories of findings were food- and eating-related values, innovativeness and experimental approach, positive eating-related multi-sensory experiences and memories, and social elements of eating. The role of social media, for example in the form of food blogs, can have a meaningful role in the socialization into the modern eating culture

Experiential media and transforming storytelling: A theoretical analysis

Journalism and media content rests on a foundation of storytelling. Shaping this storytelling is the quality of the medium of content delivery and the nature of public engagement. With the development of digital, networked media, the audience’s role is transforming to be more of an active user who experiences stories as a participant rather than as a passive receiver of content. This article proposes a new model of experiential media based on six primary qualities of the digital environment. These qualities are 1) interactivity, 2) immersion, 3) multi-sensory presentation, 4) algorithmic and data-driven, 5) first-person perspective, and 6) a natural user interface. Augmented reality and virtual reality are among the most-widely discussed experiential media forms, but others include, for instance, advanced ultra-high-definition video. Experiential media bring implications for the nature, production, impact and the future of mediated storytelling

On the influence of individual characteristics and personality traits on the user experience with multi-sensorial media: an experimental insight

Recent studies encourage the development of sensorially-enriched media to enhance the user experience by stimulating senses other than sight and hearing. Sensory effects as odor, wind, vibration and light effects, as well as an enhanced audio quality, have been found to favour media enjoyment and to have a positive influence on the sense of Presence and on the perceived quality, relevance and reality of a multimedia experience. In particular, sports is among the genres that could benefit the most from these solutions. Several works have demonstrated also the technical feasibility of implementing and deploying end-to-end solutions integrating sensory effects into a legacy system. Thus, multi-sensorial media emerges as a mean to deliver a new form of immersive experiences to the mass market in a non-disruptive manner. However, many questions remain concerning issues as the sensory effects that can better complement a given audiovisual content or the best way in which to integrate and combine them to enhance the user experience of a target audience segment. The work presented in this paper aims to gain insight into the impact of binaural audio and sensory (light and olfactory) effects on the sports media experience, both at the overall level (average effect) and as a function of users? characteristics (heterogeneous effects). To this aim, we conducted an experimental study exploring the influence of these immersive elements on the quality and Presence dimensions of the media experience. Along the quality dimension, we look for possible variations on the quality scores assigned to the overall media experience and to the media components content, image, audio and sensory effects. The potential impact on Presence is analyzed in terms of Spatial Presence and Engagement. The users? characteristics considered encompass specific personal affective, cognitive and behavioral attributes. We found that, on average, participants preferred binaural audio than standard stereo audio. The audio quality was found to have a heterogeneous impact on the quality of experience, on the perceived quality of content and image and on the levels of Spatial Presence and Engagement. Furthermore, the presence of sensory effects increased significantly the level of Spatial Presence. Additionally, highly conscious participants reported a significantly higher image quality when sensory effects were present in comparison to those conditions in which sensory effects were not administered. Personal characteristics explained most of the variation in the dependent variables, being individuals? preferences in relation to the content, knowledge of involved technologies, tendency to emotional involvement and conscientiousness among the user variables with the most generalized influence