Digitizing the chemical senses: possibilities & pitfalls

Many people are understandably excited by the suggestion that the chemical senses can be digitized; be it to deliver ambient fragrances (e.g., in virtual reality or health-related applications), or else to transmit flavour experiences via the internet. However, to date, progress in this area has been surprisingly slow. Furthermore, the majority of the attempts at successful commercialization have failed, often in the face of consumer ambivalence over the perceived benefits/utility. In this review, with the focus squarely on the domain of Human-Computer Interaction (HCI), we summarize the state-of-the-art in the area. We highlight the key possibilities and pitfalls as far as stimulating the so-called ‘lower’ senses of taste, smell, and the trigeminal system are concerned. Ultimately, we suggest that mixed reality solutions are currently the most plausible as far as delivering (or rather modulating) flavour experiences digitally is concerned. The key problems with digital fragrance delivery are related to attention and attribution. People often fail to detect fragrances when they are concentrating on something else; And even when they detect that their chemical senses have been stimulated, there is always a danger that they attribute their experience (e.g., pleasure) to one of the other senses – this is what we call ‘the fundamental attribution error’. We conclude with an outlook on digitizing the chemical senses and summarize a set of open-ended questions that the HCI community has to address in future explorations of smell and taste as interaction modalities

Thermal and wind devices for multisensory human-computer interaction: an overview

In order to create immersive experiences in virtual worlds, we need to explore different human senses (sight, hearing, smell, taste, and touch). Many different devices have been developed by both industry and academia towards this aim. In this paper, we focus our attention on the researched area of thermal and wind devices to deliver the sensations of heat and cold against people’s skin and their application to human-computer interaction (HCI). First, we present a review of devices and their features that were identified as relevant. Then, we highlight the users’ experience with thermal and wind devices, highlighting limitations either found or inferred by the authors and studies selected for this survey. Accordingly, from the current literature, we can infer that, in wind and temperature-based haptic systems (i) users experience wind effects produced by fans that move air molecules at room temperature, and (ii) there is no integration of thermal components to devices intended for the production of both cold or hot airflows. Subsequently, an analysis of why thermal wind devices have not been devised yet is undertaken, highlighting the challenges of creating such devices.Espírito Santo Research and Innovation Foundation (FAPES, Brazil) - Finance Code 2021-GL60J), the Coordination for the Improvement of Higher Education Personnel (CAPES, Brazil) - Finance Code 88881.187844/2018-01 and 88887.570688/2020-00 and by the National Council for Scientific and Technological (CNPq, Brazil) - Finance Code 307718/2020-4. The work was also funded by the European Union’s Horizon 2020 Research and Innovation programme under Grant Agreement no. 688503. E. B. Saleme additionally acknowledges aid from the Federal Institute of Espírito Santo

Smell, Taste, and Temperature Interfaces

Everyday life hinges on smell, taste, and temperature-based experiences, from eating to detecting potential hazards (e.g., smell of rotten food, microbial threats, and non-microbial threats such as from hazardous gases) to responding to thermal behavioral changes. These experiences are formative as visceral, vital signals of information, and contribute directly to our safety, well-being, and enjoyment. Despite this, contemporary technology mostly stimulates vision, audition, and - more recently - touch, unfortunately leaving out the senses of smell taste and temperature. In the last decade, smell, taste, and temperature interfaces have gained a renewed attention in the field of Human Computer Interaction, fueled by the growth of virtual reality and wearable devices. As these modalities are further explored, it is imperative to discuss underlying cultural contexts of these experiences, how researchers can robustly stimulate and sense these modalities, and in what contexts such multisensory technologies are meaningful. This workshop addresses these topics and seeks to provoke critical discussions around chemo- and thermo-sensory HCI

Multimodality in VR: A survey

Virtual reality (VR) is rapidly growing, with the potential to change the way we create and consume content. In VR, users integrate multimodal sensory information they receive, to create a unified perception of the virtual world. In this survey, we review the body of work addressing multimodality in VR, and its role and benefits in user experience, together with different applications that leverage multimodality in many disciplines. These works thus encompass several fields of research, and demonstrate that multimodality plays a fundamental role in VR; enhancing the experience, improving overall performance, and yielding unprecedented abilities in skill and knowledge transfer

The matrix revisited: A critical assessment of virtual reality technologies for modeling, simulation, and training

A convergence of affordable hardware, current events, and decades of research have advanced virtual reality (VR) from the research lab into the commercial marketplace. Since its inception in the 1960s, and over the next three decades, the technology was portrayed as a rarely used, high-end novelty for special applications. Despite the high cost, applications have expanded into defense, education, manufacturing, and medicine. The promise of VR for entertainment arose in the early 1990\u27s and by 2016 several consumer VR platforms were released. With VR now accessible in the home and the isolationist lifestyle adopted due to the COVID-19 global pandemic, VR is now viewed as a potential tool to enhance remote education. Drawing upon over 17 years of experience across numerous VR applications, this dissertation examines the optimal use of VR technologies in the areas of visualization, simulation, training, education, art, and entertainment. It will be demonstrated that VR is well suited for education and training applications, with modest advantages in simulation. Using this context, the case is made that VR can play a pivotal role in the future of education and training in a globally connected world

SIMULATING HUMAN SENSES TO IMPROVE THERMAL COMFORT

Between the synergies of environmental perception and technological advancement evolves the parallel world of the metaverse. Evolutionary virtuality intends to aid humanity in envisioning the threatened future of cities under environmental risks through tailored features. Traditionally, the sense of sight – which is the focus of virtual reality – has dominated the architectural practice. However, architects and urban designers have begun incorporating other senses into their work over the recent decade. The expanding understanding of the multimodal nature of the human mind that has evolved from cognitive neuroscience research has received little attention so far in the architecture field. This paper investigates the role of synthesized sensory experiences – such as visual, auditory, olfactory, gustatory, and thermal sensations – in designing revolutionary settings that aim to improve people’s interactions with their surrounding environments. A 15-minute experiment of an immersive experience in an office setting using virtual reality headsets is utilized to explore the role of multimodal sensory integration towards tolerance to the thermal environment. The findings revealed significant potential in using multiple senses – especially gustatory – to design thermally comfortable spaces. It is hoped that architectural design practice would progressively include our developing understanding of human senses and how they interact. This holistic approach ought to lead to the development of multisensory-inclusive workspaces that promote rather than hinder our social, cognitive, and emotional development

Breaking fresh ground in human–media interaction research

Human-Media Interaction research is devoted to methods and situations where humans individually or collectively interact with digital media, systems, devices and environments. Novel forms of interaction paradigms have been enabled by new sensor and actuator technology in the last decades, combining with advances in our knowledge of human-human interaction and human behavior in general when designing user interfaces

Bread stories: understanding the drivers of bread consumption for digital food customisation

Consumer demand for food that satisfies specific needs rather than generic mass produced food is growing. In response, the food industry is actively investigating techniques for efficient and comprehensive food customisation. Digital approaches to food customisation are starting to emerge, however, the majority is currently limited to the ingredient level thus excluding consumption drivers such as people’s practices and values around food. Using the approach of cultural probes, we identified four distinct narratives around bread consumption: the healthy bread, the fresh bread, the ethical bread, and the exceptional bread. These themes encapsulate drivers of bread consumption, which we argue can inform the design of digital food innovation platforms

Digital Creativity: Advantages, Problems, Responsibilities

The paper focuses on the rapid development of the digital culture and the challenges it imposes to human creativity. It analyses e-learning, digital entertainment, digital art and the issues of creativity and improvisation. It also presents a classification of the levels in the creative structure including hardware and software tools; product developers; creators and end users. Special attention is paid to the advantages of the new digital culture and the responsibilities of all people who create it or use it. We conclude that more attention should be paid to the threats and to ways of boosting positive creativity in the various fields of application of information and communication technologies

Multimodality in {VR}: {A} Survey

Virtual reality has the potential to change the way we create and consume content in our everyday life. Entertainment, training, design and manufacturing, communication, or advertising are all applications that already benefit from this new medium reaching consumer level. VR is inherently different from traditional media: it offers a more immersive experience, and has the ability to elicit a sense of presence through the place and plausibility illusions. It also gives the user unprecedented capabilities to explore their environment, in contrast with traditional media. In VR, like in the real world, users integrate the multimodal sensory information they receive to create a unified perception of the virtual world. Therefore, the sensory cues that are available in a virtual environment can be leveraged to enhance the final experience. This may include increasing realism, or the sense of presence; predicting or guiding the attention of the user through the experience; or increasing their performance if the experience involves the completion of certain tasks. In this state-of-the-art report, we survey the body of work addressing multimodality in virtual reality, its role and benefits in the final user experience. The works here reviewed thus encompass several fields of research, including computer graphics, human computer interaction, or psychology and perception. Additionally, we give an overview of different applications that leverage multimodal input in areas such as medicine, training and education, or entertainment; we include works in which the integration of multiple sensory information yields significant improvements, demonstrating how multimodality can play a fundamental role in the way VR systems are designed, and VR experiences created and consumed