Earth as Interface: Exploring chemical senses with Multisensory HCI Design for Environmental Health Communication

As environmental problems intensify, the chemical senses -that is smell and taste, are the most relevantsenses to evidence them.As such, environmental exposure vectors that can reach human beings comprise air,food, soil and water[1].Within this context, understanding the link between environmental exposures andhealth[2]is crucial to make informed choices, protect the environment and adapt to new environmentalconditions[3].Smell and taste lead therefore to multi-sensorial experiences which convey multi-layered information aboutlocal and global events[4]. However, these senses are usually absent when those problems are represented indigital systems. The multisensory HCIdesign framework investigateschemical sense inclusion withdigital systems[5]. Ongoing efforts tackledigitalization of smell and taste for digital delivery, transmission or substitution [6]. Despite experimentsproved technological feasibility, its dissemination depends on relevant applicationdevelopment[7].This thesis aims to fillthose gaps by demonstratinghow chemical senses provide the means to link environment and health based on scientific andgeolocation narratives [8], [9],[10]. We present a Multisensory HCI design process which accomplished symbolicdisplaying smell and taste and led us to a new multi-sensorial interaction system presented herein. We describe the conceptualization, design and evaluation of Earthsensum, an exploratory case study project.Earthsensumoffered to 16 participants in the study, environmental smell and taste experiences about real geolocations to participants of the study. These experiences were represented digitally using mobilevirtual reality (MVR) and mobile augmented reality (MAR). Its technologies bridge the real and digital Worlds through digital representations where we can reproduce the multi-sensorial experiences. Our study findings showed that the purposed interaction system is intuitive and can lead not only to a betterunderstanding of smell and taste perception as also of environmental problems. Participants comprehensionabout the link between environmental exposures and health was successful and they would recommend thissystem as education tools. Our conceptual design approach was validated and further developments wereencouraged.In this thesis,we demonstratehow to applyMultisensory HCI methodology to design with chemical senses. Weconclude that the presented symbolic representation model of smell and taste allows communicatingtheseexperiences on digital platforms. Due to its context-dependency, MVR and MAR platforms are adequatetechnologies to be applied for this purpose.Future developments intend to explore further the conceptual approach. These developments are centredon the use of the system to induce hopefully behaviourchange. Thisthesisopens up new application possibilities of digital chemical sense communication,Multisensory HCI Design and environmental health communication.À medida que os problemas ambientais se intensificam, os sentidos químicos -isto é, o cheiroe sabor, são os sentidos mais relevantes para evidenciá-los. Como tais, os vetores de exposição ambiental que podem atingir os seres humanos compreendem o ar, alimentos, solo e água [1]. Neste contexto, compreender a ligação entre as exposições ambientais e a saúde [2] é crucial para exercerescolhas informadas, proteger o meio ambiente e adaptar a novas condições ambientais [3]. O cheiroe o saborconduzemassima experiências multissensoriais que transmitem informações de múltiplas camadas sobre eventos locais e globais [4]. No entanto, esses sentidos geralmente estão ausentes quando esses problemas são representados em sistemas digitais. A disciplina do design de Interação Humano-Computador(HCI)multissensorial investiga a inclusão dossentidos químicos em sistemas digitais [9]. O seu foco atual residena digitalização de cheirose sabores para o envio, transmissão ou substituiçãode sentidos[10]. Apesar dasexperimentaçõescomprovarem a viabilidade tecnológica, a sua disseminação está dependentedo desenvolvimento de aplicações relevantes [11]. Estatese pretendepreencher estas lacunas ao demonstrar como os sentidos químicos explicitama interconexãoentre o meio ambiente e a saúde, recorrendo a narrativas científicas econtextualizadasgeograficamente[12], [13], [14]. Apresentamos uma metodologiade design HCImultissensorial que concretizouum sistema de representação simbólica de cheiro e sabor e nos conduziu a um novo sistema de interação multissensorial, que aqui apresentamos. Descrevemos o nosso estudo exploratório Earthsensum, que integra aconceptualização, design e avaliação. Earthsensumofereceu a 16participantes do estudo experiências ambientais de cheiro e sabor relacionadas com localizações geográficasreais. Essas experiências foram representadas digitalmente através derealidade virtual(VR)e realidade aumentada(AR).Estas tecnologias conectamo mundo real e digital através de representações digitais onde podemos reproduzir as experiências multissensoriais. Os resultados do nosso estudo provaramque o sistema interativo proposto é intuitivo e pode levar não apenas a uma melhor compreensão da perceção do cheiroe sabor, como também dos problemas ambientais. O entendimentosobre a interdependência entre exposições ambientais e saúde teve êxitoe os participantes recomendariam este sistema como ferramenta para aeducação. A nossa abordagem conceptual foi positivamentevalidadae novos desenvolvimentos foram incentivados. Nesta tese, demonstramos como aplicar metodologiasde design HCImultissensorialpara projetar com ossentidos químicos. Comprovamosque o modelo apresentado de representação simbólica do cheiroe do saborpermite comunicar essas experiênciasem plataformas digitais. Por serem dependentesdocontexto, as plataformas de aplicações emVR e AR são tecnologias adequadaspara este fim.Desenvolvimentos futuros pretendem aprofundar a nossa abordagemconceptual. Em particular, aspiramos desenvolvera aplicaçãodo sistema para promover mudanças de comportamento. Esta tese propõenovas possibilidades de aplicação da comunicação dos sentidos químicos em plataformas digitais, dedesign multissensorial HCI e de comunicação de saúde ambiental

The Oneiric Reality of Electronic Scents

This paper investigates the ‘oneiric’ dimension of scent, by suggesting a new design process that can be worn as a fashion accessory or integrated in textile technologies, to subtly alter reality and go beyond our senses. It fuses wearable ‘electronic scent’ delivery systems with pioneering biotechnologies as a ground-breaking ‘science fashion’ enabler. The purpose is to enhance wellbeing by reaching a day‐dream state of being through the sense of smell. The sense of smell (or olfaction) is a chemical sense and part of the limbic system which regulates emotion and memory within the brain. The power of scent makes content extremely compelling by offering a heightened sense of reality which is intensified by emotions such as joy, anger and fear. Scent helps us appreciate all the senses as we embark on a sensory journey unlike any other; it enhances mood, keeps us in the moment, diverts us from distractions, reduces boredom and encourages creativity. This paper highlights the importance of smell, the forgotten sense, and also identifies how we as humans have grown to underuse our senses. It endeavours to show how the reinvention of our sensory faculties is possible through advances in biotechnology. It introduces the new ‘data senses’ as a wearable sensory platform that triggers and fine tunes the senses with fragrances. It puts forward a new design process that is currently being developed in clothing elements, jewellery and textile technologies, offering a new method to deliver scent electronically and intelligently in fashion and everyday consumer products. It creates a personal ‘scent wave’, around the wearer, to allow the mind to wander, to give a deeper sense of life or ‘lived reality’ (verses fantasy), a new found satisfaction and confidence, and to reach new heights of creativity. By combining biology with wearable technologies, we propose a biotechnological solution that can be translated into sensory fashion elements. This is a new trend in 21st century ‘data sensing’, based on holographic biosensors that sense the human condition, aromachology (the science of the effect of fragrance and behaviour), colour-therapy, and smart polymer science. The use of biosensors in the world of fashion and textiles, enables us to act on visual cues or detect scent signals and rising stress levels, allowing immediate information to hand. An ‘oneiric’ mood is triggered by a spectrum of scents which is encased in a micro-computerised ‘scent‐cell’ and integrated into clothing elements or jewellery. When we inhale an unexpected scent, it takes us by surprise; the power of fragrance fills us with pleasurable ripples of multi‐sensations and dream‐like qualities. The aromas create a near trance‐like experience that induces a daydream state of (immediate) satisfaction, or a ‘revived reality’ in our personal scent bubble of reality. The products and jewellery items were copyrighted and designed by Slim Barrett and the technology input was from EG Technology and Epigem

Olfaction-enhanced multimedia: Perspectives and challenges

This is the post-print version of the Article. The official published version can be accessed from the link below - Copyright @ 2011 Springer VerlagOlfaction—or smell—is one of the last challenges which multimedia and multimodal applications have to conquer. Enhancing such applications with olfactory stimuli has the potential to create a more complex—and richer—user multimedia experience, by heightening the sense of reality and diversifying user interaction modalities. Nonetheless, olfaction-enhanced multimedia still remains a challenging research area. More recently, however, there have been initial signs of olfactory-enhanced applications in multimedia, with olfaction being used towards a variety of goals, including notification alerts, enhancing the sense of reality in immersive applications, and branding, to name but a few. However, as the goal of a multimedia application is to inform and/or entertain users, achieving quality olfaction-enhanced multimedia applications from the users’ perspective is vital to the success and continuity of these applications. Accordingly, in this paper we have focused on investigating the user perceived experience of olfaction-enhanced multimedia applications, with the aim of discovering the quality evaluation factors that are important from a user’s perspective of these applications, and consequently ensure the continued advancement and success of olfaction-enhanced multimedia applications

LeviSense: a platform for the multisensory integration in levitating food and insights into its effect on flavour perception

Eating is one of the most multisensory experiences in everyday life. All of our five senses (i.e. taste, smell, vision, hearing and touch) are involved, even if we are not aware of it. However, while multisensory integration has been well studied in psychology, there is not a single platform for testing systematically the effects of different stimuli. This lack of platform results in unresolved design challenges for the design of taste-based immersive experiences. Here, we present LeviSense: the first system designed for multisensory integration in gustatory experiences based on levitated food. Our system enables the systematic exploration of different sensory effects on eating experiences. It also opens up new opportunities for other professionals (e.g., molecular gastronomy chefs) looking for innovative taste-delivery platforms. We describe the design process behind LeviSense and conduct two experiments to test a subset of the crossmodal combinations (i.e., taste and vision, taste and smell). Our results show how different lighting and smell conditions affect the perceived taste intensity, pleasantness, and satisfaction. We discuss how LeviSense creates a new technical, creative, and expressive possibilities in a series of emerging design spaces within Human-Food Interaction

Which senses dominate at different stages of product experience?

In the area of product design, sensory dominance can be defined as the relative importance of different sensory modalities for product experience. Since product experience is multisensory, it is interesting to know which sensory modality plays a leading role in a particular experience, so that designers could concentrate on the creation of the most relevant product properties. It is often assumed that vision dominates other senses. In the present study, we investigated the importance of different sensory modalities during various episodes of product usage. We asked 120 respondents to describe their experiences with consumer products in the following situations: while buying a product, after the first week, the first month, and the first year of usage. The data suggest that the dominant modality depends on the period of product usage. At the moment of buying, vision is the most important modality, but at later stages other modalities become more important. The dominance of a particular modality may depend on its appropriateness for the particular task. During long-term usage, modality importance depends on product functions and the characteristics of the user-product interaction. We conclude that to create a long-lasting positive product experience, designers need to consider the user-product interaction at different stages of product usage and to determine which sensory modality dominates product experience at each stage. Keywords: Sensory Dominance; User-Product Interaction; Product Design</p

JENTIL: responsive clothing that promotes an ‘holistic approach to fashion as a new vehicle to treat psychological conditions’

This paper explores an ongoing interdisciplinary research project at the cutting edge of sensory, aroma and medical work, which seeks to change the experience of fragrance to a more intimate communication of identity, by employing emerging technologies with the ancient art of perfumery. The project illustrates .holistic' clothing called the JENTIL® Collection, following on from the Author’s SmartSecondSkin' PhD research, which describes a new movement in functional, emotional clothing that incorporates scent. The project investigates the emergent interface between the arts and biomedical sciences, around new emerging technologies and science platforms, and their applications in the domain of health and well-being. The JENTIL® Collection focuses on the development of .gentle., responsive clothing that changes with emotion, since the garments are designed for psychological end benefit to reduce stress. This is achieved by studying the mind and advancing knowledge and understanding of how known well-being fragrances embedded in holistic Fashion, could impact on mental health. This paper aims to combine applied theories about human well-being, with multisensory design, in order to create experimental strategies to improve self and social confidence for individuals suffering from depressive illnesses. The range of methodologies employed extends beyond the realm of fashion and textile techniques, to areas such as neuroscience, psychiatry, human sensory systems and affective states, and the increase in popularity of complementary therapies. In this paper the known affective potential of the sense of smell is discussed, by introducing Aroma-Chology as a tool that is worn as an emotional support system to create a personal scent bubble. around the body, with the capacity to regulate mood, physiological and psychological state and improve self-confidence in social situations. The clothing formulates a healing platform around the wearer, by creating novel olfactory experiences in textiles that are not as passive as current microencapsulated capsule systems generally are

Information and communication in a networked infosphere: a review of concepts and application in social branding

This paper aims at providing a contribution to the comprehensive review of the impact of information and communication, and their supporting technologies, in the current transformation of human life in the infosphere. The paper also offers an ex- ample of the power of new social approaches to the use of information and commu- nication technologies to foster new working models in organizations by presenting the main outcomes of a research project on social branding. A discussion about some trends of the future impact of new information and communication technologies in the infosphere is also included

User quality of experience of mulsemedia applications

User Quality of Experience (QoE) is of fundamental importance in multimedia applications and has been extensively studied for decades. However, user QoE in the context of the emerging multiple-sensorial media (mulsemedia) services, which involve different media components than the traditional multimedia applications, have not been comprehensively studied. This article presents the results of subjective tests which have investigated user perception of mulsemedia content. In particular, the impact of intensity of certain mulsemedia components including haptic and airflow on user-perceived experience are studied. Results demonstrate that by making use of mulsemedia the overall user enjoyment levels increased by up to 77%