Digitizing the chemical senses: possibilities & pitfalls

Many people are understandably excited by the suggestion that the chemical senses can be digitized; be it to deliver ambient fragrances (e.g., in virtual reality or health-related applications), or else to transmit flavour experiences via the internet. However, to date, progress in this area has been surprisingly slow. Furthermore, the majority of the attempts at successful commercialization have failed, often in the face of consumer ambivalence over the perceived benefits/utility. In this review, with the focus squarely on the domain of Human-Computer Interaction (HCI), we summarize the state-of-the-art in the area. We highlight the key possibilities and pitfalls as far as stimulating the so-called ‘lower’ senses of taste, smell, and the trigeminal system are concerned. Ultimately, we suggest that mixed reality solutions are currently the most plausible as far as delivering (or rather modulating) flavour experiences digitally is concerned. The key problems with digital fragrance delivery are related to attention and attribution. People often fail to detect fragrances when they are concentrating on something else; And even when they detect that their chemical senses have been stimulated, there is always a danger that they attribute their experience (e.g., pleasure) to one of the other senses – this is what we call ‘the fundamental attribution error’. We conclude with an outlook on digitizing the chemical senses and summarize a set of open-ended questions that the HCI community has to address in future explorations of smell and taste as interaction modalities

Earth as Interface: Exploring chemical senses with Multisensory HCI Design for Environmental Health Communication

As environmental problems intensify, the chemical senses -that is smell and taste, are the most relevantsenses to evidence them.As such, environmental exposure vectors that can reach human beings comprise air,food, soil and water[1].Within this context, understanding the link between environmental exposures andhealth[2]is crucial to make informed choices, protect the environment and adapt to new environmentalconditions[3].Smell and taste lead therefore to multi-sensorial experiences which convey multi-layered information aboutlocal and global events[4]. However, these senses are usually absent when those problems are represented indigital systems. The multisensory HCIdesign framework investigateschemical sense inclusion withdigital systems[5]. Ongoing efforts tackledigitalization of smell and taste for digital delivery, transmission or substitution [6]. Despite experimentsproved technological feasibility, its dissemination depends on relevant applicationdevelopment[7].This thesis aims to fillthose gaps by demonstratinghow chemical senses provide the means to link environment and health based on scientific andgeolocation narratives [8], [9],[10]. We present a Multisensory HCI design process which accomplished symbolicdisplaying smell and taste and led us to a new multi-sensorial interaction system presented herein. We describe the conceptualization, design and evaluation of Earthsensum, an exploratory case study project.Earthsensumoffered to 16 participants in the study, environmental smell and taste experiences about real geolocations to participants of the study. These experiences were represented digitally using mobilevirtual reality (MVR) and mobile augmented reality (MAR). Its technologies bridge the real and digital Worlds through digital representations where we can reproduce the multi-sensorial experiences. Our study findings showed that the purposed interaction system is intuitive and can lead not only to a betterunderstanding of smell and taste perception as also of environmental problems. Participants comprehensionabout the link between environmental exposures and health was successful and they would recommend thissystem as education tools. Our conceptual design approach was validated and further developments wereencouraged.In this thesis,we demonstratehow to applyMultisensory HCI methodology to design with chemical senses. Weconclude that the presented symbolic representation model of smell and taste allows communicatingtheseexperiences on digital platforms. Due to its context-dependency, MVR and MAR platforms are adequatetechnologies to be applied for this purpose.Future developments intend to explore further the conceptual approach. These developments are centredon the use of the system to induce hopefully behaviourchange. Thisthesisopens up new application possibilities of digital chemical sense communication,Multisensory HCI Design and environmental health communication.À medida que os problemas ambientais se intensificam, os sentidos químicos -isto é, o cheiroe sabor, são os sentidos mais relevantes para evidenciá-los. Como tais, os vetores de exposição ambiental que podem atingir os seres humanos compreendem o ar, alimentos, solo e água [1]. Neste contexto, compreender a ligação entre as exposições ambientais e a saúde [2] é crucial para exercerescolhas informadas, proteger o meio ambiente e adaptar a novas condições ambientais [3]. O cheiroe o saborconduzemassima experiências multissensoriais que transmitem informações de múltiplas camadas sobre eventos locais e globais [4]. No entanto, esses sentidos geralmente estão ausentes quando esses problemas são representados em sistemas digitais. A disciplina do design de Interação Humano-Computador(HCI)multissensorial investiga a inclusão dossentidos químicos em sistemas digitais [9]. O seu foco atual residena digitalização de cheirose sabores para o envio, transmissão ou substituiçãode sentidos[10]. Apesar dasexperimentaçõescomprovarem a viabilidade tecnológica, a sua disseminação está dependentedo desenvolvimento de aplicações relevantes [11]. Estatese pretendepreencher estas lacunas ao demonstrar como os sentidos químicos explicitama interconexãoentre o meio ambiente e a saúde, recorrendo a narrativas científicas econtextualizadasgeograficamente[12], [13], [14]. Apresentamos uma metodologiade design HCImultissensorial que concretizouum sistema de representação simbólica de cheiro e sabor e nos conduziu a um novo sistema de interação multissensorial, que aqui apresentamos. Descrevemos o nosso estudo exploratório Earthsensum, que integra aconceptualização, design e avaliação. Earthsensumofereceu a 16participantes do estudo experiências ambientais de cheiro e sabor relacionadas com localizações geográficasreais. Essas experiências foram representadas digitalmente através derealidade virtual(VR)e realidade aumentada(AR).Estas tecnologias conectamo mundo real e digital através de representações digitais onde podemos reproduzir as experiências multissensoriais. Os resultados do nosso estudo provaramque o sistema interativo proposto é intuitivo e pode levar não apenas a uma melhor compreensão da perceção do cheiroe sabor, como também dos problemas ambientais. O entendimentosobre a interdependência entre exposições ambientais e saúde teve êxitoe os participantes recomendariam este sistema como ferramenta para aeducação. A nossa abordagem conceptual foi positivamentevalidadae novos desenvolvimentos foram incentivados. Nesta tese, demonstramos como aplicar metodologiasde design HCImultissensorialpara projetar com ossentidos químicos. Comprovamosque o modelo apresentado de representação simbólica do cheiroe do saborpermite comunicar essas experiênciasem plataformas digitais. Por serem dependentesdocontexto, as plataformas de aplicações emVR e AR são tecnologias adequadaspara este fim.Desenvolvimentos futuros pretendem aprofundar a nossa abordagemconceptual. Em particular, aspiramos desenvolvera aplicaçãodo sistema para promover mudanças de comportamento. Esta tese propõenovas possibilidades de aplicação da comunicação dos sentidos químicos em plataformas digitais, dedesign multissensorial HCI e de comunicação de saúde ambiental

MediaSync: Handbook on Multimedia Synchronization

This book provides an approachable overview of the most recent advances in the fascinating field of media synchronization (mediasync), gathering contributions from the most representative and influential experts. Understanding the challenges of this field in the current multi-sensory, multi-device, and multi-protocol world is not an easy task. The book revisits the foundations of mediasync, including theoretical frameworks and models, highlights ongoing research efforts, like hybrid broadband broadcast (HBB) delivery and users' perception modeling (i.e., Quality of Experience or QoE), and paves the way for the future (e.g., towards the deployment of multi-sensory and ultra-realistic experiences). Although many advances around mediasync have been devised and deployed, this area of research is getting renewed attention to overcome remaining challenges in the next-generation (heterogeneous and ubiquitous) media ecosystem. Given the significant advances in this research area, its current relevance and the multiple disciplines it involves, the availability of a reference book on mediasync becomes necessary. This book fills the gap in this context. In particular, it addresses key aspects and reviews the most relevant contributions within the mediasync research space, from different perspectives. Mediasync: Handbook on Multimedia Synchronization is the perfect companion for scholars and practitioners that want to acquire strong knowledge about this research area, and also approach the challenges behind ensuring the best mediated experiences, by providing the adequate synchronization between the media elements that constitute these experiences

Musiikki ja tunteet monikulttuurisessa ympäristössä: musiikillisen sopivuuden etsiminen vuorovaikutteiselle urheiluanimaatiolle

Music is most commonly listened to influence emotions. Emotions can be expressed by musical structures and evoked by various psychological mechanisms. Evoked emotions are strongly connected to musical preference. When combined with visuals, music directs the attention of the user and enhances the emotional content of the narrative. In a commercial setting, matching the emotional qualities of the music with the qualities of the product is called musical fit. The objective of this study was to select music that would provide musical fit for Suunto Movie, which is a short, 3D animation built on GPS data recorded from the user’s own outdoor exercise. The music for Suunto Movie is built using interactive components designed to fit together flexibly to follow the dramatic curve of the video narration. This study compares and analyzes six different music samples for the Suunto Movie. The empirical data was gathered by an online self-report questionnaire. Responses were received from 148 users of the Suunto Movescount web community, most of them from Finland, Australia, USA, and UK. The respondents evaluated three music samples regarding the Liking and Musical fit. The Emotions evoked by music were evaluated using Geneva Emotional Music Scale, and the User Experience was measured with AttrakDiff semantic differential. Both scales were abridged to suit the purposes of this study. The results of the study illustrate that Liking, Musical fit, and User Experience all relate to each other, as they all examine Emotions. Liking correlates strongly with Musical fit (0,79), and correlates with the strongest positive emotion evoked (0,59). The average ratings of samples were overall fairly neutral. However, Sample B was rated as the most suitable composition as it received best ratings in Liking, Musical fit, and User Experience. Furthermore, it corresponded best with the Suunto Emotional Profile, presenting the emotions Suunto wishes to evoke as a brand. Further development for the Suunto Movie Music should include driving rhythms and even better synchronization of music with visuals. The respondents wished for recorded audio instead of synthetic sounds, and more rock music to match the sporting context and user preferences. Interesting directions for further research would be cross-cultural comparison and sonification.Useimmiten musiikkia kuunnellaan sen herättämien tunteiden takia. Musiikki ilmaisee tunteita erilaisten musiikillisten rakenteiden avulla ja herättää tunteita useiden psykologisten mekanismien kautta. Musiikin herättämät tunteet ovat vahvasti sidoksissa kuulijan musiikkimakuun. Kun musiikkia yhdistetään liikkuvaan kuvaan, se ohjaa käyttäjän huomion kuvaan ja vahvistaa kuvallisen kerronnan herättämiä tunteita. Kaupallisissa yhteyksissä musiikin emotionaalisen sisällön yhdistämistä tuotteen ominaisuuksiin kutsutaan musiikilliseksi sopivuudeksi. Tämän tutkimuksen tarkoituksena oli valita musiikillisesti sopiva musiikki Suunto Movielle, joka on lyhyt, kolmiulotteinen animaatio, joka on rakennettu käyttäjän oman urheilusuorituksen paikannustiedon pohjalta. Suunto Movien musiikki on rakennettu käyttäen interaktiivisia osia, jotka sopivat joustavasti yhteen, noudattaen samalla videokerronnan draaman kaarta. Tämä tutkimus vertailee ja analysoi kuutta erilaista musiikkinäytettä, jotka on sävelletty Suunto Movielle. Tutkimuksen empiirinen data kerättiin verkkokyselyllä, jossa vastaajat raportoivat omia tuntemuksiaan musiikkinäytteisiin liittyen. Kyselyyn vastasi 148 Suunnon Movescount -verkkoyhteisön käyttäjää, joista suurin osa oli Suomesta, Australiasta, Yhdysvalloista ja Iso-Britanniasta. Jokainen vastaaja arvioi kolmea musiikkinäytettä omien mieltymystensä ja näytteiden musiikillisen sopivuuden mukaan. Vastaajat analysoivat musiikin herättämiä tunteita Geneva Emotional Music Scalella ja arvioivat käyttäjäkokemusta AttrakDiff -semanttisella differentiaalilla Molempia työkaluja lyhennettiin tämän tutkimuksen tarpeisiin sopiviksi. Tutkimuksen tuloksista voidaan nähdä, että musiikkimaku, musiikillinen sopivuus ja käyttäjäkokemus ovat yhteydessä toisiinsa, sillä ne kaikki käsittelevät tunteita. Omat mieltymykset korreloivat vahvasti musiikillisen sopivuuden kanssa (0,79). Lisäksi ne korreloivat vahvimman koetun positiivisen tunteen kanssa (0,59). Kaikkiaan arvioiden keskiarvot olivat kohtalaisen neutraaleja. Musiikkinäyte B kuitenkin arvioitiin parhaiten sopivaksi, sillä siitä pidettiin eniten, se oli musiikillisesti sopivin ja sillä oli paras käyttäjäkokemus. Näyte B vastasi parhaiten Suunnon tunneprofiilia, joka kuvaa tunteita, joita Suunnon tuotemerkillä halutaan herättää. Suunto Movien musiikin jatkokehityksessä tulisi käyttää eteenpäin vieviä rytmejä ja synkronoida musiikki ja kuva paremmin keskenään. Vastaajat toivoivat synteettisten äänien korvaamista äänitetyillä soittimilla sekä enemmän urheilun yhteyteen ja vastaajien musiikkimakuun sopivaa rock-musiikkia. Mielenkiintoisiksi jatkotutkimuskohteiksi nousivat kulttuurierot ja sonifikaatio