Multi-Moji: Combining Thermal, Vibrotactile and Visual Stimuli to Expand the Affective Range of Feedback

This paper explores the combination of multiple concurrent modalities for conveying emotional information in HCI: temperature, vibration and abstract visual displays. Each modality has been studied individually, but can only convey a limited range of emotions within two-dimensional valencearousal space. This paper is the first to systematically combine multiple modalities to expand the available affective range. Three studies were conducted: Study 1 measured the emotionality of vibrotactile feedback by itself; Study 2 measured the perceived emotional content of three bimodal combinations: vibrotactile + thermal, vibrotactile + visual and visual + thermal. Study 3 then combined all three modalities. Results show that combining modalities increases the available range of emotional states, particularly in the problematic top-right and bottom-left quadrants of the dimensional model. We also provide a novel lookup resource for designers to identify stimuli to convey a range of emotions

Semefulness: A social semiotics of touch

This paper explores the multiple significances (semefulness) of touch, as experienced by us as embodied subjects. Prompted by the development of a range of touch-based technologies, I consider the current writings about touch in a range of fields and how these have contributed to contemporary understandings of the meanings of touch. I then explore a number of these meanings - connection, engagement, contiguity, differentiation, positioning - for their contribution to our understanding of the world and of our own embodied subjectivity. I also explore the deployment of these meanings by contemporary technologies. © 2011 Taylor & Francis

Sonification and Music, Music and Sonification

Despite it being more than twenty years since the launch of an international conference series dedicated to its study, there is still much debate over what sonification really is, and especially as regards its relationship to music. A layman’s definition of sonification might be that it is the use of non-speech audio to communicate data, the aural counterpart to visualization. Many researchers have claimed musicality for their sonifications, generally when using data-to-pitch mappings. In 2006 Bennett Hogg and I (Vickers and Hogg 2006) made a rather provocative assertion that bound music and sonification together (q.v., and further developed in Vickers (2006)), not so much to claim an ontological truth but to foreground a debate that has simmered since the first International Conference on Auditory Display (ICAD) in 1992. Since then there has been an increasing number of musical and sonic art compositions driven by the data of natural phenomena, some of which are claimed by their authors to be sonifications. This chapter looks at some of the issues surrounding the relationship between sonification and music and at developments that have the potential to draw sonification and the sonic arts into closer union

Haptic and Audio Interaction Design: First International Workshop, HAID 2006, Glasgow, UK, August 31-September 1, 2006, Proceedings

Here are the proceedings of the First International Workshop on Haptic and Audio Interaction Design, HAID 2006. The book presents 15 revised full papers, in a variety of disciplines ranging from psychology to art, showcasing how haptics and sound can improve user interaction with computers, helping people with various kinds of disabilities and visual impairment. Also addresses psychophysics, art and leisure, and mobile applications that improve selections in virtual environments

Haptic and Audio Interaction Design: First International Workshop, HAID 2006, Glasgow, UK, August 31-September 1, 2006, Proceedings

Here are the proceedings of the First International Workshop on Haptic and Audio Interaction Design, HAID 2006. The book presents 15 revised full papers, in a variety of disciplines ranging from psychology to art, showcasing how haptics and sound can improve user interaction with computers, helping people with various kinds of disabilities and visual impairment. Also addresses psychophysics, art and leisure, and mobile applications that improve selections in virtual environments

Development of the multimodal system of educational game for partially sighted and blind children

У дисертацији су приказани развој мултимодалног система за образовну игру Луграм и истраживања везана за његову примену у области редовног и специјализованог основношколског образовања. Извршено је испитивање утицаја Луграма на повишење ефеката учења геометрије у редовној разредној настави, испитивање да ли његову прилагођену мултимодалну верзију могу да користе слабовида и слепа деца и да ли се она може користити за њихову обуку за самосталну употребу рачунарске аудио верзије Луграма. Утврђено је да Луграм има утицаја на пораст успеха у учењу геометрије и да се прилагођена аудио- тактилна верзија може користити у сврху обуке слабовиде и слепе деце за самостално коришћење аудио верзије игре. Резултати истраживања су показали оправданост приступа развоју Луграма као мултимодалног система за игру и усмерили његов даљи развој ка асистивном мултимодалном систему који поседује способност говорне интеракције са корисником и могућност прилагођења различитим категоријама корисника.U disertaciji su prikazani razvoj multimodalnog sistema za obrazovnu igru Lugram i istraživanja vezana za njegovu primenu u oblasti redovnog i specijalizovanog osnovnoškolskog obrazovanja. Izvršeno je ispitivanje uticaja Lugrama na povišenje efekata učenja geometrije u redovnoj razrednoj nastavi, ispitivanje da li njegovu prilagođenu multimodalnu verziju mogu da koriste slabovida i slepa deca i da li se ona može koristiti za njihovu obuku za samostalnu upotrebu računarske audio verzije Lugrama. Utvrđeno je da Lugram ima uticaja na porast uspeha u učenju geometrije i da se prilagođena audio- taktilna verzija može koristiti u svrhu obuke slabovide i slepe dece za samostalno korišćenje audio verzije igre. Rezultati istraživanja su pokazali opravdanost pristupa razvoju Lugrama kao multimodalnog sistema za igru i usmerili njegov dalji razvoj ka asistivnom multimodalnom sistemu koji poseduje sposobnost govorne interakcije sa korisnikom i mogućnost prilagođenja različitim kategorijama korisnika.The dissertation presents the development of the multimodal system for the educational game Lugram and the research related to its application in the regular and specialized primary education. The examination of the effect which Lugram has on the improvement in learning geometry in the primary education is done, as well as the testing weather its adapted multimodal version can be used for the training of partially sighted and blind children for the independent use of the computer audio version of Lugram. It is established that Lugram has an influence on the improvement in learning geometry and that adapted audio-tactile version can be used for the training purposes for partially sighted and blind children. Results of the research have justified the approach in the development of Lugram as a multimodal system for the game. Moreover, they focus further development toward assistive multimodal system which has ability of the voice interaction with the user and also ability of adaptation to the different kind of users

O ambiente virtual áudio-háptico como instrumento para a Aprendizagem de geometria : estudo das formas para estudantes cegos

O desenvolvimento das habilidades para os objetos de conhecimento da geometria está relacionado com as formas de organização de aprendizagem matemática e com os recursos didáticos utilizados para a construção de competências para o pensamento matemático (BRASIL, 2017). Esta tese se propõe a investigar como um ambiente virtual áudio-háptico pode contribuir na aprendizagem de geometria para estudantes cegos no Ensino Fundamental, com base em uma sequência de tarefas que contemplem os objetivos de conhecimento e habilidades previstos na Base Nacional Comum Curricular (BNCC). O referencial teórico proposto para esta pesquisa destaca os seguintes temas: a Teoria da Abstração Reflexionante de Piaget (1995), que traz contribuições para a compreensão da construção de conhecimentos; a percepção háptica humana na perspectiva de Lederman e Klatzky (1987), que propõem um conjunto de procedimentos exploratórios (Exploratory Procedure – EPs) para uma pessoa examinar um objeto com ou sem visão no intuito de perceber propriedades por intermédio do tato, bem como discutem as sensações que são convertidas pelo cérebro em informações cutâneas e cinestésica; a tecnologia háptica de hardware, em que se apresentam os dispositivos que permitem a interação de pessoas com ambientes virtuais por meio do tato e feedback de força; e o estudo da geometria, evidenciando a importância da aprendizagem dos conceitos geométricos no Ensino Fundamental, e a Tecnologia Assistiva digital com foco no sentido tátil-cinestésico para inclusão de estudantes cegos no estudo da geometria. A pesquisa apresenta abordagem qualitativa de natureza aplicada e foi realizada no Instituto Benjamin Constant, instituição de ensino para deficientes visuais localizada no bairro da Urca, na cidade e no estado do Rio de Janeiro. Para a coleta de dados, foram utilizadas as técnicas de observação participante, gravação de vídeo e Think Aloud, com o intuito de explorar os fatores de eficácia e eficiência e o mecanismo de abstração reflexionante na construção de conhecimentos geométricos. Os dados coletados foram analisados por meio da técnica de análise categorial prevista no método de análise de conteúdo (BARDIN, 2016). Acredita-se que esta tese possa contribuir como um recurso assistivo que apoie a aprendizagem de geometria – estudo das formas para estudantes cegos no Ensino Fundamental.The development of skills for the objects of knowledge of geometry is related to the forms of organization of mathematical learning and the didactic resources used to build competencies for mathematical thinking (BRASIL, 2017). This thesis aims to investigate how an audiohaptic virtual environment can contribute to Geometry Learning for Blind Elementary School Students, based on a sequence of tasks that address the objectives of knowledge and skills foreseen in the Common National Curriculum Base (BNCC). The theoretical framework proposed for this research highlights the following themes: Piaget's theory of reflect abstraction, Piaget (1995), which brings contributions to the understanding of the construction of knowledge; the human pain perception from the perspective of Lederman and Klatzky (1987), who propose a set of Exploratory Procedures (EPs) for a person to examine an object with or without vision in order to perceive properties through touch, as well as discuss the sensations that are converted by the brain into cutaneous and kinesthetic information; the political technology of hardware, in which the devices that allow the interaction of people with virtual environments through touch and force feedback are presented; and the study of geometry, evidencing the importance of learning geometric concepts in Elementary School, and digital assistive technology focused on the tactile-kinesthetic sense for the inclusion of blind students in the study of geometry. The research presents a qualitative approach of an applied nature and was held at the Benjamin Constant Institute, an educational institution for the visually impaired located in the Urca neighborhood, in the city and in the state of Rio de Janeiro. For data collection of this investigation, participant observation, video recording and Think Aloud techniques were used in order to explore the factors of efficacy and efficiency and the mechanism of reflecting abstraction in the construction of geometric knowledge. The collected data were analyzed using the category analysis technique provided for in the content analysis method (BARDIN, 2016). It is believed that this can contribute as an assistive resource that supports the learning of geometry – study of forms for blind students in Elementary School