Prosody and Kinesics Based Co-analysis Towards Continuous Gesture Recognition

The aim of this study is to develop a multimodal co-analysis framework for continuous gesture recognition by exploiting prosodic and kinesics manifestation of natural communication. Using this framework, a co-analysis pattern between correlating components is obtained. The co-analysis pattern is clustered using K-means clustering to determine how well the pattern distinguishes the gestures. Features of the proposed approach that differentiate it from the other models are its less susceptibility to idiosyncrasies, its scalability, and simplicity. The experiment was performed on Multimodal Annotated Gesture Corpus (MAGEC) that we created for research on understanding non-verbal communication community, particularly the gestures

Advanced Content and Interface Personalization through Conversational Behavior and Affective Embodied Conversational Agents

Conversation is becoming one of the key interaction modes in HMI. As a result, the conversational agents (CAs) have become an important tool in various everyday scenarios. From Apple and Microsoft to Amazon, Google, and Facebook, all have adapted their own variations of CAs. The CAs range from chatbots and 2D, carton-like implementations of talking heads to fully articulated embodied conversational agents performing interaction in various concepts. Recent studies in the field of face-to-face conversation show that the most natural way to implement interaction is through synchronized verbal and co-verbal signals (gestures and expressions). Namely, co-verbal behavior represents a major source of discourse cohesion. It regulates communicative relationships and may support or even replace verbal counterparts. It effectively retains semantics of the information and gives a certain degree of clarity in the discourse. In this chapter, we will represent a model of generation and realization of more natural machine-generated output

A review of temporal aspects of hand gesture analysis applied to discourse analysis and natural conversation

Lately, there has been a\ud n increasing\ud interest in hand gesture analysis systems. Recent works have employed\ud pat\ud tern recognition techniques and have focused on the development of systems with more natural user\ud interfaces. These systems may use gestures to control interfaces or recognize sign language gestures\ud , which\ud can provide systems with multimodal interaction; o\ud r consist in multimodal tools to help psycholinguists to\ud understand new aspects of discourse analysis and to automate laborious tasks.\ud Gestures are characterized\ud by several aspects, mainly by movements\ud and sequence of postures\ud . Since data referring to move\ud ments\ud or\ud sequences\ud carry temporal information\ud , t\ud his paper presents a\ud literature\ud review\ud about\ud temporal aspects of\ud hand gesture analysis, focusing on applications related to natural conversation and psycholinguistic\ud analysis, using Systematic Literature Revi\ud ew methodology. In our results, we organized works according to\ud type of analysis, methods, highlighting the use of Machine Learning techniques, and applications.FAPESP 2011/04608-

Semi-automation of gesture annotation by machine learning and human collaboration

none6siGesture and multimodal communication researchers typically annotate video data manually, even though this can be a very time-consuming task. In the present work, a method to detect gestures is proposed as a fundamental step towards a semi-automatic gesture annotation tool. The proposed method can be applied to RGB videos and requires annotations of part of a video as input. The technique deploys a pose estimation method and active learning. In the experiment, it is shown that if about 27% of the video is annotated, the remaining parts of the video can be annotated automatically with an F-score of at least 0.85. Users can run this tool with a small number of annotations first. If the predicted annotations for the remainder of the video are not satisfactory, users can add further annotations and run the tool again. The code has been released so that other researchers and practitioners can use the results of this research. This tool has been confirmed to work in conjunction with ELAN.openIenaga, Naoto; Cravotta, Alice; Terayama, Kei; Scotney, Bryan W.; Saito, Hideo; Busà, M. GraziaIenaga, Naoto; Cravotta, Alice; Terayama, Kei; Scotney, Bryan W.; Saito, Hideo; Busà, M. Grazi

French Face-to-Face Interaction: Repetition as a Multimodal Resource

International audienceIn this chapter, after presenting the corpus as well as some of theannotations developed in the OTIM project, we then focus on the specificphenomenon of repetition. After briefly discussing this notion, we showthat different degrees of convergence can be achieved by speakersdepending on the multimodal complexity of the repetition and on thetiming in between the repeated element and the model. Although we focusmore specifically on the gestural level, we present a multimodal analysis ofgestural repetitions in which we met several issues linked to multimodalannotations of any type. This gives an overview of crucial issues in crosslevellinguistic annotation, such as the definition of a phenomenonincluding formal and/or functional categorization

Gesture and Speech in Interaction - 4th edition (GESPIN 4)

International audienceThe fourth edition of Gesture and Speech in Interaction (GESPIN) was held in Nantes, France. With more than 40 papers, these proceedings show just what a flourishing field of enquiry gesture studies continues to be. The keynote speeches of the conference addressed three different aspects of multimodal interaction:gesture and grammar, gesture acquisition, and gesture and social interaction. In a talk entitled Qualitiesof event construal in speech and gesture: Aspect and tense, Alan Cienki presented an ongoing researchproject on narratives in French, German and Russian, a project that focuses especially on the verbal andgestural expression of grammatical tense and aspect in narratives in the three languages. Jean-MarcColletta's talk, entitled Gesture and Language Development: towards a unified theoretical framework,described the joint acquisition and development of speech and early conventional and representationalgestures. In Grammar, deixis, and multimodality between code-manifestation and code-integration or whyKendon's Continuum should be transformed into a gestural circle, Ellen Fricke proposed a revisitedgrammar of noun phrases that integrates gestures as part of the semiotic and typological codes of individuallanguages. From a pragmatic and cognitive perspective, Judith Holler explored the use ofgaze and hand gestures as means of organizing turns at talk as well as establishing common ground in apresentation entitled On the pragmatics of multi-modal face-to-face communication: Gesture, speech andgaze in the coordination of mental states and social interaction.Among the talks and posters presented at the conference, the vast majority of topics related, quitenaturally, to gesture and speech in interaction - understood both in terms of mapping of units in differentsemiotic modes and of the use of gesture and speech in social interaction. Several presentations explored the effects of impairments(such as diseases or the natural ageing process) on gesture and speech. The communicative relevance ofgesture and speech and audience-design in natural interactions, as well as in more controlled settings liketelevision debates and reports, was another topic addressed during the conference. Some participantsalso presented research on first and second language learning, while others discussed the relationshipbetween gesture and intonation. While most participants presented research on gesture and speech froman observer's perspective, be it in semiotics or pragmatics, some nevertheless focused on another importantaspect: the cognitive processes involved in language production and perception. Last but not least,participants also presented talks and posters on the computational analysis of gestures, whether involvingexternal devices (e.g. mocap, kinect) or concerning the use of specially-designed computer software forthe post-treatment of gestural data. Importantly, new links were made between semiotics and mocap data

Time gestures:time conceptualisation in English with evidence from gestures in a multimodal corpus

This thesis is the first large-scale, corpus-based analysis of time conceptualisation using multimodal ‘big’ data. It exploits the NewsScape Library, an extensive television database, to investigate co-speech gestures for temporal expressions in English. This research investigates time conceptualisation by studying gesture patterns that co-occur with temporal linguistic expressions. While much has been written about time conceptualisation based on linguistic evidence, there has been comparatively little based on other modalities. Non-linguistic data has the capacity to help triangulate existing findings if found to be congruent with linguistic data or, conversely, could offer new insights into the way time is conceptualised. Gestures are an invaluable tool for the study of cognition due to their key role in human communication. Gesture is a universal phenomenon, largely unconscious and less monitored than speech and thus it can reveal information not present in language. In this thesis, I investigate three features of co-speech gestures: axis and direction; language-gesture congruency; and extent or distance. The dataset comprises gestures that co-occur with temporal linguistic expressions which can be further divided into three groups: non-spatial language (e.g. earlier than); spatial, directional language (e.g. back in those days); and spatial, non-directional language (e.g. far in the future). This research confirms that the lateral axis is frequently employed when gesturing about time, but it also suggests that spatial language usesthe sagittal axis more frequently that non-spatial language. Moreover, gestures that co-occur with spatially grounded temporal language tend to be congruent with the canonical direction of time, while this does not occur with non-spatially grounded temporal language. Finally, it seems that temporal distance expressed linguistically is usually analogous to spatial distance expressed through co-speech gestures. The thesis also reflects on theoretical and methodological issues for gesture studies and some possible steps forward in the study of gesture

Context-aware gestural interaction in the smart environments of the ubiquitous computing era

A thesis submitted to the University of Bedfordshire in partial fulfilment of the requirements for the degree of Doctor of PhilosophyTechnology is becoming pervasive and the current interfaces are not adequate for the interaction with the smart environments of the ubiquitous computing era. Recently, researchers have started to address this issue introducing the concept of natural user interface, which is mainly based on gestural interactions. Many issues are still open in this emerging domain and, in particular, there is a lack of common guidelines for coherent implementation of gestural interfaces. This research investigates gestural interactions between humans and smart environments. It proposes a novel framework for the high-level organization of the context information. The framework is conceived to provide the support for a novel approach using functional gestures to reduce the gesture ambiguity and the number of gestures in taxonomies and improve the usability. In order to validate this framework, a proof-of-concept has been developed. A prototype has been developed by implementing a novel method for the view-invariant recognition of deictic and dynamic gestures. Tests have been conducted to assess the gesture recognition accuracy and the usability of the interfaces developed following the proposed framework. The results show that the method provides optimal gesture recognition from very different view-points whilst the usability tests have yielded high scores. Further investigation on the context information has been performed tackling the problem of user status. It is intended as human activity and a technique based on an innovative application of electromyography is proposed. The tests show that the proposed technique has achieved good activity recognition accuracy. The context is treated also as system status. In ubiquitous computing, the system can adopt different paradigms: wearable, environmental and pervasive. A novel paradigm, called synergistic paradigm, is presented combining the advantages of the wearable and environmental paradigms. Moreover, it augments the interaction possibilities of the user and ensures better gesture recognition accuracy than with the other paradigms

Interacção gestual sem superfícies de apoio

Tese de mestrado em Engenharia Informática (Sistemas de Informação), apresentada à Universidade de Lisboa, através da Faculdade de Ciências, 2011Os periféricos de entrada deixaram de ser a única forma de transmitir intenç-¸ ões à máquina, sendo agora possível fazê-lo com o próprio corpo. Dispositivos que permitem interacção gestual sem recurso a periféricos intermediários têm vindo a aumentar, principalmente na área dos jogos. Esta tendência levanta várias questões a serem investigadas na área da interacção pessoa-máquina. A aproximação simplista de transferir conceitos de interacção do paradigma clássico WIMP, baseado nos dispositivos tradicionais de entrada, rato e teclado, rapidamente conduz a problemas inesperados. As características de uma interface concebida para uma interacção gestual em que não há contacto com nenhum dispositivo de entrada não se irão adequar ao paradigma utilizado nos últimos 40 anos. Estamos assim em condições de explorar como a interacção gestual com ou sem voz pode contribuir para minimizar os problemas com o paradigma clássico WIMP no tipo de interacção em que não há o contacto com nenhum periférico. Neste trabalho irá ser explorado o campo da interacção gestual, com ou sem voz. Através de aplicações pretende-se conduzir vários estudos de manipulação de objectos virtuais baseada em visão computacional. A manipulação dos objectos é realizada com dois modos de interacção (gestos e voz) podendo estes surgir integrados ou não. Pretende-se analisar se a interacção gestual é apelativa para os utilizadores para alguns tipos de aplicações e acções, enquanto para outros tipos, os gestos poderão não ser a modalidade preferida de interacção.The input peripherals aren’t anymore the only way to transmit intentions to the machine, being now possible to do it with our own body. The number of devices that allow gestural interaction, without the need of intermediate peripherals, are increasing, mainly in the area of video games. This tendency raises several questions that need to be investigated in the area of person-machine interaction. The simplistic approach of transferring interaction concepts from the classic paradigm WIMP, based on the traditional input devices, mouse and keyboard, quickly leads to unexpected problems. The characteristics of an interface conceived to a gestural interaction were there isn’t any kind of contact with an input device won’t suit with the paradigm of the last 40 years. So we’re in conditions to exploit how the gestural interaction can contribute to minimize the classic paradigm issues. In this work the field of gestural interaction, with and without voice, will be analyzed. Through the use of applications, it’s intended to lead various studies of virtual objects manipulation based on computational vision. The objects manipulation is done with two kinds of interactions, gestural and voice, that may emerge integrated our not. It’s intended to analyze if the gestural interaction is appealing to the users for some kind of applications and actions, while for other types, gestural may not be the preferred interaction modality

Interacção por toque em múltiplas superfícies

Tese de mestrado, Engenharia Informática (Arquitectura, Sistemas e Redes de Computadores), Universidade de Lisboa, Faculdade de Ciências, 2009Actualmente, com o preço dos pixéis a diminuir, os ecrãs de computador tendem a aumentar de tamanho. Os ecrãs de parede e outras superfícies de interacção de grande dimensão são agora uma opção para muitos utilizadores. Esta tendência levanta várias questões a serem investigadas na área da interacção pessoa-máquina. A aproximação simplista de transferir os principais conceitos de interacção do paradigma clássico WIMP (Windows, Icon, Menu, Pointer / Janela, Ícone, Menu, Dispositivo Apontador), baseado nos dispositivos tradicionais de entrada, rato e teclado, rapidamente conduz a problemas inesperados. Nos últimos anos também se assistiu à emergência dos primeiros produtos comerciais a suportarem interacções multi-toque. É expectável que a tecnologia do toque se torne brevemente standard, o que já é visível em alguns mercados específicos, como o dos telemóveis. Se juntarmos as possibilidades criadas pela recente “revolução táctil” e a transição a que assistimos nos últimos anos para ecrãs de grande dimensão, estamos agora em condições de explorar como a interacção gestual pode contribuir para minimizar os problemas com o paradigma clássico WIMP em ecrãs de grande dimensão. Neste trabalho é explorado o campo da interacção gestual em ecrãs de grande dimensão. Foram conduzidos vários estudos, onde os utilizadores experimentaram interacção gestual, em várias aplicações adequadas para ecrãs de grande dimensão. Os resultados demonstram que a manipulação directa através de interacção gestual é apelativa aos utilizadores para alguns tipos de aplicações e acções, enquanto para outros tipos, os gestos não são a modalidade preferida de interacção. Posteriormente, introduziu-se o uso da interacção gestual para cenários cooperativos, discutindo a sua adequação a algumas tarefas, e a forma como os utilizadores decidem cooperativamente que tarefas realizar baseados nas modalidades de entrada disponíveis e nas características das tarefas.Nowadays, with pixels getting cheaper, computer displays tend toward larger sizes. Wall sized screens and other large interaction surfaces are now an option for many users and this trend raises a number of issues to be researched in the user interface area. The simplistic approach of transferring the main interaction concepts of the classic WIMP (Window, Icon, Menu, Pointer) design paradigm, based on the traditional mouse and keyboard input devices, quickly led to unexpected problems. In recent years we also witnessed a revolution with the emergence of the first commercial products supporting multi-touch interaction. It is expected that the use of touch technology will soon become standard, and this is already visible in some specific markets, such as mobile phones. If we put together the possibilities opened up by the recent “touch revolution” and the transition we have been witnessing for the past few years to large screen displays, we are now able to explore how the use of gestural interaction can contribute to overcome the problems with the classical WIMP paradigm in large screen displays. In this work we explore the field of gestural interaction on large screen displays, conducting several studies where users experience gestural interaction in various applications suited for large displays. Results show how direct manipulation through gestural interaction appeals to users for some types of applications and actions, while demonstrating that for other types, gestures should not be the preferred interaction modality. Afterward, we introduce the use of gestural interaction for cooperative scenarios, discussing how it is more suited for some tasks, and hypothesizing on how users cooperatively decide on which tasks to perform based on the available input modalities and task characteristics