Classifying types of gesture and inferring intent

In order to infer intent from gesture, a rudimentary classification of types of gestures into five main classes is introduced. The classification is intended as a basis for incorporating the understanding of gesture into human-robot interaction (HRI). Some requirements for the operational classification of gesture by a robot interacting with humans are also suggested

Towards a Linguistically Motivated Irish Sign Language Conversational Avatar

Avatars are life-like characters that exist in a virtual world on our computer monitors. They are synthetic actors that have, in more recent times, received a significant amount of investigation and development. This is primarily due to leverage gained from advances in computing power and 3D animation technologies. Since the release of the movie “Avatar” last year, there is also a broader awareness and interest in avatars in the public domain. Ishizuka and Prendinger (2004) describe how researchers, while endeavouring to develop a creature that is believable and capable of intelligible communication, use a wide variety of terms to describe their work: avatars, anthropomorphic agents, creatures, synthetic actors, non-player characters, embodied conversational agents, bots, intelligent agents. While most of these terms are inspired from the character specific applications, some intend to draw attention to a particular aspect of the life-like character. To date it seems that there is no universal agreement with regard to terminology. The term avatar can be used to refer to the visual representation of a human being within a virtual environment whereas the term embodied conversational agent refers to a character that visually incorporates knowledge with regard to the conversational process. For the purpose of this research, the term embodied conversational agent is deemed an appropriate descriptor for the synthetic agent undergoing development. The value that RRG contributes to this is that it is a theory of grammar that is concerned with the interaction of syntax, semantics and pragmatics across grammatical systems. RRG can be characterised as a descriptive framework for the analysis of languages and also an explanatory framework for the analysis of language acquisition (Van Valin, 2008). As a lexicalist theory of grammar, RRG can be described as being well motivated cross-linguistically. The grammar model links the syntactic structure of a sentence to the semantic structure by means of a linking algorithm, which is bi-directional in nature. With respect to cognitive issues, RRG adopts the criterion of psychological adequacy formulated in Dik (1991), which states that a theory should be compatible with the results of psycholinguistic research on the acquisition, processing, production, interpretation and memorisation of linguistic expressions. It also accepts the criterion put forward in Bresnan and Kaplan (1982), that theories of linguistic structure should be directly relatable to testable theories of language production and comprehension. RRG incorporates many of the viewpoints of current functional grammar theories. RRG takes language to be a system of communicative social action, and accordingly, analysing the communicative functions of grammatical structures plays a vital role in grammatical description and theory from this perspective. The view of the lexicon in RRG is such that lexical entries for verbs should contain unique information only, while as much information as possible should be derived from general lexical rules. It is envisaged that the RRG parser/generator described in this paper will later be used as a component in the development of a computational framework for the embodied conversational agent for ISL. This poses significant technical and theoretical difficulties within both RRG and for software (Nolan and Salem 2009, Salem, Hensman and Nolan 2009). As ISL is a visual gestural language without any aural or written form, like all other sign languages, the challenge is to extend the RRG view of the lexicon and the layered structure of the word, indeed the model itself, to accommodate sign languages. In particular, the morphology of sign languages is concerned with manual and non-manual features, handshapes across the dominant and non-dominant hand in simultaneous signed constructions, head, eyebrows and mouth shape. These are the morphemes and lexemes of sign language. How can these fit into the RRG lexicon and what are the difficulties this presents for RRG at the semantic-morphosyntax interface? This paper will discuss this research as a work in progress to date. It is envisaged that the embodied conversational agent undergoing development in this research will later be employed for real-time sign language visualisation for Irish Sign Language (ISL)

Exploring the Affective Loop

Research in psychology and neurology shows that both body and mind are involved when experiencing emotions (Damasio 1994, Davidson et al. 2003). People are also very physical when they try to communicate their emotions. Somewhere in between beings consciously and unconsciously aware of it ourselves, we produce both verbal and physical signs to make other people understand how we feel. Simultaneously, this production of signs involves us in a stronger personal experience of the emotions we express. Emotions are also communicated in the digital world, but there is little focus on users' personal as well as physical experience of emotions in the available digital media. In order to explore whether and how we can expand existing media, we have designed, implemented and evaluated /eMoto/, a mobile service for sending affective messages to others. With eMoto, we explicitly aim to address both cognitive and physical experiences of human emotions. Through combining affective gestures for input with affective expressions that make use of colors, shapes and animations for the background of messages, the interaction "pulls" the user into an /affective loop/. In this thesis we define what we mean by affective loop and present a user-centered design approach expressed through four design principles inspired by previous work within Human Computer Interaction (HCI) but adjusted to our purposes; /embodiment/ (Dourish 2001) as a means to address how people communicate emotions in real life, /flow/ (Csikszentmihalyi 1990) to reach a state of involvement that goes further than the current context, /ambiguity/ of the designed expressions (Gaver et al. 2003) to allow for open-ended interpretation by the end-users instead of simplistic, one-emotion one-expression pairs and /natural but designed expressions/ to address people's natural couplings between cognitively and physically experienced emotions. We also present results from an end-user study of eMoto that indicates that subjects got both physically and emotionally involved in the interaction and that the designed "openness" and ambiguity of the expressions, was appreciated and understood by our subjects. Through the user study, we identified four potential design problems that have to be tackled in order to achieve an affective loop effect; the extent to which users' /feel in control/ of the interaction, /harmony and coherence/ between cognitive and physical expressions/,/ /timing/ of expressions and feedback in a communicational setting, and effects of users' /personality/ on their emotional expressions and experiences of the interaction