Advanced Content and Interface Personalization through Conversational Behavior and Affective Embodied Conversational Agents

Conversation is becoming one of the key interaction modes in HMI. As a result, the conversational agents (CAs) have become an important tool in various everyday scenarios. From Apple and Microsoft to Amazon, Google, and Facebook, all have adapted their own variations of CAs. The CAs range from chatbots and 2D, carton-like implementations of talking heads to fully articulated embodied conversational agents performing interaction in various concepts. Recent studies in the field of face-to-face conversation show that the most natural way to implement interaction is through synchronized verbal and co-verbal signals (gestures and expressions). Namely, co-verbal behavior represents a major source of discourse cohesion. It regulates communicative relationships and may support or even replace verbal counterparts. It effectively retains semantics of the information and gives a certain degree of clarity in the discourse. In this chapter, we will represent a model of generation and realization of more natural machine-generated output

EVERY \u27ONE\u27 AND EVERY \u27THING\u27 CAN BE LOVED : A RHETORICAL ANALYSIS OF NETWORKED SELF-REPRESENTATION BY THE OBJECTùM SEXUALITY COMMUNITY

Using rhetorical criticism informed by actor-network theory (ANT), in this dissertation I explore the emergence of queer identity and queer community building within the Objectùm Sexuality Internationale Web site (OSI)--the largest source of information related to a community of over 300 hundred individuals who experience emotional and romantic desire towards objects. My goals in this study are (1) to identify and understand how rhetorical strategies are emergent and networked (rather than individually enacted) within the OSI Web site; and (2) how these emergent rhetorical strategies promote multiplicity of sexual desire and identity through the challenging of heteronormative and anthropocentric binaries and normativities via queer posthuman forms of love and connection. Using an ANT informed rhetorical criticism, I identified four layers of communication that facilitate the emergence of actor networks within the OSI Web site: (1) translation--the process by which human actors depict experience in texts); (2) enactment--the process by which actors (human and object) interact in ways that create networks of action and agency); (3) representation--the process in which certain macroactors (actors that appear as recurring and stable categories) present the interests of other actors within the network); and (4) teleaction--the movement of representations from place to place and over time through memory and text. Within these layers, I identified four categories of translation, thirteen macroactors, and four types of teleaction. The translations that emerge on the OSI Web site include how objectùm sexuality became a term and community, what it means to be objectùm sexual, how people who identify as objectùm sexual have come to make sense of their experiences, and public pleas for acceptance regarding objectùm sexuality. The macroactors that emerge include people, communication devices, purposes of OSI, orientation, animism, sensuality/intimacy, nonverbal communication, love, gender, attraction, marriage, medicalization, and the Red Fence. The processes of teleaction that emerge include verbal, nonverbal, hybrid, and symbolic actors. These four layers then led to the emergence of four higher-level rhetorical dimensions. These include: (1) terminological dimension-- the interrelationship between terms and the OS community; (2) ontological dimension--the emergence of a higher-level philosophy about the existence of beings and the meanings and modes of being, existing, living, and loving for OS; (3) axiological dimension--the emergence of criteria for ethical values and judgments in relation to OS; and (4) epistemological dimension--where the dimensions of ontology and terminology meet and the nature and scope of knowledge about OS is represented. Together, these four transcendent levels facilitate the rhetorical construction of the OS community and critiques of heteronormative/anthropocentric frames of love, desire, and sexuality. Overall, these various strategies lead to two larger rhetorical moves: (1) OSI communicates and adapts to internal and external audiences; and (2) OSI rhetoric moves from specific meanings to larger paradigmatic shifts that reveal is function as a social movement within a single rhetorical text. This process of rhetorical strategy building positions OS within intelligible frameworks of understanding in order to: (1) provide information about OS that will mitigate fear and sensationalism and facilitate acceptance; (2) construct an OSI community identity and human-object desire more generally; and (3) direct people away from heteronormative and anthropocentric worldviews and toward a queer posthuman worldviews of love, desire, and connection

Modelling the relationship between gesture motion and meaning

There are many ways to say “Hello,” be it a wave, a nod, or a bow. We greet others not only with words, but also with our bodies. Embodied communication permeates our interactions. A fist bump, thumbs-up, or pat on the back can be even more meaningful than hearing “good job!” A friend crossing their arms with a scowl, turning away from you, or stiffening up can feel like a harsh rejection. Social communication is not exclusively linguistic, but is a multi-sensory affair. It’s not that communication without these bodily cues is impossible, but it is impoverished. Embodiment is a fundamental human experience. Expressing ourselves through our bodies provides a powerful channel through which we express a plethora of meta-social information. And integral to communication, expression, and social engagement is our utilization of conversational gesture. We use gestures to express extra-linguistic information, to emphasize our point, and to embody mental and linguistic metaphors that add depth and color to social interaction. The gesture behaviour of virtual humans when compared to human-human conversation is limited, depending on the approach taken to automate performances of these characters. The generation of nonverbal behaviour for virtual humans can be approximately classified as either: 1) data-driven approaches that learn a mapping from aspects of the verbal channel, such as prosody, to gestures; or 2) rule bases approaches that are often tailored by designers for specific applications. This thesis is an interdisciplinary exploration that bridges these two approaches, and brings data-driven analyses to observational gesture research. By marrying a rich history of gesture research in behavioral psychology with data-driven techniques, this body of work brings rigorous computational methods to gesture classification, analysis, and generation. It addresses how researchers can exploit computational methods to make virtual humans gesture with the same richness, complexity, and apparent effortlessness as you and I. Throughout this work the central focus is on metaphoric gestures. These gestures are capable of conveying rich, nuanced, multi-dimensional meaning, and raise several challenges in their generation, including establishing and interpreting a gesture’s communicative meaning, and selecting a performance to convey it. As such, effectively utilizing these gestures remains an open challenge in virtual agent research. This thesis explores how metaphoric gestures are interpreted by an observer, how one can generate such rich gestures using a mapping between utterance meaning and gesture, as well as how one can use data driven techniques to explore the mapping between utterance and metaphoric gestures. The thesis begins in Chapter 1 by outlining the interdisciplinary space of gesture research in psychology and generation in virtual agents. It then presents several studies that address presupposed assumptions raised about the need for rich, metaphoric gestures and the risk of false implicature when gestural meaning is ignored in gesture generation. In Chapter 2, two studies on metaphoric gestures that embody multiple metaphors argue three critical points that inform the rest of the thesis: that people form rich inferences from metaphoric gestures, these inferences are informed by cultural context and, more importantly, that any approach to analyzing the relation between utterance and metaphoric gesture needs to take into account that multiple metaphors may be conveyed by a single gesture. A third study presented in Chapter 3 highlights the risk of false implicature and discusses this in the context of current subjective evaluations of the qualitative influence of gesture on viewers. Chapters 4 and 5 then present a data-driven analysis approach to recovering an interpretable explicit mapping from utterance to metaphor. The approach described in detail in Chapter 4 clusters gestural motion and relates those clusters to the semantic analysis of associated utterance. Then, Chapter 5 demonstrates how this approach can be used both as a framework for data-driven techniques in the study of gesture as well as form the basis of a gesture generation approach for virtual humans. The framework used in the last two chapters ties together the main themes of this thesis: how we can use observational behavioral gesture research to inform data-driven analysis methods, how embodied metaphor relates to fine-grained gestural motion, and how to exploit this relationship to generate rich, communicatively nuanced gestures on virtual agents. While gestures show huge variation, the goal of this thesis is to start to characterize and codify that variation using modern data-driven techniques. The final chapter of this thesis reflects on the many challenges and obstacles the field of gesture generation continues to face. The potential for applications of Virtual Agents to have broad impacts on our daily lives increases with the growing pervasiveness of digital interfaces, technical breakthroughs, and collaborative interdisciplinary research efforts. It concludes with an optimistic vision of applications for virtual agents with deep models of non-verbal social behaviour and their potential to encourage multi-disciplinary collaboration

Participation as a tool for interactional work on Twitter: A sociolinguistic approach to social media 'engagement'

This work approaches the concept of social media engagement through a lens of participation theory. Following the work of Goffman (1981) and others, this dissertation uses the concepts of the participation framework and the participant role to explore engagement as a function of participation in interaction. The purposes of this dissertation are three-fold: to model participant roles as they are built in interaction on Twitter, to discover the ways in which participation is established through the linguistic choices enacted by participants, and to demonstrate the role of the medium as an important factor influencing possibilities for participation. Using discourse analysis as a methodology, tweets from accounts associated with National Hockey League (NHL) organizations are analyzed for the linguistic resources that are used to reference interactional roles traditionally understood as “speaker” and “hearer”. In turn, the linguistic and discursive resources deployed in team tweets are used to reveal these speaker and hearer roles as more detailed and complex production and reception frameworks. The modal affordances of Twitter are also investigated as to their role in influencing the building of participation frameworks through talk, including unique linguistic forms that are available to Twitter users and possibilities for hiding or revealing participants through the Twitter screen. The findings of this investigation reveal three primary models for production frameworks for NHL accounts: an Impersonal Model that eschews identification of the parties in production roles, an Interpersonal Model that highlights the individuals involved in the interaction, and a Team Model that obscures the individual to focus on the team or organization as a primary participant. Additionally, a framework for understanding recipient audiences on Twitter is proposed, incorporating both actual and intended audiences. Consistent patterns in the language choices used to construct participatory identities for production and reception roles are demonstrated, highlighting the value of using linguistic data as a resource for investigations of participation. Finally, Twitter’s modal affordances are shown to be an integral part of the ways that users enact participatory concepts, such as co-presence and address, revealing the importance of considering the role of the medium in participation studies

Development of social cognition in the early years of life in the context of the child-mother relationship

Tese de doutoramento em Psicologia (área de conhecimento de Psicologia Clínica)The aim of this PhD is twofold: 1) To investigate whether infant joint attention at 10 months is developmentally linked to later social symbolic play abilities at 3 years of age, as both are thought to be milestones of social cognition; and 2) To explore individual, relational and contextual contributions to joint attention and social symbolic play. Method: Fifty-two infants were assessed at 10 months for joint attention (following and initiating joint attention behaviors) in interaction with their mothers. At 3 years of age, 49 children were reassessed, this time focusing on their social symbolic play abilities with an experimenter. Information regarding other relevant variables was also gathered at both ages. At 10 months infants’ expression of negative emotionality, temperament and developmental level were assessed. Data on mother-infant relationship quality, maternal bids for joint attention (entertaining, teaching and attention directing behaviors) and social-demographic features were also collected. At 3 years, we assessed children’s temperament, verbal and non-verbal abilities, as well mothers’ and children’s mental state talk in a shared pretend play interaction and maternal mind-mindedness is a short interview. Relevant social-demographic information was also collected. Results: At 10 months, following joint attention was exclusively predicted by total maternal bids for joint attention, although marginal associations were found with maternal entertaining and attention-directing behaviors (but not teaching strategies). Initiating joint attention was predicted by infants’ low expression of negative emotionality and the presence of older siblings, as well as marginally predicted by less maternal teaching behaviors. At age 3, children’s social symbolic play abilities were not significantly predicted by infant joint attention. Conversely, social symbolic play was significantly predicted by children’s verbal abilities and their use of desire references in a shared pretense interaction with their mothers. Finally, we also found specific associations between children’s references to desires and their social symbolic play, and between children’s references to cognitions and their general cognitive development. Conclusion: Findings highlight the importance of a multilevel approach to the study of social cognition through infancy to preschool years, one that encompasses not only individual variables, but also a variety of social influences.Este projecto de doutoramento tem dois objectivos principais: 1) Investigar se a atenção partilhada aos 10 meses está desenvolvimentalmente associada à competência simbólica social aos 3 anos de idade; e 2) Explorar os contributos individuais, relacionais e contextuais para a atenção partilhada e o jogo simbólico social. Método: Cinquenta e dois bebés foram avaliados aos 10 meses ao nível da atenção partilhada (comportamentos de seguir e iniciar atenção partilhada) com as suas mães. Aos 3 anos, 49 crianças foram reavaliadas ao nível do seu jogo simbólico social com um experimentador. Foi ainda recolhida informação adicional relevante nas duas idades. Aos 10 meses avaliámos a expressão de emocionalidade negativa, o temperamento e o nível de desenvolvimento dos bebés. Foram igualmente recolhidos dados acerca da qualidade da relação mãe-bebé, das estratégias maternas de atenção partilhada (comportamentos orientados para entreter, ensinar ou dirigir a atenção do bebé) e de características sócio-demográficas. Aos 3 anos avaliámos o temperamento e a capacidade verbal e não-verbal das crianças, assim como o uso de palavras mentais por parte da mãe e da criança no decurso de uma brincadeira de faz-de-conta, e a mind-mindedness materna numa pequena entrevista. Foram ainda recolhidos dados relativos a variáveis sócio-demográficas de relevância. Resultados: Aos 10 meses, apenas o total de estratégias maternas de atenção partilhada foi um preditor significativo do seguimento de atenção partilhada por parte do bebé. No entanto, encontrámos associações marginalmente significativas entre o seguir da atenção partilhada e as estratégias da mãe destinadas a entreter o bebé e a direccionar a sua atenção (mas não estratégias com o objectivo de lhe ensinar algo). O iniciar da atenção partilhada foi significativamente predito pela baixa expressão de emocionalidade negativa do bebé, pela presença de irmãos mais velhos, e marginalmente predito por menos comportamentos maternos com a intenção de ensinar algo ao bebé. Aos 3 anos, verificámos que a atenção partilhada na infância não era um preditor significativo das competências de jogo simbólico social. Em contraste, a capacidade verbal e o uso de palavras relativas a desejos por parte da criança foram preditores significativos do seu jogo simbólico social. Por último, encontrámos associações específicas entre as referências a desejos por parte da criança e o seu jogo simbólico social, e entre as referências a cognições e o seu desenvolvimento cognitivo global. Conclusão: Os dados sublinham a importância de uma abordagem multinível ao estudo da cognição social nos primeiros anos de vida, uma abordagem que englobe não apenas variáveis individuais, mas igualmente uma variedade de influências sociais.Fundação para a Ciência e a Tecnologia (FCT) – SFRH/BD/27925/2006 suportado pelo Ministério da Ciência, Tecnologia e Ensino Superior, no âmbito da QREN - POPH - Typologia 4.1 - Formação Avançada, reembolsados pelo Fundo Social Europeu e por fundos MSTH

Affective Computing

This book provides an overview of state of the art research in Affective Computing. It presents new ideas, original results and practical experiences in this increasingly important research field. The book consists of 23 chapters categorized into four sections. Since one of the most important means of human communication is facial expression, the first section of this book (Chapters 1 to 7) presents a research on synthesis and recognition of facial expressions. Given that we not only use the face but also body movements to express ourselves, in the second section (Chapters 8 to 11) we present a research on perception and generation of emotional expressions by using full-body motions. The third section of the book (Chapters 12 to 16) presents computational models on emotion, as well as findings from neuroscience research. In the last section of the book (Chapters 17 to 22) we present applications related to affective computing

6th International Conference on Multimodality - Conference guide with abstracts

The aim of the conference was to contribute to moving forward the field of multimodal research and to help connect the diverse community of scholars working within it. 6ICOM is a place where we can explore the full range of different ways in which multimodality has been taken up and where we can recognize their points of connection. The conference was organised with support from the UK’s Economic and Social Research Council, who fund MODE. MODE is a node of the National Centre for Research Methods based in the Institute of Education and aims to develop and promote multimodal methods for researching digital data and environments. 6ICOM’s programme includes an impressive set of paper presentations (125) and invited keynotes (5). The presenters engage with a wide range of disciplines, ideas and methods, reflecting the diverse character of multimodality and latest developments in the field. They speak to a range of contexts, theoretical and methodological approaches, technologies and types of data

Families are for thriving, not just surviving : aged out foster youths' experiences of family, home, and estrangement

Guided by social identity (Tajfel & Turner, 1986), social constructionism (Berger & Luckmann, 1967; Gergen, 1985), and discourse-dependence (Galvin, 2006; Galvin & Braithwaite, 2014) theorizing, the present study illuminated the communicative experiences and complexities inherent to family and home life for youth who aged out of foster care. Broadly, results from 30 interviews afforded a necessary understanding of how aged out foster youth understand and process "family," "home," and family identity formation and deconstruction in conversations with others. First, findings revealed that "family" was conceptualized as those who love you unconditionally, as those who support you, and as more than blood -- signifying a valuing of function (e.g., love, support) over structure (e.g., blood/legal ties) when defining-family. The criteria aged out foster youth applied to “family” informed aged out foster youths' decision-making processes about what individuals were identified as family ingroup members, family outgroup members, and liminal group members. Second, results indicated that participants drew upon their experiences before, during, and after care when making sense of home as self, home as a place, and home as family. Third, findings revealed that aged out foster youth utilized naming, discussing, narrating, ritualizing, and normalizing to build and maintain family identity with family ingroup members. Fourth, results indicated that aged out foster youth most often estranged from biological family members due to physical actions, personal attributes, and/or lack of social support. Ultimately, participants reported estranging from family members because they defied their personal standards for family relationships. In order to actively deconstruct family identity with these family members, aged out foster youth utilized the discursive strategies of naming, discussing, deritualizing, silencing, and disassociating to deliberately defy standards of family as a way to create distance. Last, results revealed that aged out foster youth tended to focus on the positive impacts of family estrangement on the self through discussing both emotional implications (i.e., positive affect, negative affect, and mixed affect) and self-improvement implications (i.e., demonstrating empowerment, encouraging self-care, and promoting mental health). Findings contribute to family communication and family estrangement research by illuminating how aged out foster youth accounts spoke to family estrangement as a potential pathway to self-actualization. Results also advance discourse-dependence theorizing by empirically testing Galvin and Braithwaite's (2014) proposed identity deconstruction strategies and illuminating how standards for identity-building talk have heightened. Moreover, findings contribute to social identity and intergroup theorizing through revealing how the family ingroup is being reimagined and complicating our understandings of intergroup distinctions. A host of practical implications, such as offering estrangement coaching and developing practical skills for evaluating family, also emerged. Ultimately, results from the current study pave the way for future research to continue to explore how family and home life are discussed and experienced among aged out foster youth.Includes bibliographical reference

Communicative Constructions and the Refiguration of Spaces

The Open Access version of this book, available at www.taylorfrancis.com , has been made available under a Creative Commons Attribution-Non Commercial-No Derivatives 4.0 license Through a variety of empirical studies, this volume offers fresh insights into the manner in which different forms of communicative action transform urban space. With attention to the methodological questions that arise from the attempt to study such changes empirically, it offers new theoretical foundations for understanding the social construction and reconstruction of spaces through communicative action. Seeing communicative action as the basic element in the social construction of reality and conceptualizing communication not only in terms of the use of language and texts, but as involving any kind of objectification, such as technologies, bodies and non-verbal signs, it considers the roles of both direct and mediatized (or digitized) communication. An examination of the conceptualization of the communicative (re-)construction of spaces and the means by which this change might be empirically investigated, this book demonstrates the fruitfulness of the notion of refiguration as a means by which to understand the transformation of contemporary societies. As such, it will appeal to sociologists, social theorists, and geographers with interests in social construction and urban space