E-VChat : 頭部動作連動型音声駆動身体引き込みキャラクタを対面合成した実映像対話システム

Abstract – We proposed an embodied video communication system in which a humantypeavatar called “VirtualActor” which represents interactive behavior is superimposedon the other speech partner’s video image in a virtual face-to-face scene. The effectivenessof a video communication system was demonstrated in an experiment of comparisonwith the scene in which a reduced own video image is superimposed on the other talker’svideo image using the picture-in-picture method. However, this system had some problems,such as the detailed adjustment of video images and the lack of portability of sensors.In this paper, we develop a headset-type motion-capture device which reflects the talker’shead movements directly using an acceleration sensor and gyro sensor, and employ a CGcharacter which moves based on talker’s own motion and generates motion automaticallybased on the on-off pattern of talker’s voice. Further, we propose the concept ofan embodied video communication system in which the CG character is superimposedon the other talker’s video image in a face-to-face scene, and develop a prototype called“E-VChat”. A communication experiment is performed to confirm the effectiveness ofthe E-VChat system for 12 pairs of subjects using three communication modes: “Headset,”“Headset + Generated motion automatically as a talker’s avatar,” and “Headset+ Generated motion automatically as an talker’s support agent.” The results show thatall communication modes tested are affirmatively assessed by sensory evaluation, andthe “Headset + Generated motion automatically as a talker’s avatar” mode is evaluatedhighly by a paired comparison. Finally, we develop a multiple-character E-VChat systemusing an audience that nods in response to the talker’s voice, and confirm the effectivenessof the system in an interview-style communication experiment

HAI Alice -An Information-Providing Closed-Domain Dialog Corpus

International audienceThe contribution of this paper is twofold: 1) we provide a public corpus for Human-Agent Interaction (where the agent is controlled by a Wizard of Oz) and 2) we show a study on verbal alignment in Human-Agent Interaction, to exemplify the corpus' use. In our recordings for the Human-Agent Interaction Alice-corpus (HAI Alice-corpus), participants talked to a wizarded agent, who provided them with information about the book Alice in Wonderland and its author. The wizard had immediate and almost full control over the agent's verbal and nonverbal behavior, as the wizard provided the agent's speech through his own voice and his facial expressions were directly copied onto the agent. The agent's hand gestures were controlled through a button interface. Data was collected to create a corpus with unexpected situations, such as misunderstandings, (accidental) false information, and interruptions. The HAI Alice-corpus consists of transcribed audio-video recordings of 15 conversations (more than 900 utterances) between users and the wizarded agent. As a use-case example, we measured the verbal alignment between the user and the agent. The paper contains information about the setup of the data collection, the unexpected situations and a description of our verbal alignment study

Feeling the groove: shared time and its meanings for three jazz trios

The notion of groove is fundamental to jazz culture and the term yields a rich set of understandings for jazz musicians. Within the literature, no single perspective on groove exists and many questions remain about the relationship between timing processes, phenomenal experience and musical structures in making sense of groove. In this account, the experience and meaning of groove is theorised as emerging from two forms of sharedness. Firstly, a primary intersubjectivity that arises through the timing behaviours of the players; this could be likened to the 'mutual tuning-in' described in social phenomenology. It is proposed that this tuning-in is accomplished through the mechanism of entrainment. The second form of sharedness is understood as the shared temporal models, the cultural knowledge, that musicians make use of in their playing together. Methodologically, this study makes use of detailed investigation of timing data from live performances by three jazz trios, framed by in-depth, semi-structured interview material and steers a new course between existing ethnographic work on jazz and more psychologically informed studies of timing. The findings of the study point towards significant social and structural effects on the groove between players. The impact of musical role on groove and timing is demonstrated and significant temporal models, whose syntactic relations suggest musical proximity or distance, are shown to have a corresponding effect on timing within the trios. The musician's experience of groove is discussed as it relates to the objective timing data and reveals a complex set of understandings involving temporality, consciousness and communication. In the light of these findings, groove is summarised as the feeling of entrainment, inflected through cultural models and expressed through the cultural norms of jazz

A PHENOMENOLOGY OF MIMETIC LEARNING AND MULTIMODAL COGNITION: INTEGRATING EXPERIENTIAL KNOWLEDGE INTO PROGRAMS IN RHETORIC, COMPOSITION, AND TECHNICAL COMMUNICATION

My dissertation emphasizes a cognitive account of multimodality that explicitly integrates experiential knowledge work into the rhetorical pedagogy that informs so many composition and technical communication programs. In these disciplines, multimodality is widely conceived in terms of what Gunther Kress calls “socialsemiotic” modes of communication shaped primarily by culture. In the cognitive and neurolinguistic theories of Vittorio Gallese and George Lakoff, however, multimodality is described as a key characteristic of our bodies’ sensory-motor systems which link perception to action and action to meaning, grounding all communicative acts in knowledge shaped through body-engaged experience. I argue that this “situated” account of cognition – which closely approximates Maurice Merleau-Ponty’s phenomenology of perception, a major framework for my study – has pedagogical precedence in the mimetic pedagogy that informed ancient Sophistic rhetorical training, and I reveal that training’s multimodal dimensions through a phenomenological exegesis of the concept mimesis. Plato’s denigration of the mimetic tradition and his elevation of conceptual contemplation through reason, out of which developed the classic Cartesian separation of mind from body, resulted in a general degradation of experiential knowledge in Western education. But with the recent introduction into college classrooms of digital technologies and multimedia communication tools, renewed emphasis is being placed on the “hands-on” nature of inventive and productive praxis, necessitating a revision of methods of instruction and assessment that have traditionally privileged the acquisition of conceptual over experiential knowledge. The model of multimodality I construct from Merleau-Ponty’s phenomenology, ancient Sophistic rhetorical pedagogy, and current neuroscientific accounts of situated cognition insists on recognizing the significant role knowledges we acquire experientially play in our reading and writing, speaking and listening, discerning and designing practices

Music and the Evolution of Embodied Cognition

Music is a universal human activity. Its evolution and its value as a cognitive resource are starting to come into focus. This chapter endeavors to give readers a clearer sense of the adaptive aspects of music, as well as the underlying cognitive and neural structures. Special attention is given to the important emotional dimensions of music, and an evolutionary argument is made for thinking of music as a prelinguistic embodied form of cognition—a form that is still available to us as contemporary music creators and consumers

Landscapes of Affective Interaction: Young Children's Enactive Engagement with Body Metaphors

Empirical research into embodied meaning making suggests specific sensorimotor experiences can support children’s understanding of abstract science ideas. This view is aligned with enactive and grounded cognition perspectives, both centred in the view that our ability to conceptualise emerges from our experiences of interaction with our environment. While much of this research has focused on understanding action and action processes in individual children or children in pairs, less attention has been paid to affective dimensions of young children’s group interaction, and how this relates to meaning making with body metaphors. Indeed, Gallagher describes how no action exists in a vacuum, but rather revolves around a complex web of affective-pragmatic features comprising a ‘Landscape of Interaction’ (2020, p.42). This research project addresses gaps in research in understanding young children’s affective engagement from an enactivist cognition perspective. It takes a Design-Based Research approach with an iterative design orientation to examine young children’s interaction with multisensory body-based metaphors through an embodied participation framework. A series of empirical studies with young children, aged 2-7 years, comprising of experiential workshops, build iteratively upon each other. A novel theoretically informed method, Affective Imagination in Motion, is developed involving several purpose-built multisensory body metaphors prompts to enable access to dimensions of young children’s affective engagement. This research makes theoretical and methodological contributions. It extends the theoretical notion of ‘affect’ from enactive and grounded cognition perspectives through identifying key interactive processes in young children’s engagement with multisensory action metaphors. In addition, the novel method offers a contribution as a way of ‘looking’ at affect within a group situation from affective-pragmatic and social embodiment perspectives. Finally, the research contributes to embodied learning design frameworks offering a guideline for designers wishing to inform their work from enactive cognition perspective

Bridging the gap between emotion and joint action

Our daily human life is filled with a myriad of joint action moments, be it children playing, adults working together (i.e., team sports), or strangers navigating through a crowd. Joint action brings individuals (and embodiment of their emotions) together, in space and in time. Yet little is known about how individual emotions propagate through embodied presence in a group, and how joint action changes individual emotion. In fact, the multi-agent component is largely missing from neuroscience-based approaches to emotion, and reversely joint action research has not found a way yet to include emotion as one of the key parameters to model socio-motor interaction. In this review, we first identify the gap and then stockpile evidence showing strong entanglement between emotion and acting together from various branches of sciences. We propose an integrative approach to bridge the gap, highlight five research avenues to do so in behavioral neuroscience and digital sciences, and address some of the key challenges in the area faced by modern societies