Collaboration in Augmented Reality: How to establish coordination and joint attention?

Schnier C, Pitsch K, Dierker A, Hermann T. Collaboration in Augmented Reality: How to establish coordination and joint attention? In: Boedker S, Bouvin NO, Lutters W, Wulf V, Ciolfi L, eds. Proceedings of the 12th European Conference on Computer Supported Cooperative Work (ECSCW 2011). Springer-Verlag London; 2011: 405-416.We present an initial investigation from a semi-experimental setting, in which an HMD-based AR-system has been used for real-time collaboration in a task-oriented scenario (design of a museum exhibition). Analysis points out the specific conditions of interacting in an AR environment and focuses on one particular practical problem for the participants in coordinating their interaction: how to establish joint attention towards the same object or referent. Analysis allows insights into how the pair of users begins to familarize with the environment, the limitations and opportunities of the setting and how they establish new routines for e.g. solving the ʻjoint attentionʼ-problem

Object Handovers: a Review for Robotics

This article surveys the literature on human-robot object handovers. A handover is a collaborative joint action where an agent, the giver, gives an object to another agent, the receiver. The physical exchange starts when the receiver first contacts the object held by the giver and ends when the giver fully releases the object to the receiver. However, important cognitive and physical processes begin before the physical exchange, including initiating implicit agreement with respect to the location and timing of the exchange. From this perspective, we structure our review into the two main phases delimited by the aforementioned events: 1) a pre-handover phase, and 2) the physical exchange. We focus our analysis on the two actors (giver and receiver) and report the state of the art of robotic givers (robot-to-human handovers) and the robotic receivers (human-to-robot handovers). We report a comprehensive list of qualitative and quantitative metrics commonly used to assess the interaction. While focusing our review on the cognitive level (e.g., prediction, perception, motion planning, learning) and the physical level (e.g., motion, grasping, grip release) of the handover, we briefly discuss also the concepts of safety, social context, and ergonomics. We compare the behaviours displayed during human-to-human handovers to the state of the art of robotic assistants, and identify the major areas of improvement for robotic assistants to reach performance comparable to human interactions. Finally, we propose a minimal set of metrics that should be used in order to enable a fair comparison among the approaches.Comment: Review paper, 19 page

Image-Enabled Discourse: Investigating the Creation of Visual Information as Communicative Practice

Anyone who has clarified a thought or prompted a response during a conversation by drawing a picture has exploited the potential of image making as an interactive tool for conveying information. Images are increasingly ubiquitous in daily communication, in large part due to advances in visually enabled information and communication technologies (ICT), such as information visualization applications, image retrieval systems and visually enabled collaborative work tools. Human abilities to use images to communicate are however far more sophisticated and nuanced than these technologies currently support. In order to learn more about the practice of image making as a specialized form of information and communication behavior, this study examined face-to-face conversations involving the creation of ad hoc visualizations (i.e., napkin drawings ). A model of image-enabled discourse is introduced, which positions image making as a specialized form of communicative practice. Multimodal analysis of video-recorded conversations focused on identifying image-enabled communicative activities in terms of interactional sociolinguistic concepts of conversational involvement and coordination, specifically framing, footing and stance. The study shows that when drawing occurs in the context of an ongoing dialogue, the activity of visual representation performs key communicative tasks. Visualization is a form of social interaction that contributes to the maintenance of conversational involvement in ways that are not often evident in the image artifact. For example, drawing enables us to coordinate with each other, to introduce alternative perspectives into a conversation and even to temporarily suspend the primary thread of a discussion in order to explore a tangential thought. The study compares attributes of the image artifact with those of the activity of image making, described as a series of contrasting affordances. Visual information in complex systems is generally represented and managed based on the affordances of the artifact, neglecting to account for all that is communicated through the situated action of creating. These finding have heuristic and best-practice implications for a range of areas related to the design and evaluation of virtual collaboration environments, visual information extraction and retrieval systems, and data visualization tools

"It's cleaner, definitely": Collaborative Process in Audio Production.

Working from vague client instructions, how do audio producers collaborate to diagnose what specifically is wrong with a piece of music, where the problem is and what to do about it? This paper presents a design ethnography that uncovers some of the ways in which two music producers co-ordinate their understanding of complex representations of pieces of music while working together in a studio. Our analysis shows that audio producers constantly make judgements based on audio and visual evidence while working with complex digital tools, which can lead to ambiguity in assessments of issues. We show how multimodal conduct guides the process of work and that complex media objects are integrated as elements of interaction by the music producers. The findings provide an understanding how people currently collaborate when producing audio, to support the design of better tools and systems for collaborative audio production in the future

Situated and distributed cognition in artifact negotiation and trade-specific skills: A cognitive ethnography of Kashmiri carpet weaving practice

This article describes various ways actors in Kashmiri carpet weaving practice deploy a range of artifacts, from symbolic, to material, to hybrid, in order to achieve diverse cognitive accomplishments in their particular task domains: information representation, inter and intra-domain communication, distribution of cognitive labor across people and time, coordination of team activities, and carrying of cultural heritage. In this repertoire, some artifacts position themselves as naïve tools in the actors’ environment to the point of being ignored; however, their usage-in-context unfolds their cognitive involvement in the tasks. These usages-in-context are shown through artifact analysis of their routine, improvised, and opportunistic uses, where cognitive artifacts like talim—the central artifact of this practice—are shown to play not only multifunctional roles beyond representation, but are also complemented by trade-specific skills bearing strong cognitive implications in a task

Moving together: the organisation of non-verbal cues during multiparty conversation

PhDConversation is a collaborative activity. In face-to-face interactions interlocutors have mutual access to a shared space. This thesis aims to explore the shared space as a resource for coordinating conversation. As well demonstrated in studies of two-person conversations, interlocutors can coordinate their speech and non-verbal behaviour in ways that manage the unfolding conversation. However, when scaling up from two people to three people interacting, the coordination challenges that the interlocutors face increase. In particular speakers must manage multiple listeners. This thesis examines the use of interlocutors’ bodies in shared space to coordinate their multiparty dialogue. The approach exploits corpora of motion captured triadic interactions. The thesis first explores how interlocutors coordinate their speech and non-verbal behaviour. Inter-person relationships are examined and compared with artificially created triples who did not interact. Results demonstrate that interlocutors avoid speaking and gesturing over each other, but tend to nod together. Evidence is presented that the two recipients of an utterance have different patterns of head and hand movement, and that some of the regularities of movement are correlated with the task structure. The empirical section concludes by uncovering a class of coordination events, termed simultaneous engagement events, that are unique to multiparty dialogue. They are constructed using combinations of speaker head orientation and gesture orientation. The events coordinate multiple recipients of the dialogue and potentially arise as a result of the greater coordination challenges that interlocutors face. They are marked in requiring a mutually accessible shared space in order to be considered an effective interactional cue. The thesis provides quantitative evidence that interlocutors’ head and hand movements are organised by their dialogue state and the task responsibilities that the bear. It is argued that a shared interaction space becomes a more important interactional resource when conversations scale up to three people

Bridging the gap between emotion and joint action

Our daily human life is filled with a myriad of joint action moments, be it children playing, adults working together (i.e., team sports), or strangers navigating through a crowd. Joint action brings individuals (and embodiment of their emotions) together, in space and in time. Yet little is known about how individual emotions propagate through embodied presence in a group, and how joint action changes individual emotion. In fact, the multi-agent component is largely missing from neuroscience-based approaches to emotion, and reversely joint action research has not found a way yet to include emotion as one of the key parameters to model socio-motor interaction. In this review, we first identify the gap and then stockpile evidence showing strong entanglement between emotion and acting together from various branches of sciences. We propose an integrative approach to bridge the gap, highlight five research avenues to do so in behavioral neuroscience and digital sciences, and address some of the key challenges in the area faced by modern societies

Conversational Movement Dynamics and Nonverbal Indicators of Second Language Development: A Microgenetic Approach

This dissertation study extends on current understandings of gesture and embodied interaction with the eco-social environment in second language development (SLD) while introducing new aspects of movement analysis through dynamical modeling. To understand the role of embodiment during learning activities, a second language learning task has been selected. Dyads consisting of a non-native English-speaking student and a native English-speaking tutor were video recorded during writing consultations centered on class assignments provided by the student. Cross-recurrence quantification analysis was used to measure interactional movement synchrony between the members of each dyad. Results indicate that students with varied English proficiency levels synchronize movements with their tutors over brief, frequent periods of time. Synchronous movement pattern complexity is highly variable across and within the dyads. Additionally, co-speech gesture and gesture independent of speech were analyzed qualitatively to identify the role of gesture as related to SLD events. A range of movement types were used during developmental events by the students and tutors to interact with their partner. The results indicated that language development occurs within a movement rich context through negotiated interaction which depends on a combination of synchronized and synergistic movements. Synchronized movements exhibited complex, dynamical behaviors including variability, self-organization, and emergent properties. Synergistic movement emergence revealed how the dualistic presence of the self/other in each dyad creates a functioning intersubjective space. Overall, the dyads demonstrated that movement is a salient factor in the writing consultation activity