57,906 research outputs found
Moving together: the organisation of non-verbal cues during multiparty conversation
PhDConversation is a collaborative activity. In face-to-face interactions interlocutors have mutual
access to a shared space. This thesis aims to explore the shared space as a resource for coordinating
conversation. As well demonstrated in studies of two-person conversations, interlocutors
can coordinate their speech and non-verbal behaviour in ways that manage the unfolding conversation.
However, when scaling up from two people to three people interacting, the coordination
challenges that the interlocutors face increase. In particular speakers must manage multiple listeners.
This thesis examines the use of interlocutors’ bodies in shared space to coordinate their
multiparty dialogue.
The approach exploits corpora of motion captured triadic interactions. The thesis first explores
how interlocutors coordinate their speech and non-verbal behaviour. Inter-person relationships
are examined and compared with artificially created triples who did not interact. Results demonstrate
that interlocutors avoid speaking and gesturing over each other, but tend to nod together.
Evidence is presented that the two recipients of an utterance have different patterns of head and
hand movement, and that some of the regularities of movement are correlated with the task structure.
The empirical section concludes by uncovering a class of coordination events, termed simultaneous
engagement events, that are unique to multiparty dialogue. They are constructed using
combinations of speaker head orientation and gesture orientation. The events coordinate multiple
recipients of the dialogue and potentially arise as a result of the greater coordination challenges
that interlocutors face. They are marked in requiring a mutually accessible shared space in order
to be considered an effective interactional cue.
The thesis provides quantitative evidence that interlocutors’ head and hand movements are
organised by their dialogue state and the task responsibilities that the bear. It is argued that a
shared interaction space becomes a more important interactional resource when conversations
scale up to three people
Speakers Raise their Hands and Head during Self-Repairs in Dyadic Conversations
People often encounter difficulties in building shared understanding during everyday conversation. The most common symptom of these difficulties are self-repairs, when a speaker restarts, edits or amends their utterances mid-turn. Previous work has focused on the verbal signals of self-repair, i.e. speech disfluences (filled pauses, truncated words and phrases, word substitutions or reformulations), and computational tools now exist that can automatically detect these verbal phenomena. However, face-to-face conversation also exploits rich non-verbal resources and previous research suggests that self-repairs are associated with distinct hand movement patterns. This paper extends those results by exploring head and hand movements of both speakers and listeners using two motion parameters: height (vertical position) and 3D velocity. The results show that speech sequences containing self-repairs are distinguishable from fluent ones: speakers raise their hands and head more (and move more rapidly) during self-repairs. We obtain these results by analysing data from a corpus of 13 unscripted dialogues, and we discuss how these findings could support the creation of improved cognitive artificial systems for natural human-machine and human-robot interaction
Measuring, analysing and artificially generating head nodding signals in dyadic social interaction
Social interaction involves rich and complex behaviours where verbal and non-verbal signals are exchanged in dynamic patterns. The aim of this thesis is to explore new ways of measuring and analysing interpersonal coordination as it naturally occurs in social interactions. Specifically, we want to understand what different types of head nods mean in different social contexts, how they are used during face-to-face dyadic conversation, and if they relate to memory and learning. Many current methods are limited by time-consuming and low-resolution data, which cannot capture the full richness of a dyadic social interaction. This thesis explores ways to demonstrate how high-resolution data in this area can give new insights into the study of social interaction. Furthermore, we also want to demonstrate the benefit of using virtual reality to artificially generate interpersonal coordination to test our hypotheses about the meaning of head nodding as a communicative signal.
The first study aims to capture two patterns of head nodding signals – fast nods and slow nods – and determine what they mean and how they are used across different conversational contexts. We find that fast nodding signals receiving new information and has a different meaning than slow nods. The second study aims to investigate a link between memory and head nodding behaviour. This exploratory study provided initial hints that there might be a relationship, though further analyses were less clear. In the third study, we aim to test if interactive head nodding in virtual agents can be used to measure how much we like the virtual agent, and whether we learn better from virtual agents that we like. We find no causal link between memory performance and interactivity. In the fourth study, we perform a cross-experimental analysis of how the level of interactivity in different contexts (i.e., real, virtual, and video), impacts on memory and find clear differences between them
Eyebrow raising in dialogue: discourse structure, utterance function, and pitch accents
Some studies have suggested a relationship between eyebrow raising and different aspects of the verbal message, but our knowledge about this link is still
very limited. If we could establish and characterise a relation between eyebrow
raises and the linguistic signal we could better understand human multimodal
communication behaviour. We could also improve the credibility and efficiency
of computer animated conversational agents in multimodal communication systems.This thesis investigated eyebrow raising in a corpus of task-oriented English
dialogues. Applying a standard dialogue coding scheme (Conversational Game
Analysis, Carletta et al., 1997), eyebrow raises were studied in connection with
discourse structure and utterance function. Supporting the prediction, more
frequent and longer eyebrow raising occurred in the initial utterance of highlevel discourse segments than anywhere else in the dialogue (where 'high-level
discourse segment' = transaction, and 'utterance' = move, following Carletta et al.).
Additionally, eyebrow raises were more frequent in instructions than in requests
for or acknowledgements of information. Instructions also had longer eyebrow
raising than any other type of utterance. Contrary to the prediction, the start of a
lower-level discourse segment (conversational game) did not have more eyebrow
raising than any other position in the dialogue, and queries did not have more
eyebrow raising than any other type of utterance.Eyebrow raises were also studied in relation to intonational events, namely pitch
accents. Results showed evidence of alignment between the brow raise start and
the start of a pitch accent. Most pitch accents were not associated with brow raising, but when brow raises occurred they tended to immediately precede a pitch
accent on the speech signal. To investigate what could explain the alignment
between the two events, pitch accents aligned with eyebrow raises were compared to all other pitch accents in terms of: phonological characteristics (primary
vs. secondary pitch accents, and downstep-initial vs. non-initial pitch accents), information structure (given vs. new information in referring expressions, and the
last quarter vs. earlier parts of the utterance length) and type of utterance in which
they occurred (instruction vs. non-instruction). Those comparisons suggested that
brow raises may be aligned more frequently with pitch accents in downstepinitial position and in instructions. No differences were found in terms of information structure or between primary/secondary accents.The results provide evidence of a link between eyebrow raising and spoken language. Eyebrow raises may signal the start of linguistic units such as discourse
segments and some prosodic phenomena, they may be related to utterance function, and they are aligned with pitch accents. Possible linguistic functions are
proposed, such as structuring and emphasising information in the verbal message
Are You on My Wavelength? Interpersonal Coordination in Dyadic Conversations
Conversation between two people involves subtle nonverbal coordination in addition to speech. However, the precise parameters and timing of this coordination remain unclear, which limits our ability to theorize about the neural and cognitive mechanisms of social coordination. In particular, it is unclear if conversation is dominated by synchronization (with no time lag), rapid and reactive mimicry (with lags under 1 s) or traditionally observed mimicry (with several seconds lag), each of which demands a different neural mechanism. Here we describe data from high-resolution motion capture of the head movements of pairs of participants (n = 31 dyads) engaged in structured conversations. In a pre-registered analysis pathway, we calculated the wavelet coherence of head motion within dyads as a measure of their nonverbal coordination and report two novel results. First, low-frequency coherence (0.2–1.1 Hz) is consistent with traditional observations of mimicry, and modeling shows this behavior is generated by a mechanism with a constant 600 ms lag between leader and follower. This is in line with rapid reactive (rather than predictive or memory-driven) models of mimicry behavior, and could be implemented in mirror neuron systems. Second, we find an unexpected pattern of lower-than-chance coherence between participants, or hypo-coherence, at high frequencies (2.6–6.5 Hz). Exploratory analyses show that this systematic decoupling is driven by fast nodding from the listening member of the dyad, and may be a newly identified social signal. These results provide a step towards the quantification of real-world human behavior in high resolution and provide new insights into the mechanisms of social coordination
Directional adposition use in English, Swedish and Finnish
Directional adpositions such as to the left of describe where a Figure is in relation to a Ground. English and Swedish directional adpositions refer to the location of a Figure in relation to a Ground, whether both are static or in motion. In contrast, the Finnish directional adpositions edellä (in front of) and jäljessä (behind) solely describe the location of a moving Figure in relation to a moving Ground (Nikanne, 2003).
When using directional adpositions, a frame of reference must be assumed for interpreting the meaning of directional adpositions. For example, the meaning of to the left of in English can be based on a relative (speaker or listener based) reference frame or an intrinsic (object based) reference frame (Levinson, 1996). When a Figure and a Ground are both in motion, it is possible for a Figure to be described as being behind or in front of the Ground, even if neither have intrinsic features. As shown by Walker (in preparation), there are good reasons to assume that in the latter case a motion based reference frame is involved. This means that if Finnish speakers would use edellä (in front of) and jäljessä (behind) more frequently in situations where both the Figure and Ground are in motion, a difference in reference frame use between Finnish on one hand and English and Swedish on the other could be expected.
We asked native English, Swedish and Finnish speakers’ to select adpositions from a language specific list to describe the location of a Figure relative to a Ground when both were shown to be moving on a computer screen. We were interested in any differences between Finnish, English and Swedish speakers.
All languages showed a predominant use of directional spatial adpositions referring to the lexical concepts TO THE LEFT OF, TO THE RIGHT OF, ABOVE and BELOW. There were no differences between the languages in directional adpositions use or reference frame use, including reference frame use based on motion.
We conclude that despite differences in the grammars of the languages involved, and potential differences in reference frame system use, the three languages investigated encode Figure location in relation to Ground location in a similar way when both are in motion.
Levinson, S. C. (1996). Frames of reference and Molyneux’s question: Crosslingiuistic evidence. In P. Bloom, M.A. Peterson, L. Nadel & M.F. Garrett (Eds.) Language and Space (pp.109-170). Massachusetts: MIT Press.
Nikanne, U. (2003). How Finnish postpositions see the axis system. In E. van der Zee & J. Slack (Eds.), Representing direction in language and space. Oxford, UK: Oxford University Press.
Walker, C. (in preparation). Motion encoding in language, the use of spatial locatives in a motion context. Unpublished doctoral dissertation, University of Lincoln, Lincoln. United Kingdo
- …