What does not happen: quantifying embodied engagement using NIMI and self-adaptors

Previous research into the quantification of embodied intellectual and emotional engagement using non-verbal movement parameters has not yielded consistent results across different studies. Our research introduces NIMI (Non-Instrumental Movement Inhibition) as an alternative parameter. We propose that the absence of certain types of possible movements can be a more holistic proxy for cognitive engagement with media (in seated persons) than searching for the presence of other movements. Rather than analyzing total movement as an indicator of engagement, our research team distinguishes between instrumental movements (i.e. physical movement serving a direct purpose in the given situation) and non-instrumental movements, and investigates them in the context of the narrative rhythm of the stimulus. We demonstrate that NIMI occurs by showing viewers’ movement levels entrained (i.e. synchronised) to the repeating narrative rhythm of a timed computer-presented quiz. Finally, we discuss the role of objective metrics of engagement in future context-aware analysis of human behaviour in audience research, interactive media and responsive system and interface design

Feeling the groove: shared time and its meanings for three jazz trios

The notion of groove is fundamental to jazz culture and the term yields a rich set of understandings for jazz musicians. Within the literature, no single perspective on groove exists and many questions remain about the relationship between timing processes, phenomenal experience and musical structures in making sense of groove. In this account, the experience and meaning of groove is theorised as emerging from two forms of sharedness. Firstly, a primary intersubjectivity that arises through the timing behaviours of the players; this could be likened to the 'mutual tuning-in' described in social phenomenology. It is proposed that this tuning-in is accomplished through the mechanism of entrainment. The second form of sharedness is understood as the shared temporal models, the cultural knowledge, that musicians make use of in their playing together. Methodologically, this study makes use of detailed investigation of timing data from live performances by three jazz trios, framed by in-depth, semi-structured interview material and steers a new course between existing ethnographic work on jazz and more psychologically informed studies of timing. The findings of the study point towards significant social and structural effects on the groove between players. The impact of musical role on groove and timing is demonstrated and significant temporal models, whose syntactic relations suggest musical proximity or distance, are shown to have a corresponding effect on timing within the trios. The musician's experience of groove is discussed as it relates to the objective timing data and reveals a complex set of understandings involving temporality, consciousness and communication. In the light of these findings, groove is summarised as the feeling of entrainment, inflected through cultural models and expressed through the cultural norms of jazz

Producing Acoustic-Prosodic Entrainment in a Robotic Learning Companion to Build Learner Rapport

abstract: With advances in automatic speech recognition, spoken dialogue systems are assuming increasingly social roles. There is a growing need for these systems to be socially responsive, capable of building rapport with users. In human-human interactions, rapport is critical to patient-doctor communication, conflict resolution, educational interactions, and social engagement. Rapport between people promotes successful collaboration, motivation, and task success. Dialogue systems which can build rapport with their user may produce similar effects, personalizing interactions to create better outcomes. This dissertation focuses on how dialogue systems can build rapport utilizing acoustic-prosodic entrainment. Acoustic-prosodic entrainment occurs when individuals adapt their acoustic-prosodic features of speech, such as tone of voice or loudness, to one another over the course of a conversation. Correlated with liking and task success, a dialogue system which entrains may enhance rapport. Entrainment, however, is very challenging to model. People entrain on different features in many ways and how to design entrainment to build rapport is unclear. The first goal of this dissertation is to explore how acoustic-prosodic entrainment can be modeled to build rapport. Towards this goal, this work presents a series of studies comparing, evaluating, and iterating on the design of entrainment, motivated and informed by human-human dialogue. These models of entrainment are implemented in the dialogue system of a robotic learning companion. Learning companions are educational agents that engage students socially to increase motivation and facilitate learning. As a learning companion’s ability to be socially responsive increases, so do vital learning outcomes. A second goal of this dissertation is to explore the effects of entrainment on concrete outcomes such as learning in interactions with robotic learning companions. This dissertation results in contributions both technical and theoretical. Technical contributions include a robust and modular dialogue system capable of producing prosodic entrainment and other socially-responsive behavior. One of the first systems of its kind, the results demonstrate that an entraining, social learning companion can positively build rapport and increase learning. This dissertation provides support for exploring phenomena like entrainment to enhance factors such as rapport and learning and provides a platform with which to explore these phenomena in future work.Dissertation/ThesisDoctoral Dissertation Computer Science 201

Pass it on: towards a political economy of propensity

The paper argues that the work of Gabriel Tarde on imitation provides a fertile means of understanding how capitalism is forging a new affective technology which conforms to a logic of propensity rather than to means-end reasoning. This it does by drawing together a biological understanding of semiconscious cognition with various practical geometric arts so as to re-stage the world as a series of susceptible situations which can be ridden rather than rigidly controlled. The paper examines the advent of technologies which attend to the variable geometry of so-called animal spirits in the realm of business and then, using Tarde's work as a springboard, considers some alternative means of understanding imitative rays which have less instrumental undertones. The paper is an illustration of the way in which biology and culture have increasingly become intertwined

Bridging the gap between emotion and joint action

Our daily human life is filled with a myriad of joint action moments, be it children playing, adults working together (i.e., team sports), or strangers navigating through a crowd. Joint action brings individuals (and embodiment of their emotions) together, in space and in time. Yet little is known about how individual emotions propagate through embodied presence in a group, and how joint action changes individual emotion. In fact, the multi-agent component is largely missing from neuroscience-based approaches to emotion, and reversely joint action research has not found a way yet to include emotion as one of the key parameters to model socio-motor interaction. In this review, we first identify the gap and then stockpile evidence showing strong entanglement between emotion and acting together from various branches of sciences. We propose an integrative approach to bridge the gap, highlight five research avenues to do so in behavioral neuroscience and digital sciences, and address some of the key challenges in the area faced by modern societies

Ethical issues in the use of video observations with people with advanced dementia and their caregivers in nursing home environments

The use of video allows researchers to gather rich, evocative and contextualized data, yet it also opens a space fraught with ethical challenges. In this paper, the authors describe their use of video methods in ethnographic case study research examining the experiences of persons with advanced dementia who reside in nursing homes and are nearing the end-of-life. In this research, video is used to help garner a deeper understanding of the person with advanced dementia’s being-in-the-world as well as the embodied workplace practices of care staff. Drawing upon notions of emplacement and embodiment, we unpack ethical issues that arise from conducting research with a vulnerable population within a complex environment.  In addition to discussing general ethical principles such as consent, assent and privacy, we argue that to conduct ethically sound research, the researcher needs a solid understanding of the complex and dynamic nature of the nursing home environment. This is, at once, a communal living setting and a home for the residents, a place of work for a diverse group of care staff, and an organizational structure emplaced in a larger socio-political environment.  We shed light on, and discuss potential solutions to, the challenges and complexities of bringing a video camera into the nursing home space where ethical questions arise in ambiguous situations, where relationships shift and the ethical ground reconfigures over time

The effect of electronically mediated sound on group musical interaction: A case study of the practice and development of the Automatic Writing Circle

The interaction between musicians has been one of the traditional strengths of music: it stretches to include an audience and ritual participants but has its origins in group activity, the interpersonal responses of one musician to another. This thesis examines the way that electronic media have transformed the interactions between musicians, particularly in the context of live performance. A central theme is the way in which mediatisation creates new splits within previously integrated musical situations and also merges differences usually defined by physical boundaries. The theories of Gregory Bateson on schizophrenia and Irving Goffman on Situationism are brought together to create a new understanding of the term "schizophonia". This rehabilitated concept is proposed as the key to a creative exploration of new situations and discontinuities which make up group performance in a mediatised environment. In practical terms the exploration of new musical situations is documented in the following projects: the material created for the group "Automatic Writing Circle" during its evolution over a period of six years (compositions, software, instruments), development of the Ouija Board and accompanying software, composition of the piece Lipsync and the earlier piece I slept by numbers for flute and live electronics

Is Vivaldi smooth and takete? Non-verbal sensory scales for describing music qualities

Studies on the perception of music qualities (such as induced or perceived emotions, performance styles, or timbre nuances) make a large use of verbal descriptors. Although many authors noted that particular music qualities can hardly be described by means of verbal labels, few studies have tried alternatives. This paper aims at exploring the use of non-verbal sensory scales, in order to represent different perceived qualities in Western classical music. Musically trained and untrained listeners were required to listen to six musical excerpts in major key and to evaluate them from a sensorial and semantic point of view (Experiment 1). The same design (Experiment 2) was conducted using musically trained and untrained listeners who were required to listen to six musical excerpts in minor key. The overall findings indicate that subjects\u2019 ratings on non-verbal sensory scales are consistent throughout and the results support the hypothesis that sensory scales can convey some specific sensations that cannot be described verbally, offering interesting insights to deepen our knowledge on the relationship between music and other sensorial experiences. Such research can foster interesting applications in the field of music information retrieval and timbre spaces explorations together with experiments applied to different musical cultures and contexts