Behavioural and electrophysiological indices of error processing and perception in joint musical action

Errors are rare, important events. During joint action, agency of errors helps initiate corrective, adaptive responses. If agency becomes ambiguous, errors may become more difficult to process. Models of joint action posit that internal models are developed for own and others’ actions during joint action. If agency is ambiguous, own and other internal models may overlap, causing confusion as to who produced what action. Further, claiming agency over past actions, whether performed, observed, or neither, may affect neural indices of past performance and past errors. The first experiment investigated how agency ambiguity affects behavioural and neural responses to errors and the role of agency in models of joint action. Results from this experiment showed no effects of the agency manipulation, but revealed differences in how distinct types of errors are processed. Uncorrected errors showed a pattern of performance breakdown and were processed at initiation of the erroneous motor command. Corrected errors showed a pattern of pre-response conflict and were processed after the error onset. The second experiment investigated the role of belief of agency on behavioural and neural indices of the perception of errors in previously performed actions. Results showed participants struggled to identify the performer and errors in past performance, but neural activity suggested familiarity may have aided in processing of previously performed actions. Overall findings suggest the brain supports fluent interpersonal coordination in real-time joint action by employing distinct neural mechanisms to manage different types of errors and that a sense of familiarity may play a role in the processing of previously performed actions

Neural alpha oscillations index the balance between self-other integration and segregation in real-time joint action

Shared knowledge and interpersonal coordination are prerequisites for most forms of social behavior. Influential approaches to joint action have conceptualized these capacities in relation to the separate constructs of co-representation (knowledge) and self-other entrainment (coordination). Here we investigated how brain mechanisms involved in co-representation and entrainment interact to support joint action. To do so, we used a musical joint action paradigm to show that the neural mechanisms underlying co-representation and self-other entrainment are linked via a process – indexed by EEG alpha oscillations – regulating the balance between self-other integration and segregation in real time. Pairs of pianists performed short musical items while action familiarity and interpersonal (behavioral) synchronization accuracy were manipulated in a factorial design. Action familiarity referred to whether or not pianists had rehearsed the musical material performed by the other beforehand. Interpersonal synchronization was manipulated via congruent or incongruent tempo change instructions that biased performance timing towards the impending, new tempo. It was observed that, when pianists were familiar with each other's parts, millisecond variations in interpersonal synchronized behavior were associated with a modulation of alpha power over right centro-parietal scalp regions. Specifically, high behavioral entrainment was associated with self-other integration, as indexed by alpha suppression. Conversely, low behavioral entrainment encouraged reliance on internal knowledge and thus led to self-other segregation, indexed by alpha enhancement. These findings suggest that alpha oscillations index the processing of information about self and other depending on the compatibility of internal knowledge and external (environmental) events at finely resolved timescales

Not all errors are alike : modulation of error-related neural responses in musical joint action

During joint action, the sense of agency enables interaction partners to implement corrective and adaptive behaviour in response to performance errors. When agency becomes ambiguous (e.g. when action similarity encourages perceptual self– other overlap), confusion as to who produced what may disrupt this process. The current experiment investigated how ambiguity of agency affects behavioural and neural responses to errors in a joint action domain where self–other overlap is common: musical duos. Pairs of pianists performed piano pieces in synchrony, playing either the same pitches (ambiguous agency) or different pitches (unambiguous agency) while electroencephalography (EEG) was recorded for each individual. Behavioural and event-related potential results showed no effects of the agency manipulation but revealed differences in how distinct error types are processed. Self-produced ‘wrong note’ errors (substitutions) were left uncorrected, showed post-error slowing and elicited an error-related negativity (ERN) peaking before erroneous keystrokes (pre-ERN). In contrast, self-produced ‘extra note’ errors (additions) exhibited pre-error slowing, error and post-error speeding, were rapidly corrected and elicited the ERN. Other-produced errors evoked a feedback-related negativity but no behavioural effects. Overall findings shed light upon how the nervous system supports fluent interpersonal coordination in real-time joint action by employing distinct mechanisms to manage different types of errors

Timing markers of interaction quality during semi-hocket singing

Music is believed to work as a bio-social tool enabling groups of people to establish joint action and group bonding experiences. However, little is known about the quality of the group members’ interaction needed to bring about these effects. To investigate the role of interaction quality, and its effect on joint action and bonding experience, we asked dyads (two singers) to perform music in medieval “hocket” style, in order to engage their co-regulatory activity. The music contained three relative inter-onset-interval (IOI) classes: quarter note, dotted quarter note and eight note, marking time intervals between successive onsets (generated by both singers). We hypothesized that singers co-regulated their activity by minimizing prediction errors in view of stable IOI-classes. Prediction errors were measured using a dynamic Bayesian inference approach that allows us to identify three different types of error called fluctuation (micro-timing errors measured in milliseconds), narration (omission errors or misattribution of an IOI to a wrong IOI class), and collapse errors (macro-timing errors that cause the breakdown of a performance). These three types of errors were correlated with the singers’ estimated quality of the performance and the experienced sense of joint agency. We let the singers perform either while moving or standing still, under the hypothesis that the moving condition would have reduced timing errors and increased We-agency as opposed to Shared-agency (the former portraying a condition in which the performers blend into one another, the latter portraying a joint, but distinct, control of the performance). The results show that estimated quality correlates with fluctuation and narration errors, while agency correlates (to a lesser degree) with narration errors. Somewhat unexpectedly, there was a minor effect of movement, and it was beneficial only for good performers. Joint agency resulted in a “shared,” rather than a “we,” sense of joint agency. The methodology and findings open up promising avenues for future research on social embodied music interaction

Searching for Roots of Entrainment and Joint Action in Early Musical Interactions

When people play music and dance together, they engage in forms of musical joint action that are often characterized by a shared sense of rhythmic timing and affective state (i.e., temporal and affective entrainment). In order to understand the origins of musical joint action, we propose a model in which entrainment is linked to dual mechanisms (motor resonance and action simulation), which in turn support musical behavior (imitation and complementary joint action). To illustrate this model, we consider two generic forms of joint musical behavior: chorusing and turn-taking. We explore how these common behaviors can be founded on entrainment capacities established early in human development, specifically during musical interactions between infants and their caregivers. If the roots of entrainment are found in early musical interactions which are practiced from childhood into adulthood, then we propose that the rehearsal of advanced musical ensemble skills can be considered to be a refined, mimetic form of temporal and affective entrainment whose evolution begins in infancy

Annual Report Of Research and Creative Productions by Faculty and Staff, January to December, 2013

Annual Report Of Research and Creative Productions by Faculty and Staff from January to December, 2013

Analysis and enhancement of interpersonal coordination using inertial measurement unit solutions

Die heutigen mobilen Kommunikationstechnologien haben den Umfang der verbalen und textbasierten Kommunikation mit anderen Menschen, sozialen Robotern und künstlicher Intelligenz erhöht. Auf der anderen Seite reduzieren diese Technologien die nonverbale und die direkte persönliche Kommunikation, was zu einer gesellschaftlichen Thematik geworden ist, weil die Verringerung der direkten persönlichen Interaktionen eine angemessene Wahrnehmung sozialer und umgebungsbedingter Reizmuster erschweren und die Entwicklung allgemeiner sozialer Fähigkeiten bremsen könnte. Wissenschaftler haben aktuell die Bedeutung nonverbaler zwischenmenschlicher Aktivitäten als soziale Fähigkeiten untersucht, indem sie menschliche Verhaltensmuster in Zusammenhang mit den jeweilgen neurophysiologischen Aktivierungsmustern analzsiert haben. Solche Querschnittsansätze werden auch im Forschungsprojekt der Europäischen Union "Socializing sensori-motor contingencies" (socSMCs) verfolgt, das darauf abzielt, die Leistungsfähigkeit sozialer Roboter zu verbessern und Autismus-Spektrumsstörungen (ASD) adäquat zu behandeln. In diesem Zusammenhang ist die Modellierung und das Benchmarking des Sozialverhaltens gesunder Menschen eine Grundlage für theorieorientierte und experimentelle Studien zum weiterführenden Verständnis und zur Unterstützung interpersoneller Koordination. In diesem Zusammenhang wurden zwei verschiedene empirische Kategorien in Abhängigkeit von der Entfernung der Interagierenden zueinander vorgeschlagen: distale vs. proximale Interaktionssettings, da sich die Struktur der beteiligten kognitiven Systeme zwischen den Kategorien ändert und sich die Ebene der erwachsenden socSMCs verschiebt. Da diese Dissertation im Rahmen des socSMCs-Projekts entstanden ist, wurden Interaktionssettings für beide Kategorien (distal und proximal) entwickelt. Zudem wurden Ein-Sensor-Lösungen zur Reduzierung des Messaufwands (und auch der Kosten) entwickelt, um eine Messung ausgesuchter Verhaltensparameter bei einer Vielzahl von Menschen und sozialen Interaktionen zu ermöglichen. Zunächst wurden Algorithmen für eine kopfgetragene Trägheitsmesseinheit (H-IMU) zur Messung der menschlichen Kinematik als eine Ein-Sensor-Lösung entwickelt. Die Ergebnisse bestätigten, dass die H-IMU die eigenen Gangparameter unabhängig voneinander allein auf Basis der Kopfkinematik messen kann. Zweitens wurden—als ein distales socSMC-Setting—die interpersonellen Kopplungen mit einem Bezug auf drei interagierende Merkmale von „Übereinstimmung“ (engl.: rapport) behandelt: Positivität, gegenseitige Aufmerksamkeit und Koordination. Die H-IMUs überwachten bestimmte soziale Verhaltensereignisse, die sich auf die Kinematik der Kopforientierung und Oszillation während des Gehens und Sprechens stützen, so dass der Grad der Übereinstimmung geschätzt werden konnte. Schließlich belegten die Ergebnisse einer experimentellen Studie, die zu einer kollaborativen Aufgabe mit der entwickelten IMU-basierten Tablet-Anwendung durchgeführt wurde, unterschiedliche Wirkungen verschiedener audio-motorischer Feedbackformen für eine Unterstützung der interpersonellen Koordination in der Kategorie proximaler sensomotorischer Kontingenzen. Diese Dissertation hat einen intensiven interdisziplinären Charakter: Technologische Anforderungen in den Bereichen der Sensortechnologie und der Softwareentwicklung mussten in direktem Bezug auf vordefinierte verhaltenswissenschaftliche Fragestellungen entwickelt und angewendet bzw. gelöst werden—und dies in zwei unterschiedlichen Domänen (distal, proximal). Der gegebene Bezugsrahmen wurde als eine große Herausforderung bei der Entwicklung der beschriebenen Methoden und Settings wahrgenommen. Die vorgeschlagenen IMU-basierten Lösungen könnten dank der weit verbreiteten IMU-basierten mobilen Geräte zukünftig in verschiedene Anwendungen perspektiv reich integriert werden.Today’s mobile communication technologies have increased verbal and text-based communication with other humans, social robots and intelligent virtual assistants. On the other hand, the technologies reduce face-to-face communication. This social issue is critical because decreasing direct interactions may cause difficulty in reading social and environmental cues, thereby impeding the development of overall social skills. Recently, scientists have studied the importance of nonverbal interpersonal activities to social skills, by measuring human behavioral and neurophysiological patterns. These interdisciplinary approaches are in line with the European Union research project, “Socializing sensorimotor contingencies” (socSMCs), which aims to improve the capability of social robots and properly deal with autism spectrum disorder (ASD). Therefore, modelling and benchmarking healthy humans’ social behavior are fundamental to establish a foundation for research on emergence and enhancement of interpersonal coordination. In this research project, two different experimental settings were categorized depending on interactants’ distance: distal and proximal settings, where the structure of engaged cognitive systems changes, and the level of socSMCs differs. As a part of the project, this dissertation work referred to this spatial framework. Additionally, single-sensor solutions were developed to reduce costs and efforts in measuring human behaviors, recognizing the social behaviors, and enhancing interpersonal coordination. First of all, algorithms using a head worn inertial measurement unit (H-IMU) were developed to measure human kinematics, as a baseline for social behaviors. The results confirmed that the H-IMU can measure individual gait parameters by analyzing only head kinematics. Secondly, as a distal sensorimotor contingency, interpersonal relationship was considered with respect to a dynamic structure of three interacting components: positivity, mutual attentiveness, and coordination. The H-IMUs monitored the social behavioral events relying on kinematics of the head orientation and oscillation during walk and talk, which can contribute to estimate the level of rapport. Finally, in a new collaborative task with the proposed IMU-based tablet application, results verified effects of different auditory-motor feedbacks on the enhancement of interpersonal coordination in a proximal setting. This dissertation has an intensive interdisciplinary character: Technological development, in the areas of sensor and software engineering, was required to apply to or solve issues in direct relation to predefined behavioral scientific questions in two different settings (distal and proximal). The given frame served as a reference in the development of the methods and settings in this dissertation. The proposed IMU-based solutions are also promising for various future applications due to widespread wearable devices with IMUs.European Commission/HORIZON2020-FETPROACT-2014/641321/E

The effects of duet reading on the reading fluency and comprehension of high school students with learning disabilities

The purpose of this study was to examine the effectiveness of Duet Reading, a peer-mediated repeated reading strategy, for increasing the reading fluency of four diverse ninth and tenth grade students with learning disabilities in a large public high school. The effects of Duet Reading on comprehension were explored as well. A multiple probe single-subject design across two groups of students was used to evaluate the effectiveness of the intervention. The students were placed in dyads based on similar fluency training levels and the intervention was administered in a staggered fashion according to the dictates of the multiple probe design. Dependent measures included the number of words correct per minute and errors per minute along with pre-post data on the Test of Oral Reading Fluency (TORF) and the comprehension subtest of the Woodcock Reading Mastery Test. Social validity data were also collected. Results of the study revealed that no functional relationship existed between the Duet Reading intervention and the students' oral reading fluency. However, two of the students' oral reading fluency rates increased, with modest gains in comprehension. Despite limited results, overall, the students felt that the intervention was beneficial and helped them become better readers. Results are discussed in terms of previous research, limitations, as well as implications for current practice

The Phenomenology of Joint Action: Self-Agency vs. Joint-Agency

This chapter aims at investigating the phenomenology of joint action and at gaining a better understanding of (1) how the sense of agency one experiences when engaged in a joint action differs from the sense of agency one has for individual actions and (2) how the sense of agency one experiences when engaged in a joint action differs according to the type of joint action and to the role one plays in it