Implementing process methods in learning research:targeting emotional responses in collaborative learning

Abstract. Context and aim: While interacting with peers and teachers in a collaborative learning task, students experience socio-emotional challenges and display emotional responses. These responses have two major components: arousal and valence, which influence the learning process and its outcomes. The aim of the study was twofold: first, to explore how group members’ arousal levels vary across different phases of a collaborative learning task; and second, to investigate how case students’ emotional responses are distributed in the arousal-valence space across the phases of the collaborative task. Methods: Twelve 6th graders from a school of Finland participated in a collaborative task, in groups of three students. The task was to build an energy efficient house in three distinct phases: brainstorming, planning, and building. While performing the activity, students wore Empatica E4 wristbands to measure their electrodermal activity (EDA) and were video-recorded with 360° cameras. Arousal levels were calculated in peaks per min (ppm) and classified as low, middle, and high. Emotional valence was classified from video analysis into positive, neutral, and negative. Results: The ranges for arousal levels were established between 26 and 88 ppm. Only two students displayed the same arousal level across the three phases of the experiment. Three students displayed higher arousal at first and then fell in to lower levels. Four students had the opposite experience and three students did not display a pattern. As for the case students, the student leading a poorly collaborating group experienced oscillating levels of arousal, from middle to high, and displayed a mix of negative and positive valence most of the time. The student loafing around experienced all arousal levels and positive valence most of the time. Overall conclusions and relevance: The study allowed to establish measurement thresholds for arousal as a starting point for future studies in collaborative learning and the arousal-valence space provided a quantifiable picture to help teachers understand the importance of emotional responses in classroom during collaborative learning

Complexity Science in Human Change

This reprint encompasses fourteen contributions that offer avenues towards a better understanding of complex systems in human behavior. The phenomena studied here are generally pattern formation processes that originate in social interaction and psychotherapy. Several accounts are also given of the coordination in body movements and in physiological, neuronal and linguistic processes. A common denominator of such pattern formation is that complexity and entropy of the respective systems become reduced spontaneously, which is the hallmark of self-organization. The various methodological approaches of how to model such processes are presented in some detail. Results from the various methods are systematically compared and discussed. Among these approaches are algorithms for the quantification of synchrony by cross-correlational statistics, surrogate control procedures, recurrence mapping and network models.This volume offers an informative and sophisticated resource for scholars of human change, and as well for students at advanced levels, from graduate to post-doctoral. The reprint is multidisciplinary in nature, binding together the fields of medicine, psychology, physics, and neuroscience

Social and Affective Neuroscience of Everyday Human Interaction

This Open Access book presents the current state of the art knowledge on social and affective neuroscience based on empirical findings. This volume is divided into several sections first guiding the reader through important theoretical topics within affective neuroscience, social neuroscience and moral emotions, and clinical neuroscience. Each chapter addresses everyday social interactions and various aspects of social interactions from a different angle taking the reader on a diverse journey. The last section of the book is of methodological nature. Basic information is presented for the reader to learn about common methodologies used in neuroscience alongside advanced input to deepen the understanding and usability of these methods in social and affective neuroscience for more experienced readers

Neural and Cognitive Mechanisms of Real-World Interaction during Adult Learning

The goal of this thesis is to understand what makes a social interaction successful, and whether it supports learning of conceptual knowledge. Crucially, it distinguishes learning via the social from learning about the social, and asks the question of how social interaction supports declarative processing of non-social material. In doing so, it priorities ecological validity: all experiments involve relatively unconstrained teacher-learner interaction, and learning material resembled documentary-like content. The first half of the thesis shows a series of studies on how adults learn in online contexts (Study 1 and 2): Study 1 presents two online experiments, where social contingency (i.e. being part of a live interaction vs observing a pre-recorded one) and social cues (i.e. teacher’s webcam on vs off vs showing a slide only) were manipulated. Results showed that learning in live interaction was associated with the best performance, and live social interaction with a full view of the teacher provided the optimal setting for learning, while seeing a slide had greater benefit during recorded sessions specifically. Study 2 replicates the live-learning advantage across three experiments and a large sample of adults with Autistic Spectrum Condition (ASC). The second half of this thesis (Study 3 and 4) investigates face-to-face interaction, using functional Near-Infrared Spectroscopy (fNIRS) hyperscanning and wavelet transform coherence (WTC) analysis, to measure brain synchrony in naturalistic interactions. Study 3 tests the hypothesis that being in the same room and engaging in conversation affects people’s brain response to later novel stimuli. Study 4 asks whether teacher-student brain synchrony can be a marker of learning success and, if so, how it is modulated by social behaviours. Findings reveal a complex dynamic between neural responses and behavioural metrics, in particular mutual gaze and joint attention. Results are discussed in the frame of the mutual-prediction hypothesis, and advocate for a multi-modal investigation of social learning to fully understand its underlying cognitive mechanisms. Overall, this work advances the current understanding of naturalistic social interaction and has theoretical implications for cognitive models of information exchange and mutual prediction, as well as practical significance for educational policies. The novel multi-modal and highly ecological approach used in this thesis makes this work an important example for real-world second person social neuroscience

Big data analysis of cyclic alternating pattern during sleep using deep learning

Sleep scoring has been of great interest since the invention of the polysomnography method, which enabled the recording of physiological signals overnight. With the surge in wearable devices in recent years, the topic of what is high-quality sleep, how can it be determined and how can it be achieved attracted increasing interest. In the last two decades, cyclic alternating pattern (CAP) was introduced as a scoring alternative to traditional sleep staging. CAP is known as a synonym for sleep microstructure and describes sleep instability. Manual CAP scoring performed by sleep experts is a very exhausting and time-consuming task. Hence, an automatic method would facilitate the processing of sleep data and provide a valuable tool to enhance the understanding of the role of CAP. This thesis aims to expand the knowledge about CAP by developing a high-performance automated CAP scoring system that can reliably detect and classify CAP events in sleep recordings. The automated system is equipped with state-of-the-art signal processing methods and exploits the dynamic, temporal information in brain activity using deep learning. The automated scoring system is validated using large community-based cohort studies and comparing the output to verified values in the literature. Our findings present novel clinical results on the relationship between CAP and age, gender, subjective sleep quality, and sleep disorders demonstrating that automated CAP analysis of large population based studies can lead to new findings on CAP and its subcomponents. Next, we study the relationship between CAP and behavioural, cognitive, and quality-of-life measures and the effect of adenotonsillectomy on CAP in children with obstructive sleep apnoea as the link between CAP and cognitive functioning in children is largely unknown. Finally, we investigate cortical-cardiovascular interactions during CAP to gain novel insights into the causal relationships between cortical and cardiovascular activity that are underpinning the microstructure of sleep. In summary, the research outcomes in this thesis outline the importance of a fully automated end-to-end CAP scoring solution for future studies on sleep microstructure. Furthermore, we present novel critical information for a better understanding of CAP and obtain first evidence on physiological network dynamics between the central nervous system and the cardiovascular system during CAP.Thesis (Ph.D.) -- University of Adelaide, School of Electrical and Electronic Engineering, 202

Social and Affective Neuroscience of Everyday Human Interaction

This Open Access book presents the current state of the art knowledge on social and affective neuroscience based on empirical findings. This volume is divided into several sections first guiding the reader through important theoretical topics within affective neuroscience, social neuroscience and moral emotions, and clinical neuroscience. Each chapter addresses everyday social interactions and various aspects of social interactions from a different angle taking the reader on a diverse journey. The last section of the book is of methodological nature. Basic information is presented for the reader to learn about common methodologies used in neuroscience alongside advanced input to deepen the understanding and usability of these methods in social and affective neuroscience for more experienced readers

Instrumenting the Musician: Measuring and Enhancing A ective and Behavioural Interaction During Collaborative Music Making

Modern sensor technologies facilitate the measurement and interpretation of human affective and behavioural signals, and have consequently become widely used tools in the fields of affective computing, social signal processing and psychophysiology. This thesis investigates the use and development of these tools for measuring and enhancing aff ective and behavioural interaction during collaborative music making. Drawing upon work in the aforementioned fields, an exploratory study is designed, where self-report and continuous behavioural and physiological measures are collected from pairs of improvising percussionists. The findings lead to the selection of gaze, motion, and cardiac activity as input measures in the design of a device to enhance affective and behavioural interaction between co-present musicians. The device provides musicians with real-time visual feedback on the glances or body motions of their co-performers, whilst also recording cardiac activity as a potential measure of musical decision making processes. Quantitative evidence is found for the effects of this device on the communicative behaviours of collaborating musicians during an experiment designed to test the device in a controlled environment. This study also reports findings on discrete and time series relationships between cardiac activity and musical decision-making. A further, qualitative study is designed to evaluate the appropriation and impact of the device during long-term use in naturalistic settings. The results provide insights into earlier findings and contribute towards an empirical understanding of affective and behavioural interaction during collaborative music making, as well as implications for the design and deployment of sensor-based technologies to enhance such interactions. This thesis advances the dominant single-user paradigm within human-computer interaction and affective computing research, towards multi-user scenarios, where the concern is human-human interaction. It achieves this by focusing on the emotionally rich, and under-studied context of co-present musical collaboration; contributing new methods and findings that pave the way for further research and real-world applications.This work was funded by the Engineering and Physical Sciences Research Council (EPSRC) as part of the Centre for Doctoral Training in Media and Arts Technology at Queen Mary University of London (ref: EP/G03723X/1)

Connecting people through physiosocial technology

Social connectedness is one of the most important predictors of health and well-being. The goal of this dissertation is to investigate technologies that can support social connectedness. Such technologies can build upon the notion that disclosing emotional information has a strong positive influence on social connectedness. As physiological signals are strongly related to emotions, they might provide a solid base for emotion communication technologies. Moreover, physiological signals are largely lacking in unmediated communication, have been used successfully by machines to recognize emotions, and can be measured relatively unobtrusively with wearable sensors. Therefore, this doctoral dissertation examines the following research question: How can we use physiological signals in affective technology to improve social connectedness? First, a series of experiments was conducted to investigate if computer interpretations of physiological signals can be used to automatically communicate emotions and improve social connectedness (Chapters 2 and 3). The results of these experiments showed that computers can be more accurate at recognizing emotions than humans are. Physiological signals turned out to be the most effective information source for machine emotion recognition. One advantage of machine based emotion recognition for communication technology may be the increase in the rate at which emotions can be communicated. As expected, experiments showed that increases in the number of communicated emotions increased feelings of closeness between interacting people. Nonetheless, these effects on feelings of closeness are limited if users attribute the cause of the increases in communicated emotions to the technology and not to their interaction partner. Therefore, I discuss several possibilities to incorporate emotion recognition technologies in applications in such a way that users attribute the communication to their interaction partner. Instead of using machines to interpret physiological signals, the signals can also be represented to a user directly. This way, the interpretation of the signal is left to be done by the user. To explore this, I conducted several studies that employed heartbeat representations as a direct physiological communication signal. These studies showed that people can interpret such signals in terms of emotions (Chapter 4) and that perceiving someone's heartbeat increases feelings of closeness between the perceiver and sender of the signal (Chapter 5). Finally, we used a field study (Chapter 6) to investigate the potential of heartbeat communication mechanisms in practice. This again confirmed that heartbeat can provide an intimate connection to another person, showing the potential for communicating physiological signals directly to improve connectedness. The last part of the dissertation builds upon the notion that empathy has positive influences on social connectedness. Therefore, I developed a framework for empathic computing that employed automated empathy measurement based on physiological signals (Chapter 7). This framework was applied in a system that can train empathy (Chapter 8). The results showed that providing users frequent feedback about their physiological synchronization with others can help them to improve empathy as measured through self-report and physiological synchronization. In turn, this improves understanding of the other and helps people to signal validation and caring, which are types of communication that improve social connectedness. Taking the results presented in this dissertation together, I argue that physiological signals form a promising modality to apply in communication technology (Chapter 9). This dissertation provides a basis for future communication applications that aim to improve social connectedness

NON-VERBAL COMMUNICATION WITH PHYSIOLOGICAL SENSORS. THE AESTHETIC DOMAIN OF WEARABLES AND NEURAL NETWORKS

Historically, communication implies the transfer of information between bodies, yet this phenomenon is constantly adapting to new technological and cultural standards. In a digital context, it’s commonplace to envision systems that revolve around verbal modalities. However, behavioural analysis grounded in psychology research calls attention to the emotional information disclosed by non-verbal social cues, in particular, actions that are involuntary. This notion has circulated heavily into various interdisciplinary computing research fields, from which multiple studies have arisen, correlating non-verbal activity to socio-affective inferences. These are often derived from some form of motion capture and other wearable sensors, measuring the ‘invisible’ bioelectrical changes that occur from inside the body. This thesis proposes a motivation and methodology for using physiological sensory data as an expressive resource for technology-mediated interactions. Initialised from a thorough discussion on state-of-the-art technologies and established design principles regarding this topic, then applied to a novel approach alongside a selection of practice works to compliment this. We advocate for aesthetic experience, experimenting with abstract representations. Atypically from prevailing Affective Computing systems, the intention is not to infer or classify emotion but rather to create new opportunities for rich gestural exchange, unconfined to the verbal domain. Given the preliminary proposition of non-representation, we justify a correspondence with modern Machine Learning and multimedia interaction strategies, applying an iterative, human-centred approach to improve personalisation without the compromising emotional potential of bodily gesture. Where related studies in the past have successfully provoked strong design concepts through innovative fabrications, these are typically limited to simple linear, one-to-one mappings and often neglect multi-user environments; we foresee a vast potential. In our use cases, we adopt neural network architectures to generate highly granular biofeedback from low-dimensional input data. We present the following proof-of-concepts: Breathing Correspondence, a wearable biofeedback system inspired by Somaesthetic design principles; Latent Steps, a real-time auto-encoder to represent bodily experiences from sensor data, designed for dance performance; and Anti-Social Distancing Ensemble, an installation for public space interventions, analysing physical distance to generate a collective soundscape. Key findings are extracted from the individual reports to formulate an extensive technical and theoretical framework around this topic. The projects first aim to embrace some alternative perspectives already established within Affective Computing research. From here, these concepts evolve deeper, bridging theories from contemporary creative and technical practices with the advancement of biomedical technologies.Historicamente, os processos de comunicação implicam a transferência de informação entre organismos, mas este fenómeno está constantemente a adaptar-se a novos padrões tecnológicos e culturais. Num contexto digital, é comum encontrar sistemas que giram em torno de modalidades verbais. Contudo, a análise comportamental fundamentada na investigação psicológica chama a atenção para a informação emocional revelada por sinais sociais não verbais, em particular, acções que são involuntárias. Esta noção circulou fortemente em vários campos interdisciplinares de investigação na área das ciências da computação, dos quais surgiram múltiplos estudos, correlacionando a actividade nãoverbal com inferências sócio-afectivas. Estes são frequentemente derivados de alguma forma de captura de movimento e sensores “wearable”, medindo as alterações bioeléctricas “invisíveis” que ocorrem no interior do corpo. Nesta tese, propomos uma motivação e metodologia para a utilização de dados sensoriais fisiológicos como um recurso expressivo para interacções mediadas pela tecnologia. Iniciada a partir de uma discussão aprofundada sobre tecnologias de ponta e princípios de concepção estabelecidos relativamente a este tópico, depois aplicada a uma nova abordagem, juntamente com uma selecção de trabalhos práticos, para complementar esta. Defendemos a experiência estética, experimentando com representações abstractas. Contrariamente aos sistemas de Computação Afectiva predominantes, a intenção não é inferir ou classificar a emoção, mas sim criar novas oportunidades para uma rica troca gestual, não confinada ao domínio verbal. Dada a proposta preliminar de não representação, justificamos uma correspondência com estratégias modernas de Machine Learning e interacção multimédia, aplicando uma abordagem iterativa e centrada no ser humano para melhorar a personalização sem o potencial emocional comprometedor do gesto corporal. Nos casos em que estudos anteriores demonstraram com sucesso conceitos de design fortes através de fabricações inovadoras, estes limitam-se tipicamente a simples mapeamentos lineares, um-para-um, e muitas vezes negligenciam ambientes multi-utilizadores; com este trabalho, prevemos um potencial alargado. Nos nossos casos de utilização, adoptamos arquitecturas de redes neurais para gerar biofeedback altamente granular a partir de dados de entrada de baixa dimensão. Apresentamos as seguintes provas de conceitos: Breathing Correspondence, um sistema de biofeedback wearable inspirado nos princípios de design somaestético; Latent Steps, um modelo autoencoder em tempo real para representar experiências corporais a partir de dados de sensores, concebido para desempenho de dança; e Anti-Social Distancing Ensemble, uma instalação para intervenções no espaço público, analisando a distância física para gerar uma paisagem sonora colectiva. Os principais resultados são extraídos dos relatórios individuais, para formular um quadro técnico e teórico alargado para expandir sobre este tópico. Os projectos têm como primeiro objectivo abraçar algumas perspectivas alternativas às que já estão estabelecidas no âmbito da investigação da Computação Afectiva. A partir daqui, estes conceitos evoluem mais profundamente, fazendo a ponte entre as teorias das práticas criativas e técnicas contemporâneas com o avanço das tecnologias biomédicas

The rhythm of therapy: psychophysiological synchronization in clinical dyads

Rhythmicity and synchronization are fundamental mechanisms employed by countless natural phenomena to communicate. Previous research has found evidence for synchronization in patients and therapists during clinical activity, for instance in their body movements (Ramseyer & Tschacher, 2011) and physiological activations (e.g. Marci et al, 2007; Kleinbub et al., 2012; Messina et al., 2013). While this phenomenon has been found associated with different important aspects of clinical relationship, such as empathy, rapport, and outcome, and many authors suggested that it may describe crucial dimensions of the therapeutic dyad interaction and change, a clear explanation of its meaning is still lacking. The goals of the present work were to: 1) Provide a solid theoretical and epistemological background, in which to inscribe the phenomenon. This was pursued by crossing neurophenomenology’s sophisticated ideas on mind-body integration (Varela, 1996) and Infant Research’s detailed observations on development of infants’ Self through their relationships. The common ground for this connection was the complex systems theory (von Bertalanffy, 1968; Haken, 2006). 2) Contribute to literature through two replications of existing studies (Kleinbub et al., 2012; Messina et al., 2013) on skin conductance (SC) synchronization. In addition to the original designs, secure attachment priming (Mikulincer & Shaver, 2007) was introduced to explore if observed SC linkage was susceptible to manipulation, accordingly to the developmental premises defined in the theoretical chapters. Study 1 focused on synchrony between students and psychotherapists in simulated clinical sessions; Study 2 reprised the same methodology with two principal changes: first the clinician’s role was played by psychologists without further clinical trainings, and second, each psychologist was involved in two distinct interviews, in order to assess the impact of individual characteristics on SC synchrony. 3) Provide an ideographical exploration of the psychotherapy processes linked to matched SC activity. In study 3 the highest and lowest synchrony sequences of 6 sessions of psychodynamic psychotherapy were subject of a detailed phenomenological content analysis. These micro-categories were synthetized in more abstract ones, in order to attempt the recognizing of regularities that could shed light on the phenomenon. 4) To explore the pertinence of employing mathematical properties derived from the application of system theory in psychological contexts. In study 4, Shannon’s entropy and order equations (1948) were applied on the transcribed verbal content of 12 depression psychotherapies, to assess both intra-personal and inter-personal (dyad) order in verbal categories. Results from these studies provided further evidence for the existence of a synchronization mechanism in the clinical dyads. Furthermore the various findings were generally supporting the dyad system theoretical model, and its description of regulatory dynamics as a good explanation of the synchronization phenomena. Discrepancies with previous literature highlighted the need for further studies to embrace more methodological sophistication (such as employing lag analysis), and cautiousness in the interpretation of results