Using Wearable Sensors to Measure Interpersonal Synchrony in Actors and Audience Members During a Live Theatre Performance

Studying social interaction in real-world settings is of increasing importance to social cognitive researchers. Theatre provides an ideal opportunity to study rich face-to-face interactions in a controlled, yet natural setting. Here we collaborated with Flute Theatre to investigate interpersonal synchrony between actors-actors, actors-audience and audience-audience within a live theatrical setting. Our 28 participants consisted of 6 actors and 22 audience members, with 5 of these audience members being audience participants in the show. The performance was a compilation of acting, popular science talks and demonstrations, and an audience participation period. Interpersonal synchrony was measured using inertial measurement unit (IMU) wearable accelerometers worn on the heads of participants, whilst audio-visual data recorded everything that occurred on the stage. Participants also completed post-show self-report questionnaires on their engagement with the overall scientists and actors performance. Cross Wavelet Transform (XWT) and Wavelet Coherence Transform (WCT) analysis were conducted to extract synchrony at different frequencies, pairing with audio-visual data. Findings revealed that XWT and WCT analysis are useful methods in extracting the multiple types of synchronous activity that occurs when people perform or watch a live performance together. We also found that audience members with higher ratings on questionnaire items such as the strength of their emotional response to the performance, or how empowered they felt by the performance, showed a high degree of interpersonal synchrony with actors during the acting segments of performance. We further found that audience members rated the scientists performance higher than the actors performance on questions related to their emotional response to the performance as well as, how uplifted, empowered, and connected to social issues they felt. This shows the types of potent connections audience members can have with live performances. Additionally, our findings highlight the importance of the performance context for audience engagement, in our case a theatre performance as part of public engagement with science rather than a stand-alone theatre performance. In sum we conclude that interdisciplinary real-world paradigms are an important and understudied route to understanding in-person social interactions

Feeling the Temperature of the Room: Unobtrusive Thermal Display of Engagement during Group Communication

Thermal signals have been explored in HCI for emotion-elicitation and enhancing two-person communication, showing that temperature invokes social and emotional signals in individuals. Yet, extending these findings to group communication is missing. We investigated how thermal signals can be used to communicate group affective states in a hybrid meeting scenario to help people feel connected over a distance. We conducted a lab study (N=20 participants) and explored wrist-worn thermal feedback to communicate audience emotions. Our results show that thermal feedback is an effective method of conveying audience engagement without increasing workload and can help a presenter feel more in tune with the audience. We outline design implications for real-world wearable social thermal feedback systems for both virtual and in-person communication that support group affect communication and social connectedness. Thermal feedback has the potential to connect people across distances and facilitate more effective and dynamic communication in multiple contexts.Comment: In IMWUT 202

Frisson Waves: Exploring Automatic Detection, Triggering and Sharing of Aesthetic Chills in Music Performances

Frisson is the feeling and experience of physical reactions such as shivers, tingling skin, and goosebumps. Using entrainment through facilitating interpersonal transmissions of embodied sensations, we present "Frisson Waves" with the aim to enhance live music performance experiences. "Frisson Waves" is an exploratory real-time system to detect, trigger and share frisson in a wave-like pattern over audience members during music performances. The system consists of a physiological sensing wristband for detecting frisson and a thermo-haptic neckband for inducing frisson. In a controlled environment, we evaluate detection (n=19) and triggering of frisson (n=15). Based on our findings, we conducted an in-the-wild music concert with 48 audience members using our system to share frisson. This paper summarizes a framework for accessing, triggering and sharing frisson. We report our research insights, lessons learned, and limitations of "Frisson Waves". Yan He, George Chernyshov, Jiawen Han, Dingding Zheng, Ragnar Thomsen, Danny Hynds, Muyu Liu, Yuehui Yang, Yulan Ju, Yun Suen Pai, Kouta Minamizawa, Kai Kunze, and Jamie A War

CorrNet: Fine-grained emotion recognition for video watching using wearable physiological sensors

Recognizing user emotions while they watch short-form videos anytime and anywhere is essential for facilitating video content customization and personalization. However, most works either classify a single emotion per video stimuli, or are restricted to static, desktop environments. To address this, we propose a correlation-based emotion recognition algorithm (CorrNet) to recognize the valence and arousal (V-A) of each instance (fine-grained segment of signals) using only wearable, physiological signals (e.g., electrodermal activity, heart rate). CorrNet takes advantage of features both inside each instance (intra-modality features) and between different instances for the same video stimuli (correlation-based features). We first test our approach on an indoor-desktop affect dataset (CASE), and thereafter on an outdoor-mobile affect dataset (MERCA) which we collected using a smart wristband and wearable eyetracker. Results show that for subject-independent binary classification (high-low), CorrNet yields promising recognition accuracies: 76.37% and 74.03% for V-A on CASE, and 70.29% and 68.15% for V-A on MERCA. Our findings show: (1) instance segment lengths between 1–4 s result in highest recognition accuracies (2) accuracies between laboratory-grade and wearable sensors are comparable, even under low sampling rates (≤64 Hz) (3) large amounts of neu-tral V-A labels, an artifact of continuous affect annotation, result in varied recognition performance

Evaluating the Reproducibility of Physiological Stress Detection Models

Recent advances in wearable sensor technologies have led to a variety of approaches for detecting physiological stress. Even with over a decade of research in the domain, there still exist many significant challenges, including a near-total lack of reproducibility across studies. Researchers often use some physiological sensors (custom-made or off-the-shelf), conduct a study to collect data, and build machine-learning models to detect stress. There is little effort to test the applicability of the model with similar physiological data collected from different devices, or the efficacy of the model on data collected from different studies, populations, or demographics. This paper takes the first step towards testing reproducibility and validity of methods and machine-learning models for stress detection. To this end, we analyzed data from 90 participants, from four independent controlled studies, using two different types of sensors, with different study protocols and research goals. We started by evaluating the performance of models built using data from one study and tested on data from other studies. Next, we evaluated new methods to improve the performance of stress-detection models and found that our methods led to a consistent increase in performance across all studies, irrespective of the device type, sensor type, or the type of stressor. Finally, we developed and evaluated a clustering approach to determine the stressed/not-stressed classification when applying models on data from different studies, and found that our approach performed better than selecting a threshold based on training data. This paper\u27s thorough exploration of reproducibility in a controlled environment provides a critical foundation for deeper study of such methods, and is a prerequisite for tackling reproducibility in free-living conditions

The potential of emerging wearable physiological sensing in the space of human-subject studies

PhD ThesisIn recent years, novel sensing means in the form of smartwatches and fitness trackers with integrated sophisticated sensing emerged on the consumer market. While their primary purpose is to provide consumers with an overview of rough-grained health-related metrics, these signals offer to pick up fine-grained changes within the human body. This thesis considers the suitability of these novel wearable sensing devices to be used in affective research. Firstly, and based on the work with concrete state-of-the-art wearables, issues around the access of research-suitable data are discussed. The findings are put in context by examining common wearable device architectures and data access means provided. The discussion concludes with aspects researchers need to consider when seeking data access from state-of-the-art or future wearables. Secondly, two research probes explore the application of four exemplary devices to detect stress and affect in the wild and in the lab. Issues around the data reliability and participant comfort arose. The experiences are reflected upon to provide researchers with a summary of aspects to consider when applying wearable sensing devices in affective research. Lastly, this thesis contributes a Design Space for Physiological Measurement Tools. This design space was evaluated with a qualitative study enquiring research experts experiences. The resulting Design Space presents seven distinct dimensions of factors to consider when choosing a wearable sensing device for research. This design space has been applied to a novel sensing device which was used for a study on interpersonal synchrony. The insights and the ‘Design Space for Physiological Measurement Tools’ provide researchers with a tool to apply when they consider to use wearable physiological sensing devices in research

EDA-signaalin automaattinen virheensuodatus

Tiivistelmä. Tässä kandidaatintyössä tutkin ihmiskehosta mitatun sähkönjohtavuussignaalin eli EDA-signaalin automaattista virheensuodatusta. EDA-signaali on kytköksissä ihmisen parasympaattiseen hermostoon, joten sen avulla voidaan tulkita ihmisen tunnetiloja ja vireystilaa. Tavallinen EDA-signaali etenee muutaman sekunnin mittaisissa nousevissa ja laskevissa aalloissa. Tässä tutkielmassa käyttämäni data on kerätty Empatica E4 -rannekkeella, joka mittaa EDA-signaalin lisäksi myös kiihtyvyysdataa. Koska mittalaite asetetaan koehenkilön käteen, se on altis henkilön liikkeiden aiheuttamille virheille. Nämä virheet saattavat näkyä signaalissa sekä nollaan pudonneena signaalina että epätavallisen jyrkkinä piikkeinä. EDA-signaalin keräämistä, sen ongelmia ja virheensuodatusta on käsitelty kirjallisuudessa ja siihen on esitetty erilaisia ratkaisuja. Tässä tutkielmassa käyn läpi aiheeseen liittyviä artikkeleita ja esittelen muutamia ratkaisuehdotuksia virhepiikkien eli niin sanottujen artefaktien automaattiseksi tunnistamiseksi. Esittelen myös oman ratkaisuni, joka perustuu signaalin pätkien luokitteluun normaaleiksi tai todennäköisesti virheellisiksi muodon ja amplitudin perusteella. Leimoja on neljää eri tyyppiä: 0 — normaalia signaalia; 1 — signaali on pudonnut alle minimiamplitudin; 2 — signaalissa on liikkumisesta johtuva jyrkkä piikki; ja 3 — signaalissa on tuntemattomasta syystä johtuva jyrkkä piikki. Luokittelussa käytetään apuna EDA-signaalin kanssa samanaikaisesti kerättyä kiihtyvyysdataa. Analysoitu ja luokiteltu signaali esitetään graafisesti väriä leiman mukaan vaihtavalla viivalla. Esittelen lyhyesti ohjelmani tuottamia tuloksia ja arvioin niiden oikeellisuutta. Lopuksi esitän ehdotuksia siitä, kuinka työtä voisi jatkaa, ja arvioin ratkaisematta jääneitä ongelmia.Automatic error detection in EDA signal. Abstract. This is a bachelor’s thesis on automatic detection of artefacts in EDA signal (Electodermal Activity). EDA measures electrical characteristics of human skin. These characteristics are connected to the parasympathetic nerve system and thus they reflect emotions and arousal of the person being monitored. My data is collected using Empatica E4 wristband and it consists of several hours of EDA and acceleration data from three different days. Wristbands are liable to errors due to rapid movement of the test person, and these errors can be seen in the signal as steep peaks or very low amplitude level. Filtering these errors from EDA signal has been discussed in several articles, many of which also provide solutions to this problem. In this thesis I present some of these articles. I also suggest my own solution, which is a Python program that measures amplitude and derivative of the signal. Concurrently gathered acceleration data is used in determining whether the signal is erroneous due to rapid movements of the test person. Every sample of the signal is being labeled based on these properties. There are four different labels: 0 — normal signal; 1 — amplitude of the signal is too low; 2 — fast change in signal level due to movement of the test person; and 3 — fast change in signal level due to unknown reason. The program creates metadata, which contains information about proportions of the different labels. Lastly, labeled signal is presented as a multicoloured line. After discussing my methodology I present and assess the results yielded by my program. In the last section I discuss unsolved problems and propose possible themes for future work