CLERA: A Unified Model for Joint Cognitive Load and Eye Region Analysis in the Wild

Non-intrusive, real-time analysis of the dynamics of the eye region allows us to monitor humans' visual attention allocation and estimate their mental state during the performance of real-world tasks, which can potentially benefit a wide range of human-computer interaction (HCI) applications. While commercial eye-tracking devices have been frequently employed, the difficulty of customizing these devices places unnecessary constraints on the exploration of more efficient, end-to-end models of eye dynamics. In this work, we propose CLERA, a unified model for Cognitive Load and Eye Region Analysis, which achieves precise keypoint detection and spatiotemporal tracking in a joint-learning framework. Our method demonstrates significant efficiency and outperforms prior work on tasks including cognitive load estimation, eye landmark detection, and blink estimation. We also introduce a large-scale dataset of 30k human faces with joint pupil, eye-openness, and landmark annotation, which aims to support future HCI research on human factors and eye-related analysis.Comment: ACM Transactions on Computer-Human Interactio

Interactive Tango Milonga: An Interactive Dance System for Argentine Tango Social Dance

abstract: When dancers are granted agency over music, as in interactive dance systems, the actors are most often concerned with the problem of creating a staged performance for an audience. However, as is reflected by the above quote, the practice of Argentine tango social dance is most concerned with participants internal experience and their relationship to the broader tango community. In this dissertation I explore creative approaches to enrich the sense of connection, that is, the experience of oneness with a partner and complete immersion in music and dance for Argentine tango dancers by providing agency over musical activities through the use of interactive technology. Specifically, I create an interactive dance system that allows tango dancers to affect and create music via their movements in the context of social dance. The motivations for this work are multifold: 1) to intensify embodied experience of the interplay between dance and music, individual and partner, couple and community, 2) to create shared experience of the conventions of tango dance, and 3) to innovate Argentine tango social dance practice for the purposes of education and increasing musicality in dancers.Dissertation/ThesisDoctoral Dissertation Music 201

A Physiological Signal Processing System for Optimal Engagement and Attention Detection.

In today’s high paced, hi-tech and high stress environment, with extended work hours, long to-do lists and neglected personal health, sleep deprivation has become common in modern culture. Coupled with these factors is the inherent repetitious and tedious nature of certain occupations and daily routines, which all add up to an undesirable fluctuation in individuals’ cognitive attention and capacity. Given certain critical professions, a momentary or prolonged lapse in attention level can be catastrophic and sometimes deadly. This research proposes to develop a real-time monitoring system which uses fundamental physiological signals such as the Electrocardiograph (ECG), to analyze and predict the presence or lack of cognitive attention in individuals during task execution. The primary focus of this study is to identify the correlation between fluctuating level of attention and its implications on the physiological parameters of the body. The system is designed using only those physiological signals that can be collected easily with small, wearable, portable and non-invasive monitors and thereby being able to predict well in advance, an individual’s potential loss of attention and ingression of sleepiness. Several advanced signal processing techniques have been implemented and investigated to derive multiple clandestine and informative features. These features are then applied to machine learning algorithms to produce classification models that are capable of differentiating between the cases of a person being attentive and the person not being attentive. Furthermore, Electroencephalograph (EEG) signals are also analyzed and classified for use as a benchmark for comparison with ECG analysis. For the study, ECG signals and EEG signals of volunteer subjects are acquired in a controlled experiment. The experiment is designed to inculcate and sustain cognitive attention for a period of time following which an attempt is made to reduce cognitive attention of volunteer subjects. The data acquired during the experiment is decomposed and analyzed for feature extraction and classification. The presented results show that to a fairly reasonable accuracy it is possible to detect the presence or lack of attention in individuals with just their ECG signal, especially in comparison with analysis done on EEG signals. The continual work of this research includes other physiological signals such as Galvanic Skin Response, Heat Flux, Skin Temperature and video based facial feature analysis

Enhancing user experience and safety in the context of automated driving through uncertainty communication

Operators of highly automated driving systems may exhibit behaviour characteristic of overtrust issues due to an insufficient awareness of automation fallibility. Consequently, situation awareness in critical situations is reduced and safe driving performance following emergency takeovers is impeded. Previous research has indicated that conveying system uncertainties may alleviate these issues. However, existing approaches require drivers to attend the uncertainty information with focal attention, likely resulting in missed changes when engaged in non-driving-related tasks. This research project expands on existing work regarding uncertainty communication in the context of automated driving. Specifically, it aims to investigate the implications of conveying uncertainties under consideration of non-driving-related tasks and, based on the outcomes, develop and evaluate an uncertainty display that enhances both user experience and driving safety. In a first step, the impact of visually conveying uncertainties was investigated under consideration of workload, trust, monitoring behaviour, non-driving-related tasks, takeover performance, and situation awareness. For this, an anthropomorphic visual uncertainty display located in the instrument cluster was developed. While the hypothesised benefits for trust calibration and situation awareness were confirmed, the results indicate that visually conveying uncertainties leads to an increased perceived effort due to a higher frequency of monitoring glances. Building on these findings, peripheral awareness displays were explored as a means for conveying uncertainties without the need for focused attention to reduce monitoring glances. As a prerequisite for developing such a display, a systematic literature review was conducted to identify evaluation methods and criteria, which were then coerced into a comprehensive framework. Grounded in this framework, a peripheral awareness display for uncertainty communication was developed and subsequently compared with the initially proposed visual anthropomorphic uncertainty display in a driving simulator study. Eye tracking and subjective workload data indicate that the peripheral awareness display reduces the monitoring effort relative to the visual display, while driving performance and trust data highlight that the benefits of uncertainty communication are maintained. Further, this research project addresses the implications of increasing the functional detail of uncertainty information. Results of a driving simulator study indicate that particularly workload should be considered when increasing the functional detail of uncertainty information. Expanding upon this approach, an augmented reality display concept was developed and a set of visual variables was explored in a forced choice sorting task to assess their ordinal characteristics. Particularly changes in colour hue and animation-based variables received high preference ratings and were ordered consistently from low to high uncertainty. This research project has contributed a series of novel insights and ideas to the field of human factors in automated driving. It confirmed that conveying uncertainties improves trust calibration and situation awareness, but highlighted that using a visual display lessens the positive effects. Addressing this shortcoming, a peripheral awareness display was designed applying a dedicated evaluation framework. Compared with the previously employed visual display, it decreased monitoring glances and, consequentially, perceived effort. Further, an augmented reality-based uncertainty display concept was developed to minimise the workload increments associated with increases in the functional detail of uncertainty information.</div

Exploring professional engineers' knowings-in-practice in an emerging industry: An Actor-Network Theory approach

This thesis presents a sociomaterial perspective on how everyday engineering work practices are being changed by the complexities and tensions prevalent in emerging industries. Presenting the wind energy industry, in the renewable energy sector, as a case, this study contends that current engineering education practices are not adequately preparing and supporting students and professionals for work in highly volatile, precarious industries. This study pays close attention to how engineers enact competent knowing and learning strategies to respond to, and navigate, these complexities and tensions. Traditional engineering education practices tend to frame engineering work as a bounded, stable, rational, and technical endeavor, where knowledge is regarded as a commodity to be acquired. Rather than treating professional knowledge as an independent reality of the engineering field, this thesis argues that education practices can be informed by making visible mundane and taken-for-granted aspects of engineers’ everyday work, and reconfiguring conceptualisations of engineering knowledge as situated, collective, on-going, and materially-mediated performances. To do so, this study draws on concepts of knowing-in-practice and Actor-Network Theory, which position engineering work as heterogeneous assemblages of social and material relations. An ethnographic methodology afforded the tracing of social and material relations between 13 participating engineers and the objects of their practice in a wind energy organisation located in a Scottish city. Following six months of observations and interviews, three activities that generated high intensity in the engineers' everyday work were analysed: securing a signature on a contract, the unfolding of a specific organising process, and implementing a new technology. Analysis revealed four tensions that needed to be constantly negotiated, which included balancing: commercial objectives and client needs with traditional engineering concerns; standardising practices with innovating practices; acceptable practice with allowable deviation; and visibility with invisibility. Emerging from the findings were clear indications that the multiple knowings-in-practice enacted to negotiate these tensions were interdependent, yet partial, fluid and multiple, sociomaterial performances. This thesis offers recommendations for education practices based on these findings, which challenge dominant representational and individualistic conceptualisations of engineering education and workplace learning. Furthermore, a ‘dynamic stability’ sensibility is offered as a pedagogical approach that encourages attunement to the performance of fluid and informal infrastructuring practices, which tolerate volatility and high-change in work practices

Embodied Customer Experience in Human Touch Services : A phenomenological perspective

Elämä tapahtuu kehossa ja mielessä – nämä kaksi kietoutuvat tiiviisti toisiinsa, kun muodostamme merkityksiä ja kokemuksia. Aktiivinen ja tunteva keho on läsnä, kun osallistumme joogatunnille, olemme vaateostoksilla, vietämme iltaa ravintolassa tai teemme päätöksiä pankkineuvotteluissa. Tällaisessa maailmassa me kaikki elämme ja luomme omia kokemuksiamme. Keho kuitenkin loistaa poissaolollaan niin yritysten päivittäisissä keskusteluissa kuin asiakaskokemusta käsittelevissä tutkimuksissa. Tämänhetkiset tutkimukset käsittelevät kokemusta pääasiassa kognitiivisten käsitysten kautta ja kehon roolia pidetään lähinnä itsestäänselvyytenä. On siis olemassa selkeä tarve vaihtoehtoiselle näkökulmalle, joka laajentaa kirjallisuuden tarjoamaa rajoittunutta näkemystä. Tämä tutkimus rakentaa uuden, kehollisen näkökulman asiakaskokemuksen tarkasteluun. Tätä tavoitetta lähestytään kahdella tutkimuskysymyksellä: 1) Kuinka kehollinen asiakaskokemus muodostuu? 2) Millainen on kehon rooli asiakaskokemuksessa ihmisläheisissä palveluissa? Vastatakseen näihin kysymyksiin, tämä väitöskirja sisältää neljä tutkimusartikkelia ja tämän johdannon. Artikkelit tarkastelevat kehon keskeistä roolia kokemuksen muodostumisessa moninäkökulmaisesti. Artikkeli I kartoittaa tunteiden roolia asiakaskokemuksessa huomioiden, kuinka tunteet ovat keskeinen osa kokemusta. Artikkeli II luo uutta ymmärrystä vuorovaikutuksesta kehollisena ja monitasoisena ilmiönä ryhmäliikunnan kontekstissa. Artikkeli III keskittyy tarkastelemaan palvelukohtaamisia kehollisina käytäntöinä. Artikkeli IV tarkastelee sitä, kuinka asiakaskokemus muodostuu asiakkaan subjektiivisessa kokemusmaailmassa. Tämä osatutkimus kuvaa, kuinka kokemusmaailma muovautuu ja määrittyy kehon kautta. Tutkimus omaksuu fenomenologisen näkökulman, jonka kautta asiakaskokemusta voidaan tarkastella, ensiksi, kokijan subjektiivisesta näkökulmasta ja toiseksi, ymmärtää kokemus lähtökohtaisesti kehollisena ilmiönä. Tutkimuksen tarkoitus ohjaa artikkeleiden menetelmävalintoja. Menetelminä hyödynnetään fenomenologisia haastatteluja, fokusryhmähaastatteluita, autoetnografiaa sekä havainnointia. Tutkimus arvioi kriittisesti asiakaskokemus- ja palvelututkimuksen käsityksiä, joissa kehon aktiivista roolia ei vielä ole tarkasteltu. Tutkimus tuo kehollisuuden käsitteen keskusteluun ammentamalla tutkimusvirroista, joissa kehon keskeinen rooli kokemuksessa on jo vahvasti tunnistettu. Sosiologian, organisaatio- ja johtamistutkimuksen sekä kulutustutkimuksen suuntautuksiin nojaava monitieteinen näkökulma rikastaa asiakaskokemusta koskevaa keskustelua. Väitöskirja haastaa asiakaskokemusta käsittelevän tutkimuksen perinteisiä, mekaanisia käsityksiä, jotka eivät ole huomioineet kehon keskeistä roolia kokemuksen muodostumisessa. Tutkimus tuo esiin uuden, vaihtoehtoisen ajattelutavan käsittää, mistä asiakaskokemus muodostuu ja kuinka sitä tulisi johtaa. Tutkimuksen tulokset rakentavat yksityiskohtaista ymmärrystä siitä, kuinka kehollinen asiakaskokemus muodostuu monimutkaisten kehollisten sidosten ja luovan toiminnan kautta. Kehon keskeinen rooli asiakaskokemuksessa ihmisläheisissä palveluissa tunnistetaan aktiivisena ja kokonaisvaltaisena: keho on kokemuksen ulottuvuuksien keskus sekä yhdyspiste menneen ja tulevan, yksityisen ja sosiaalisen, tiedon ja intuition välillä, mikä mahdollistaa asiakaskokemuksen muodostumisen. Tutkimuksen keskeiset teoreettiset kontribuutiot kirkastetaan neljän avainlöydöksen kautta. Menetelmällisesti väitöskirjani nostaa esiin, minkälaiset metodit ja tutkimusasetelmat soveltuvat kehollisuuden tutkimukseen. Tutkimuksessa luodaan palveluyrityksille malli, jonka avulla kehollinen lähestymistapa voidaan valjastaa liiketoiminnan ajuriksi. Malli koostuu neljästä osa- alueesta, jotka ohjaavat yritysjohtajia luomaan liiketoimintakäsityksensä uudelleen: kehollisen lähestymistavan haltuunotto, liiketoimintakäytäntöjen uudelleenluominen, asiakkaan uudelleenarviointi sekä palvelukohtaamisen uudelleenorientointi. Tutkimustulokset rohkaisevat yritysjohtajia ymmärtämään, kuinka aktiivinen ja tunteva keho on menestyksekkään palveluliiketoiminnan ytimessä.Life happens through our body and mind, and the two are closely intertwined as we form meanings and experiences. An active and sensing human body is present when we participate in a yoga class, shop for clothing, spend an evening in a restaurant, or make decisions in banking negotiations. This is the world in which we all live and create our experiences. However, despite the essential role of the body in our doings, the human body is conspicuous in its absence in daily conversations in companies’ agendas, as well as in customer experience (CX) studies in service research. Existing studies examine CX mainly through cognitive perceptions, while the role of the body is largely taken for granted. There is an evident need for an alternative perspective that expands on this restricted view. The purpose of this study is to create an understanding of CX as embodied in human touch service contexts. To achieve this purpose, two research questions are set: 1) How the embodied CX is formed? 2) What kind of a role the body plays in CX in human touch services? To answer these questions, this dissertation comprises four articles with this introduction. The articles explore the central role of the body in experiences from multiple viewpoints. Article I maps the role of emotions in CX and acknowledges how they are embedded in these experiences. Article II creates a new understanding of interaction as a bodily and multilevel phenomenon in the group fitness context. Article III focuses on service encounters as bodily practices. Article IV examines how customer experience emerges in the customer’s subjective life-world, illustrating how the life-world is shaped and defined through the body. This research applies a phenomenological perspective, which enables the study of customer experience from a first-person view and helps us understand experience as a bodily phenomenon. The aim of the research guides the methodology of the articles in the dissertation, which include phenomenological interviews, focus group interviews, autoethnography, and observation. This study critically evaluates the prevailing assumptions of CX in service research, in which the active role of the body has not been examined. The concept of embodiment is introduced by drawing on research streams in which the central role of the body in experience has been identified. A multidisciplinary perspective based on sociology, consumption research, and organizational and management studies enriches the discussion of CX. This dissertation challenges the assumptions of the traditional, mechanical conceptions of research on CX that have not taken into account the central role of the body in the formation of experience. It offers an alternative view for understanding how customer experience emerges and how it should be managed. The findings build detailed understanding on how embodied CXs are formed through complex combination of bodily linkages and creative acts. The body’s role in CX in human touch services is identified as active and holistic: the body is the locus of dimensions of experience, as well as a connection point of previous and becoming, private and social, knowledge and intuition, enabling CX formation. The theoretical contributions of this study are highlighted through four key propositions. The methods and research designs suitable for studying embodiment are discussed. The key managerial contributions are presented in a model that provides guidelines for service companies on how to harness an embodied approach as a driver of a successful business. The guidelines consist of four components for business managers to recreate their business understanding: takeover of the embodied approach, recreation of business practices, reconsideration of the customer, and reorientation of the service encounter. Altogether, the findings help business managers understand how the active and sensing body is at the center of a successful service business