Tunteiden Havaitseminen Arkielämässä Koneoppimisen ja Puettavien Laitteiden Avulla

Tavoitteet. Tämän tutkimuksen tavoitteena on arvioida tunteiden havaitsemisen mahdollisuutta arkielämässä puettavien laitteiden ja koneoppimismallien avulla. Tunnetiloilla on tärkeä rooli päätöksenteossa, havaitsemisessa ja käyttäytymisessä, mikä tekee objektiivisesta tunnetilojen havaitsemisesta arvokkaan tavoitteen, sekä mahdollisten sovellusten että tunnetiloja koskevan ymmärryksen syventämisen kannalta. Tunnetiloihin usein liittyy mitattavissa olevia fysiologisia ja käyttäymisen muutoksia, mikä mahdollistaa koneoppimismallien kouluttamisen muutoksia aiheuttaneen tunnetilan havaitsemiseksi. Suurin osa tunteiden havaitsemiseen liittyvästä tutkimuksesta on toteutettu laboratorio-olosuhteissa käyttämällä tunteita herättäviä ärsykkeitä tai tehtäviä, mikä herättää kysymyksen siitä että yleistyvätkö näissä olosuhteissa saadut tulokset arkielämään. Vaikka puettavien laitteiden ja kännykkäkyselyiden kehittyminen on helpottanut aiheen tutkimista arkielämässä, tutkimusta tässä ympäristössä on vielä niukasti. Tässä tutkimuksessa itseraportoituja tunnetiloja ennustetaan koneoppimismallien avulla arkielämässä havaittavissa olevien tunnetilojen selvittämiseksi. Lisäksi tutkimuksessa käytetään mallintulkintamenetelmiä mallien hyödyntämien yhteyksien tunnistamiseksi. Metodit. Aineisto tätä tutkielmaa varten on peräisin tutkimuksesta joka suoritettiin osana Helsingin Yliopiston ja VTT:n Sisu at Work projektia, missä 82:ta tietotyöläistä neljästä suomalaisesta organisaatiosta tutkittiin kolmen viikon ajan. Osallistujilla oli jakson aikana käytettävissään mittalaitteet jotka mittasivat fotoplethysmografiaa (PPG), ihon sähkönjohtavuutta (EDA) ja kiihtyvyysanturi (ACC) signaaleita, lisäksi heille esitettiin kysymyksiä koetuista tunnetiloista kolmesti päivässä puhelinsovelluksen avulla. Signaalinkäsittelymenetelmiä sovellettiin signaaleissa esiintyvien liikeartefaktien ja muiden ongelmien korjaamiseksi. Sykettä (HR) ja sykevälinvaihtelua (HRV) kuvaavia piirteitä irroitettiin PPG signaalista, fysiologista aktivaatiota kuvaavia piirteitä EDA signaalista, sekä liikettä kuvaavia piirteitä ACC signaalista. Seuraavaksi koneoppimismalleja koulutettiin ennustamaan raportoituja tunnetiloja irroitetujen piirteiden avulla. Mallien suoriutumista vertailtiin suhteessa odotusarvoihin havaittavissa olevien tunnetilojen määrittämiseksi. Lisäksi permutaatiotärkeyttä sekä Shapley additive explanations (SHAP) arvoja hyödynnettiin malleille tärkeiden yhteyksien selvittämiseksi. Tulokset ja johtopäätökset. Mallit tunnetiloille virkeä, keskittynyt ja innostunut paransivat suoriutumistaan yli odotusarvon, joista mallit tunnetilalle virkeä paransivat suoriutumista tilastollisesti merkitsevästi. Permutaatiotärkeys korosti liike- ja HRV-piirteiden merkitystä, kun SHAP arvojen tarkastelu nosti esiin matalan liikkeen, matalan EDA:n, sekä korkean HRV:n merkityksen mallien ennusteille. Nämä tulokset ovat lupaavia korkean aktivaation positiivisten tunnetilojen havaitsemiselle arkielämässä, sekä nostavat esiin mahdollisia yhteyksiä jatkotutkimusta varten.Objectives. This study aims to evaluate feasibility of affect detection in daily life using wearable devices and machine learning models. Affective states play an important role in decision making, perception and behaviour, making objective detection of affective states a desirable goal both for potential applications and as a way to gain insight into affective phenomena. Affective states have been found to have measurable physiological and behavioral changes, which allows training of machine learning models for detecting the underlying affects. Majority of affect detection studies have been conducted in laboratory conditions using affect elicitation stimuli or tasks, raising the question whether results from these studies will generalize to daily life. Although development of wearable devices and mobile surveys have facilitated evaluation in the context of daily life, research here remains sparse. In this study, self-reported affective states are predicted using machine learning models to identify which affective states can be detected in daily life. Additionally, model interpretation methods will be used to identify which relationships the models found important for their predictions. Methods. Data for this thesis came from a study conducted as a part of Sisu at Work project between University of Helsinki and VTT, where 82 knowledge workers from four Finnish organizations were studied for a period of three weeks. During this period, the participants were queried by mobile surveys about their affective states thrice a day, while they also used wearable devices to record photoplethysmography (PPG), electrodermal activity (EDA) and accelerometry (ACC) signals. A signal processing pipeline was implemented to deal with movement artefacts and other issues with the data. Features describing heart rate (HR) and heart rate variation (HRV) were extraced from PPG, physiological activation from EDA and movement from ACC signals. Models were then fitted to predict the reported affective states using the extracted features. Model performance was compared against a baseline to identify which affects could be reliably detected, while permutation importance and Shapley additive explanations (SHAP) values were used to identify important relationships established by the models. Results and conclusions. Models for affective state vigor showed improvements over baseline with statistical significance, while improvements were also noted for affects focused and enthusiastic. Permutation importance highlighted the significance of movement and HRV features, while examination of SHAP values indicated that low movement, low EDA and high HRV impacted model predictions the most. These results indicate potential for detecting high activation affective states in daily life and propose potential relationships for future research

Exploring heart rate variability as a human performance optimisation metric for law enforcement

Tactical personnel, inclusive of police officers, face complex challenges over potentially decades-long careers. These cumulative exposures may manifest as allostatic load, impairing health, fitness, and performance. Allostatic load describes increased vulnerability to psychophysiological dysfunction resulting from prolonged overstress exposure. Monitoring for this risk is an important step towards its mitigation. Heart rate variability (HRV) analysis can noninvasively acquire psychophysiological overstress information in tactical environments. HRV theory and principles are well established, however, the integration of HRV in tactical workflows, especially for end-users, has received limited research attention. Therefore, the overarching aim of this programme of research was to determine the utility of HRV assessment in the support of specialist police and their organisations and to alert stakeholders to potential instances of psychophysiological overstress. Chapter 1 introduces HRV concepts within tactical contexts. Components of tactical work that may be best appreciated with HRV analysis are highlighted. Principles in this introduction are further articulated in a systematic review (Chapter 2). Chapter 2 reports on the undertaken systematic review which summarised and critically appraised studies of HRV applications across tactical populations. Of 296 initially identified studies, twenty were included. The volume of evidence suggested that HRV effectively supports health and performance measures in tactical environments. However, literature gaps were identified; most notably, there was limited evidence available regarding HRV in specialist police professions, thus warranting this research. As professional requirements and potential allostatic load sources differ during specialist police selection and subsequent specialist police operational contexts, two research arms were devised to pragmatically address this critical gap. Chapter 3 illustrates the research structure in further detail and outlines which studies address specific literature gaps within specialist police and in which of the two developed research arms. Methodological approaches are also described. Chapters 4, 5, 6, and 7 encompass HRV application in initial specialist police selection. Chapter 4 introduces the first field study, building on the findings from Chapter 2 (Studyiii1) that HRV assessment may be more valuable than traditional heart rate (HR)measurement for monitoring tactical training as HRV is capable of measuring stress holistically. The primary aim therefore was to investigate whether HRV was more sensitive than HR at monitoring workload during specialist police selection activities. As aerobic fitness is associated with workload during these tasks, a secondary aim was to investigate relationships between HRV, HR, and maximal aerobic fitness. As illustrated by a time-series plot, HR values were unremarkable while HRV values were potentially depressed, and tentatively indicative of overstress. Estimated maximal aerobic fitness (20-m shuttle run) was significantly positively correlated with HRV, but there was no relationship with HR. When a linear regression model was applied, neither HRV nor HR were predicted by 20-m shuttle run scores. Chapter 5 aimed to determine the effectiveness of HRV in differentiating between candidates that failed to complete specialist selection from those who succeeded. HRV was defined as the percentage of R-R intervals that varied by at least 50ms (pRR50). Data were summarised in a heat map. A logistic regression model was generated that effectively predicted attrition but did not identify the most successful candidate. Therefore, the aim of Chapter 6 was to profile HRV characteristics of that successful candidate and consequently a detailed HRV time series plot was generated. Contextual analysis was applied, and the candidate demonstrated continued performance even under apparent duress, both physical and psychological in nature. The subsequent studies (7-9)then aimed to consider HRV monitoring at the operational level where such duress exposures occur frequently. Noting that success in training is distinct from operational performance, Chapters 7, 8, 9,and 10 examined the use of HRV monitoring in operational contexts. The purpose of Chapter 7 was to identify if HRV analysis could classify candidate performance in specialist police selection during occupationally realistic tactical operations scenarios which required fluid psychomotor skills, teamwork, and leadership while under duress. Qualitative analysis of descriptive statistics indicated that the HRV data of one participant were substantially different from his peers. This candidate was also the highest performer, suggesting a relationship between HRV and occupational aptitude. Given that specialist police often work rotating shift schedules which may lead to sleep deprivation, introducing another source of allostatic load, the aim of Chapter 8 was to determine the extent to which HRV may detect differences between specialist police that worked an overnight shift and those that were off duty overnight. HRV was analysed in11 male specialist police officers who were either off-duty or on overnight duty prior to engaging in specialist assessment activities. All officers experienced HRV perturbations from the assessment, but post-assessment HRV was greater amongst those who were coming on duty. HRV values continued to decline after assessment success amongst those that worked the night prior to training, potentially indicating greater stress loads in those that worked the overnight shift. Chapter 9 further explored HRV changes observed in Chapter 8. The aim was to identify relationships between physical fitness as measured by completion time on a primarily anaerobic occupational obstacle course, and HRV response during firearms qualification and subsequent stress training. HRV was assessed as the within-operator change from pre- to post-qualification and post-training. HRV was reduced after training but not after qualification. A linear regression model indicated that obstacle course completion time predicted HRV changes from baseline to both post-qualification and post-training. While stressful training and overnight shifts are regularly encountered in specialist police work, other tasks, such as serving in Directing Staff (DS) roles on selection courses for future candidates are also important duties and present as a nexus between operations and selection. Thus, Chapter 10 considered the critical operational role of DS cadre. The purpose of this study was to monitor and analyse the HRV of one DS member during their 24-hour shift on a candidate selection course. The findings of this case study suggested that DS may be subject to stress levels not unlike those of candidates. This is of note as selection courses are highly taxing and arduous, and officers may serve as DS on more than one course per year and still be required to perform their operational duties.DS requirements during selection courses should therefore be considered appropriately in the overall deployment and operational task scheduling paradigm.Each previous chapter considered important elements of service in a specialist police organisation. The final chapter (Chapter 11) summated the findings from this programme of study, contextualised the works in terms of the bodies of literature with which they were most associated, and highlighted overall limitations as well as plausible future directions. A final supplementary chapter, aimed to provide an operational guide for utilising HRV data in tactical settings, contributed to further support translation of research to practice. In this supplementary chapter, shortcomings of using HRV were reviewed and solutions to avoid flawed analysis provided, as are the key lessons learned from this thesis. In essence, the presentation and visualisation of HRV data may be as critical to the application of HRV analysis as the measurements themselves in tactical settings

Psychophysiology in the digital age

The research I performed for my thesis revolved around the question how affect-physiology dynamics can be best measured in daily life. In my thesis I focused on three aspects of this question: 1) Do wearable wristband devices have sufficient validity to capture ANS activity? 2) To what extent is the laboratory design suitable to measure affect-ANS dynamics? 3) Are the affect-ANS dynamics subject to individual differences, both in the laboratory and in daily life? In chapter 2, I validated a shortened version of the Sing-a-Song Stress (SSST) test, the SSSTshort. The purpose of this test is to create social-evaluative stress in participants through a simple and brief design that does not require the involvement of multiple confederates. The results indicated that the SSSTshort was effective in inducing ANS and affective reactivity. This makes the SSSTshort a cost-effective alternative to the well-known Trier-Social-Stress task (TSST), which can be easily incorporated into large-scale studies to expand the range of stress types that can be studied in laboratory designs. In chapter 3, I validated a new wrist worn technology for measuring electrodermal activity (EDA). As expected, the overall EDA levels measured on the wrist were lower than those measured on the palm, likely due to the lower density of sweat glands on the wrist. The analysis demonstrated that the frequency measure of non-specific skin conductance response (ns.SCR) was superior to the commonly used measure of skin conductance level (SCL) for both the palm and wrist. The wrist-based ns.SCR measure was sensitive to the experimental manipulations and showed similar correspondence to the pre-ejection period (PEP) as palm-based ns.SCR. Moreover, wrist-based ns.SCR demonstrated similar predictive validity for affective state as PEP. However, the predictive validity of both wrist-based ns.SCR and PEP was lower compared to palm-based ns.SCR. These findings suggest that wrist-based ns.SCR EDA parameter has a promising future for use in psychophysiological research. In Chapter 4 of my thesis, I conducted the first study to directly compare the relationship between affect and ANS activity in a laboratory setting to that in daily life. To elicit stress in the laboratory, four different stress paradigms were employed, while stressful events in daily life were left to chance. In both settings, a valence and arousal scale was constructed from a nine-item affect questionnaire, and ANS activity was collected using the same devices. Data was collected from a single population, and the affect-ANS dynamics were analyzed using the same methodology for both laboratory and daily life settings. The results showed a remarkable similarity between the laboratory and daily life affect-ANS relationships. In Chapter 5 of my thesis, I investigated the influence of individual differences in physical activity and aerobic fitness on ANS and affective stress reactivity. Previous research has yielded inconsistent results due to heterogeneity issues in the population studied, stressor type, and the way fitness was measured. My study made a unique contribution to this field by measuring physical activity in three ways: 1) as objective aerobic fitness, 2) leisure time exercise behavior, and 3) total moderate-to-vigorous exercise (including both exercise and all other regular physical activity behaviors). In addition, we measured the physiological and affective stress response in both a laboratory and daily life setting. The total amount of physical activity showed more relationships with stress reactivity compared to exercise behavior alone, suggesting that future research should include a total physical activity variable. Our results did not support the cross-stressor adaptation hypotheses, suggesting that if exercise has a stress-reducing effect, it is unlikely to be mediated by altered ANS regulation due to repeated exposure to physical stress

Heart rate variability in marketing research: A systematic review and methodological perspectives

Heart rate variability is a promising physiological measurement that accesses psychophysiological variations in response to a marketing stimulus. While its application spans diverse fields, there is a limited understanding of the usability and interpretation of heart rate variability in marketing research. Therefore, this hybrid literature review provides an overview of the emerging use of heart rate variability in marketing research, along with essential methodological considerations. In this context, we blend marketing mix framework with stimulus-organism-response theory, segregating the use of heart rate variability in various marketing research contexts. We follow the preferred reporting items for systematic reviews and meta-analyses (PRISMA) framework to reflect on 33 records obtained from six databases. Our findings suggest that 42% of studies used heart rate variability to investigate promotion-related topics. Overall, heart rate variability is mostly used in combination with Galvanic skin response (48%). Further, 39% of studies used non-portable systems for data collection. Last, using the theory characteristics methodology (TCM) framework, we identified six research avenues: (1) affective, cognitive, and sensorial constructs; (2) personality, thinking style, and demographics; (3) product experience; (4) advertising and branding; (5) correlation with immersive technologies; and (6) triangulation with other neurophysiological tools

Understanding disease through remote monitoring technology:A mobile health perspective on disease and diagnosis in three conditions: stress, epilepsy, and COVID-19

Mobile systems and wearable technology have developed substantially over the last decade and provide a unique long-term and continuous insight and monitoring into medical condi- tions in health research. The opportunities afforded by mobile health in access, scale, and round-the-clock recording are counterbalanced by pronounced issues in areas like participant engagement, labelling, and dataset size. Throughout this thesis the different aspects of an mHealth study are addressed, from software development and study design to data collection and analysis. Three medically relevant fields are investigated: detection of stress from physiological signals, seizure detection in epilepsy and the characterisation and monitoring of COVID-19 through mobile health techniques.The first two analytical chapters of the thesis focus on models for acute stress and epileptic seizure detection, two conditions with autonomic and physiological manifestations. Firstly, a multi-modal machine learning pipeline is developed targetting focal and general motor seizures in patients with epilepsy. The heterogenity and inter-individual differences present in this study motivated the investigation of methods to personalise models with relatively little data. I subsequently consider meta-learning for few-shot model personalisation within acute stress classification, finding increased performance compared to standard methods.As the COVID-19 pandemic gripped the world the work of this thesis reoriented around using mHealth to understand the disease. Firstly, the study design and software development of Covid Collab, a crowdsourced, remote-enrollment COVID-19 study, are examined. Within these chapters, the patterns of participant enrolment and adherence in Covid Col- lab are also considered. Adherence could impact scientific interpretations if not properly accounted for. While basic drop-out and percent completion are often considered, a more dynamic view of a participant’s behaviour can also be important. A hidden Markov model approach is used to compare participant engagement over time.Secondly, the long-term effects of COVID are investigated through data collected in the Covid Collab study, giving insight into prevalence, risk factors, and symptom manifestation with respect to wearable-recorded physiological signals. Long-term and historical data accessed retrospectively facilitated the findings of significant correlations between development of long-COVID and mHealth-derived fitness and behaviour

Machine learning approaches for predicting sleep arousal response based on heart rate variability, oxygen saturation, and body profiles.

OBJECTIVE: Obstructive sleep apnea is a global health concern, and several tools have been developed to screen its severity. However, most tools focus on respiratory events instead of sleep arousal, which can also affect sleep efficiency. This study employed easy-to-measure parameters-namely heart rate variability, oxygen saturation, and body profiles-to predict arousal occurrence. METHODS: Body profiles and polysomnography recordings were collected from 659 patients. Continuous heart rate variability and oximetry measurements were performed and then labeled based on the presence of sleep arousal. The dataset, comprising five body profiles, mean heart rate, six heart rate variability, and five oximetry variables, was then split into 80% training/validation and 20% testing datasets. Eight machine learning approaches were employed. The model with the highest accuracy, area under the receiver operating characteristic curve, and area under the precision recall curve values in the training/validation dataset was applied to the testing dataset and to determine feature importance. RESULTS: InceptionTime, which exhibited superior performance in predicting sleep arousal in the training dataset, was used to classify the testing dataset and explore feature importance. In the testing dataset, InceptionTime achieved an accuracy of 76.21%, an area under the receiver operating characteristic curve of 84.33%, and an area under the precision recall curve of 86.28%. The standard deviations of time intervals between successive normal heartbeats and the square roots of the means of the squares of successive differences between normal heartbeats were predominant predictors of arousal occurrence. CONCLUSIONS: The established models can be considered for screening sleep arousal occurrence or integrated in wearable devices for home-based sleep examination