A Review of Psychophysiological Measures to Assess Cognitive States in Real-World Driving

As driving functions become increasingly automated, motorists run the risk of becoming cognitively removed from the driving process. Psychophysiological measures may provide added value not captured through behavioral or self-report measures alone. This paper provides a selective review of the psychophysiological measures that can be utilized to assess cognitive states in real-world driving environments. First, the importance of psychophysiological measures within the context of traffic safety is discussed. Next, the most commonly used physiology-based indices of cognitive states are considered as potential candidates relevant for driving research. These include: electroencephalography and event-related potentials, optical imaging, heart rate and heart rate variability, blood pressure, skin conductance, electromyography, thermal imaging, and pupillometry. For each of these measures, an overview is provided, followed by a discussion of the methods for measuring it in a driving context. Drawing from recent empirical driving and psychophysiology research, the relative strengths and limitations of each measure are discussed to highlight each measures' unique value. Challenges and recommendations for valid and reliable quantification from lab to (less predictable) real-world driving settings are considered. Finally, we discuss measures that may be better candidates for a near real-time assessment of motorists' cognitive states that can be utilized in applied settings outside the lab. This review synthesizes the literature on in-vehicle psychophysiological measures to advance the development of effective human-machine driving interfaces and driver support systems

Ubiquitous Technologies for Emotion Recognition

Emotions play a very important role in how we think and behave. As such, the emotions we feel every day can compel us to act and influence the decisions and plans we make about our lives. Being able to measure, analyze, and better comprehend how or why our emotions may change is thus of much relevance to understand human behavior and its consequences. Despite the great efforts made in the past in the study of human emotions, it is only now, with the advent of wearable, mobile, and ubiquitous technologies, that we can aim to sense and recognize emotions, continuously and in real time. This book brings together the latest experiences, findings, and developments regarding ubiquitous sensing, modeling, and the recognition of human emotions

Cluster-analytic classification of facial expressions using infrared measurements of facial thermal features

In previous research, scientists were able to use transient facial thermal features extracted from Thermal Infra-Red Images (TIRIs) for making binary distinction between the affective states. For example, thermal asymmetries localised in facial TIRIs have been used to distinguish anxiety and deceit. Since affective human-computer interaction would require machines to distinguish between the subtle facial expressions of affective states, computers’ able to make such binary distinctions would not suffice a robust human-computer interaction. This work, for the first time, uses affective-state-specific transient facial thermal features extracted from TIRIs to recognise a much wider range of facial expressions under a much wider range of conditions. Using infrared thermal imaging within the 8-14 μm, a database of 324 discrete, time-sequential, visible-spectrum and thermal facial images was acquired, representing different facial expressions from 23 participants in different situations. A facial thermal feature extraction and pattern classification approach was developed, refined and tested on various Gaussian mixture models constructed using the image database. Attempts were made to classify: neutral and pretended happy and sad faces; multiple positive and negative facial expressions; six (pretended) basic facial expressions; partially covered or occluded faces; and faces with evoked happiness, sadness, disgust and anger. The cluster-analytic classification in this work began by segmentation and detection of thermal faces in the acquired TIRIs. The affective-state-specific temperature distributions on the facial skin surface were realised through the pixel grey-level analysis. Examining the affectivestate- specific temperature variations within the selected regions of interest in the TIRIs led to the discovery of some significant Facial Thermal Feature Points (FTFPs) along the major facial muscles. Following a multivariate analysis of the Thermal Intensity values (TIVs) measured at the FTFPs, the TIRIs were represented along the Principal Components (PCs) of a covariance matrix. The resulting PCs were ranked in the order of their effectiveness in the between-cluster separation. Only the most effective PCs were retained to construct an optimised eigenspace. A supervised learning algorithm was invoked for linear subdivision of the optimised eigenspace. The statistical significance levels of the classification results were estimated for validating the discriminant functions. The main contribution of this research has been to show that: the infrared imaging of facial thermal features within the 8-14 μm bandwidth may be used to observe affective-state-specific thermal variations on the face; the pixel-grey level analysis of TIRIs can help localise FTFPs along the major facial muscles of the face; cluster-analytic classification of transient thermal features may help distinguish between the facial expressions of affective states in an optimized eigenspace of input thermal feature vectors. The Gaussian mixture model with one cluster per affect worked better for some facial expressions than others. This made the influence of the Gaussian mixture model structure on the accuracy of the classification results obvious. However, the linear discrimination and confusion patterns observed in this work were consistent with the ones reported in several earlier studies. This investigation also unveiled some important dimensions of the future research on use of facial thermal features in affective human-computer interaction.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Quantifying the sense of presence in virtual reality using physiological data

Les mesures fisiològiques de l'experiència de joc (PX) durant el joc han anat incrementant en popularitat en les comunitats de recerca sobre jocs. La sensació de presència, que es refereix a la impressió psicològica d'estar en un entorn virtual, és vista com un factor principal influent en el PX. Aquesta tesi explora com les mesures fisiològiques hi estan relacionades mentre es juga a un joc de rol de realitat virtual (VR). En trobem una correlació negativa significativa amb la temperatura de la pell, la qual interpretem com a relacionada amb que la manca de confort tèrmic estigui associada amb nivells menors de sensació de presència. Un dispositiu wearable no invasiu comunament utilitzat per a mesures fisiológiques és la pulsera E4 d'Empatica, que ofereix múltiples mesures fisiològiques incloent l'activitat electrodèrmica, la freqüència cardíaca i la temperatura de la pell. Tanmateix, la integració de l'E4 amb motors de joc populars com Unity resulta difícil a causa d'obstacles com errors de programari crítics gens evidents a la llibreria i la limitada aplicabilitat de la documentació en el context de Unity. Així doncs presentem E4UnityIntegration-MIT, un plugin de Unity de codi obert dissenyat per a mitigar els reptes associats amb integrar l'E4 a projectes de Unity. El plugin exposa l'API de l'E4 permetent-ne la interacció amb scripts C# de Unity, habilitant així el recull i seguiment de dades en temps real, i proveeix la funcionalitat de guardar-les en un fitxer extern per permetre'n l'anàlisi. L'estudi presentat, que va dependre de E4UnityIntegration-MIT, alhora en serveix de validació.Las medidas fisiológicas de la experiencia de juego (PX) durante el juego han ido incrementando en popularidad en los círculos de investigación de juegos. La sensación de presencia, que se refiere a la impresión psicológica de estar en un entorno virtual, es vista como un factor principal influyente en el PX. Esta tesis explora como las medidas fisiológicas estan relacionadas con ella mientras se juega a un juego de rol de realidad virtual (VR). Hallamos una correlación negativa significativa con la temperatura de la piel, la cual interpretamos como relacionada con que el disconfort térmico esté asociado con niveles menores de sensación de presencia. Un dispositivo wearable comúnmente usado para medidas fisiológicas es la pulsera E4 de Empatica, que ofrece múltiples medidas fisiológicas incluyendo la actividad electrodérmica, la frecuencia cardíaca y la temperatura de la piel. Sin embargo, la integración del E4 con motores de juego populares como Unity resulta difícil a causa de obstáculos como errores de software críticos nada obvios en la librería y la limitada aplicabilidad de su documentación en el contexto de Unity. Así pues presentamos E4UnityIntegration-MIT, un plugin de Unity de código abierto diseñado para mitigar los retos asociados con integrar el E4 en proyectos de Unity. El plugin expone la API del E4 permitiendo su interacción con scripts de C# de Unity, habilitando así la recolección y el seguimiento de los datos en tiempo real, y provee la funcionalidad de guardarlos en un fichero externo para permitir su análisis. El estudio presentado, que dependió de E4UnityIntegration-MIT, sirve al mismo tiempo como su validación.Physiological measurement of player experience (PX) during gameplay has been of increasing interest within game research circles. Sense of presence, which refers to players’ psychological feeling of being in a virtual environment, is seen as a major factor influencing PX. This thesis work explores how physiological measurements relate to sense of presence while playing a virtual reality (VR) roleplaying game. We find a significant negative correlation with skin temperature, which we interpret as having to do with thermal discomfort being associated with lower levels of sense of presence in VR. A commonly-used non-invasive wearable device for physiological measurement is the Empatica E4 wristband, which offers multiple physiological metrics including electrodermal activity, heart rate and skin temperature. That said, the E4’s integration with popular game engines such as Unity proves difficult due to obstacles such as non-obvious critical bugs in the library and limited documentation applicability within the Unity context. We thus present E4UnityIntegration-MIT, an open-source Unity plugin designed to mitigate the challenges associated with integrating the E4 into Unity projects. The plugin exposes the E4’s API for interfacing with Unity C# scripts, thereby enabling realtime data collection and monitoring, and provides the affordance of saving the data in an external file for data analysis purposes. The study here presented, which relied on E4UnityIntegration-MIT, also serves as a validation study for the plugin.Outgoin