Affective Man-Machine Interface: Unveiling human emotions through biosignals

As is known for centuries, humans exhibit an electrical profile. This profile is altered through various psychological and physiological processes, which can be measured through biosignals; e.g., electromyography (EMG) and electrodermal activity (EDA). These biosignals can reveal our emotions and, as such, can serve as an advanced man-machine interface (MMI) for empathic consumer products. However, such a MMI requires the correct classification of biosignals to emotion classes. This chapter starts with an introduction on biosignals for emotion detection. Next, a state-of-the-art review is presented on automatic emotion classification. Moreover, guidelines are presented for affective MMI. Subsequently, a research is presented that explores the use of EDA and three facial EMG signals to determine neutral, positive, negative, and mixed emotions, using recordings of 21 people. A range of techniques is tested, which resulted in a generic framework for automated emotion classification with up to 61.31% correct classification of the four emotion classes, without the need of personal profiles. Among various other directives for future research, the results emphasize the need for parallel processing of multiple biosignals

Preliminary study for the measurement of Biosignals in Driving Simulators

openThis preliminary study focuses on the goal of developing and testing a setup and method for non-invasive monitoring of individuals using biosensors in a professional driving simulator (VI-grade Compact Simulator). This involves the synchronization and integration of hardware and software components. To detect the emotional and cognitive state of the driver, it is crucial to identify which signals provide reliable information about their condition. The objective of this study is to observe individuals in a controlled and repeatable environment designed to stimulate cognitive workload. This was achieved using a multimodal assessment method (iMotions), which includes eye tracking, galvanic skin response (GSR), electromyography (EMG), and respiration measurements, all conducted during two distinct controlled driving simulation scenarios. Four healthy subjects (average age = 24, standard deviation = ±2) were monitored during the first scenario, a highway with repeated emergency maneuvers (slalom through cones and double lane change), and the second, five laps of the Paul Ricard circuit. All of this for a total duration of approximately 20 minutes. The participants were not aware that the scenarios were designed to provoke different reactions. This experimental thesis aims to be the continuation and evolution of a testing phase previously conducted during an internship at iMotions, a company that develops multimodal streaming software and distributes commercial hardware. The hardware was supplied to the NAVLAB at the University of Padua, where the simulator is located. The results obtained, at first analysis, appear to be consistent with the literature, suggesting that a multimodal approach to physiological signals may characterize emotional and cognitive states in driving scenarios.This preliminary study focuses on the goal of developing and testing a setup and method for non-invasive monitoring of individuals using biosensors in a professional driving simulator (VI-grade Compact Simulator). This involves the synchronization and integration of hardware and software components. To detect the emotional and cognitive state of the driver, it is crucial to identify which signals provide reliable information about their condition. The objective of this study is to observe individuals in a controlled and repeatable environment designed to stimulate cognitive workload. This was achieved using a multimodal assessment method (iMotions), which includes eye tracking, galvanic skin response (GSR), electromyography (EMG), and respiration measurements, all conducted during two distinct controlled driving simulation scenarios. Four healthy subjects (average age = 24, standard deviation = ±2) were monitored during the first scenario, a highway with repeated emergency maneuvers (slalom through cones and double lane change), and the second, five laps of the Paul Ricard circuit. All of this for a total duration of approximately 20 minutes. The participants were not aware that the scenarios were designed to provoke different reactions. This experimental thesis aims to be the continuation and evolution of a testing phase previously conducted during an internship at iMotions, a company that develops multimodal streaming software and distributes commercial hardware. The hardware was supplied to the NAVLAB at the University of Padua, where the simulator is located. The results obtained, at first analysis, appear to be consistent with the literature, suggesting that a multimodal approach to physiological signals may characterize emotional and cognitive states in driving scenarios

The Application of Physiological Metrics in Validating User Experience Evaluation on Automotive Human Machine Interface Systems

Automotive in-vehicle information systems have seen an era of continuous development within the industry and are recognised as a key differentiator for prospective customers. This presents a significant challenge for designers and engineers in producing effective next generation systems which are helpful, novel, exciting, safe and easy to use. The usability of any new human machine interface (HMI) has an implicit cost in terms of the perceived aesthetic perception and associated user experience. Achieving the next engaging automotive interface, not only has to address the user requirements but also has to incorporate established safety standards whilst considering new interaction technologies. An automotive (HMI) evaluation may combine a triad of physiological, subjective and performance-based measurements which are employed to provide relevant and valuable data for product evaluation. However, there is also a growing interest and appreciation that determining real-time quantitative metrics to drivers’ affective responses provide valuable user affective feedback. The aim of this research was to explore to what extent physiological metrics such as heart rate variability could be used to quantify or validate subjective testing of automotive HMIs. This research employed both objective and subjective metrics to assess user engagement during interactions with an automotive infotainment system. The mapping of both physiological and self-report scales was examined over a series of studies in order to provide a greater understanding of users’ responses. By analysing the data collected it may provide guidance within the early stages of in-vehicle design evaluation in terms of usability and user satisfaction. This research explored these metrics as an objective, quantitative, diagnostic measure of affective response, in the assessment of HMIs. Development of a robust methodology was constructed for the application and understanding of these metrics. Findings from the three studies point towards the value of using a combination of methods when examining user interaction with an in-car HMI. For the next generation of interface systems, physiological measures, such as heart rate variability may offer an additional dimension of validity when examining the complexities of the driving task that drivers perform every day. There appears to be no boundaries on technology advancements and with this, comes extra pressure for car manufacturers to produce similar interactive and connective devices to those that are already in use in homes. A successful in-car HMI system will be intuitive to use, aesthetically pleasing and possess an element of pleasure however, the design components that are needed for a highly usable HMI have to be considered within the context of the constraints of the manufacturing process and the risks associated with interacting with an in-car HMI whilst driving. The findings from the studies conducted in this research are discussed in relation to the usability and benefits of incorporating physiological measures that can assist in our understanding of driver interaction with different automotive HMIs

Evaluating Mental Stress Among College Students Using Heart Rate and Hand Acceleration Data Collected from Wearable Sensors

Stress is various mental health disorders including depression and anxiety among college students. Early stress diagnosis and intervention may lower the risk of developing mental illnesses. We examined a machine learning-based method for identification of stress using data collected in a naturalistic study utilizing self-reported stress as ground truth as well as physiological data such as heart rate and hand acceleration. The study involved 54 college students from a large campus who used wearable wrist-worn sensors and a mobile health (mHealth) application continuously for 40 days. The app gathered physiological data including heart rate and hand acceleration at one hertz frequency. The application also enabled users to self-report stress by tapping on the watch face, resulting in a time-stamped record of the self-reported stress. We created, evaluated, and analyzed machine learning algorithms for identifying stress episodes among college students using heart rate and accelerometer data. The XGBoost method was the most reliable model with an AUC of 0.64 and an accuracy of 84.5%. The standard deviation of hand acceleration, standard deviation of heart rate, and the minimum heart rate were the most important features for stress detection. This evidence may support the efficacy of identifying patterns in physiological reaction to stress using smartwatch sensors and may inform the design of future tools for real-time detection of stress

Estimating and Visualizing Drivers’ Emotions Using the Internet of Digital Reality

Recently, the development of self-driving technology has progressed rapidly. However, self-driving cars have not yet become widespread. Thus, with an aging population, accidents such as road rage and acceleration and brake accidents are likely to continue. Stress is one key reason for such dangerous driving. Thus, technologies must be developed to provide mental support to drivers as required. In this study, we considered estimating driver emotions as a first step along these lines. To this end, we developed a technology to estimate emotions by collecting data on biological signals such as brain waves, heart rate, body movement, and data on a driver's operating status while they are driving. In addition, we introduce a Positive and Negative Affect Schedule (PANAS) to express the psychological states experienced by drivers. We further present the results of an analysis of data on a driver's emotions from PANAS and data obtained from electroencephalogram (EEG) readings and other biological signals from a car. In addition, the relationship between this experimental environment and the Internet of Digital Reality (IoD) is described

Experience with Using the Sensewear BMS Sensor System in the Context of a Health and Wellbeing Application

An assessment of a sensor designed for monitoring energy expenditure, activity, and sleep was conducted in the context of a research project which develops a weight management application. The overall goal of this project is to affect sustainable behavioural change with respect to diet and exercise in order to improve health and wellbeing. This paper reports results of a pretrial in which three volunteers wore the sensor for a total of 11 days. The aim was to gain experience with the sensor and determine if it would be suitable for incorporation into the ICT system developed by the project to be trialled later on a larger population. In this paper we focus mainly on activity monitoring and user experience. Data and results including visualizations and reports are presented and discussed. User experience proved positive in most respects. Exercise levels and sleep patterns correspond to user logs relating to exercise sessions and sleep patterns. Issues raised relate to accuracy, one source of possible interference, the desirability of enhancing the system with real-time data transmission, and analysis to enable real-time feedback. It is argued that automatic activity classification is needed to properly analyse and interpret physical activity data captured by accelerometry

Leveraging user-related internet of things for continuous authentication: a survey

Among all Internet of Things (IoT) devices, a subset of them are related to users. Leveraging these user-related IoT elements, itis possible to ensure the identity of the user for a period of time, thus avoiding impersonation. This need is known as ContinuousAuthentication (CA). Since 2009, a plethora of IoT-based CA academic research and industrial contributions have been proposed. Weoffer a comprehensive overview of 58 research papers regarding the main components of such a CA system. The status of the industryis studied as well, covering 32 market contributions, research projects and related standards. Lessons learned, challenges and openissues to foster further research in this area are finally presented.This work was supported by the MINECO grant TIN2016-79095-C2-2-R (SMOG-DEV) and by the CAM grants S2013/ICE-3095 (CIBERDINE) and P2018/TCS4566 (CYNAMON-CM) both co-funded with European FEDER funds