Affective Man-Machine Interface: Unveiling human emotions through biosignals

As is known for centuries, humans exhibit an electrical profile. This profile is altered through various psychological and physiological processes, which can be measured through biosignals; e.g., electromyography (EMG) and electrodermal activity (EDA). These biosignals can reveal our emotions and, as such, can serve as an advanced man-machine interface (MMI) for empathic consumer products. However, such a MMI requires the correct classification of biosignals to emotion classes. This chapter starts with an introduction on biosignals for emotion detection. Next, a state-of-the-art review is presented on automatic emotion classification. Moreover, guidelines are presented for affective MMI. Subsequently, a research is presented that explores the use of EDA and three facial EMG signals to determine neutral, positive, negative, and mixed emotions, using recordings of 21 people. A range of techniques is tested, which resulted in a generic framework for automated emotion classification with up to 61.31% correct classification of the four emotion classes, without the need of personal profiles. Among various other directives for future research, the results emphasize the need for parallel processing of multiple biosignals

An EMG Gesture Recognition System with Flexible High-Density Sensors and Brain-Inspired High-Dimensional Classifier

EMG-based gesture recognition shows promise for human-machine interaction. Systems are often afflicted by signal and electrode variability which degrades performance over time. We present an end-to-end system combating this variability using a large-area, high-density sensor array and a robust classification algorithm. EMG electrodes are fabricated on a flexible substrate and interfaced to a custom wireless device for 64-channel signal acquisition and streaming. We use brain-inspired high-dimensional (HD) computing for processing EMG features in one-shot learning. The HD algorithm is tolerant to noise and electrode misplacement and can quickly learn from few gestures without gradient descent or back-propagation. We achieve an average classification accuracy of 96.64% for five gestures, with only 7% degradation when training and testing across different days. Our system maintains this accuracy when trained with only three trials of gestures; it also demonstrates comparable accuracy with the state-of-the-art when trained with one trial

Prerequisites for Affective Signal Processing (ASP) - Part V: A response to comments and suggestions

In four papers, a set of eleven prerequisites for affective signal processing (ASP) were identified (van den Broek et al., 2010): validation, triangulation, a physiology-driven approach, contributions of the signal processing community, identification of users, theoretical specification, integration of biosignals, physical characteristics, historical perspective, temporal construction, and real-world baselines. Additionally, a review (in two parts) of affective computing was provided. Initiated by the reactions on these four papers, we now present: i) an extension of the review, ii) a post-hoc analysis based on the eleven prerequisites of Picard et al.(2001), and iii) a more detailed discussion and illustrations of temporal aspects with ASP

Biosensing and Actuation—Platforms Coupling Body Input-Output Modalities for Affective Technologies

Research in the use of ubiquitous technologies, tracking systems and wearables within mental health domains is on the rise. In recent years, affective technologies have gained traction and garnered the interest of interdisciplinary fields as the research on such technologies matured. However, while the role of movement and bodily experience to affective experience is well-established, how to best address movement and engagement beyond measuring cues and signals in technology-driven interactions has been unclear. In a joint industry-academia effort, we aim to remodel how affective technologies can help address body and emotional self-awareness. We present an overview of biosignals that have become standard in low-cost physiological monitoring and show how these can be matched with methods and engagements used by interaction designers skilled in designing for bodily engagement and aesthetic experiences. Taking both strands of work together offers unprecedented design opportunities that inspire further research. Through first-person soma design, an approach that draws upon the designer’s felt experience and puts the sentient body at the forefront, we outline a comprehensive work for the creation of novel interactions in the form of couplings that combine biosensing and body feedback modalities of relevance to affective health. These couplings lie within the creation of design toolkits that have the potential to render rich embodied interactions to the designer/user. As a result we introduce the concept of “orchestration”. By orchestration, we refer to the design of the overall interaction: coupling sensors to actuation of relevance to the affective experience; initiating and closing the interaction; habituating; helping improve on the users’ body awareness and engagement with emotional experiences; soothing, calming, or energising, depending on the affective health condition and the intentions of the designer. Through the creation of a range of prototypes and couplings we elicited requirements on broader orchestration mechanisms. First-person soma design lets researchers look afresh at biosignals that, when experienced through the body, are called to reshape affective technologies with novel ways to interpret biodata, feel it, understand it and reflect upon our bodies

Beyond Biometrics

Throughout the last 40 years, the essence of automated identification of users has remained the same. In this article, a new class of biometrics is proposed that is founded on processing biosignals, as opposed to images. After a brief introduction on biometrics, biosignals are discussed, including their advantages, disadvantages, and guidelines for obtaining them. This new class of biometrics increases biometrics’ robustness and enables cross validation. Next, biosignals’ use is illustrated by two biosignal-based biometrics: voice identification and handwriting recognition. Additionally, the concept of a digital human model is introduced. Last, some issues will be touched upon that will arise when biosignal-based biometrics are brought to practice

Methods for enhanced learning using wearable technologies. A study of the maritime sector

Maritime safety is a critical concern due to the potential for serious consequences or accidents for the crew, passengers, environment, and assets resulting from navigation errors or unsafe acts. Traditional training methods face challenges in the rapidly evolving maritime industry, and innovative training methods are being explored. This study explores the use of wearable sensors with biosignal data collection to improve training performance in the maritime sector. Three experiments were conducted progressively to investigate the relationship between navigators' experience levels and biosignal data results, the effects of different training methods on cognitive workload, trainees' stress levels, and their decision-making skills, and the classification of scenario complexity and the biosignal data obtained by the trainees. questionnaire data on stress levels, workload, and user satisfaction of auxiliary training equipment; performance evaluation data on navigational abilities, decision-making skills, and ship-handling abilities; and biosignal data, including electrodermal activity (EDA), body temperature, blood volume pulse (BVP), inter-beat interval (IBI), and heart rate (HR). Several statistical methods and machine-learning algorithms were used in the data analysis. The present dissertation contributes to the advancement of the field of maritime education and training by exploring methods for enhancing learning in complex situations. The use of biosignal data provides insights into the interplay between stress levels and training outcomes in the maritime industry. The proposed conceptual training model underscores the relationship between trainees' stress and safety factors and offers a framework for the development and evaluation of advanced biosignal data-based training systems

New visualization model for large scale biosignals analysis

Benefits of long-term monitoring have drawn considerable attention in healthcare. Since the acquired data provides an important source of information to clinicians and researchers, the choice for long-term monitoring studies has become frequent. However, long-term monitoring can result in massive datasets, which makes the analysis of the acquired biosignals a challenge. In this case, visualization, which is a key point in signal analysis, presents several limitations and the annotations handling in which some machine learning algorithms depend on, turn out to be a complex task. In order to overcome these problems a novel web-based application for biosignals visualization and annotation in a fast and user friendly way was developed. This was possible through the study and implementation of a visualization model. The main process of this model, the visualization process, comprised the constitution of the domain problem, the abstraction design, the development of a multilevel visualization and the study and choice of the visualization techniques that better communicate the information carried by the data. In a second process, the visual encoding variables were the study target. Finally, the improved interaction exploration techniques were implemented where the annotation handling stands out. Three case studies are presented and discussed and a usability study supports the reliability of the implemented work

Biosignal‐based human–machine interfaces for assistance and rehabilitation : a survey

As a definition, Human–Machine Interface (HMI) enables a person to interact with a device. Starting from elementary equipment, the recent development of novel techniques and unobtrusive devices for biosignals monitoring paved the way for a new class of HMIs, which take such biosignals as inputs to control various applications. The current survey aims to review the large literature of the last two decades regarding biosignal‐based HMIs for assistance and rehabilitation to outline state‐of‐the‐art and identify emerging technologies and potential future research trends. PubMed and other databases were surveyed by using specific keywords. The found studies were further screened in three levels (title, abstract, full‐text), and eventually, 144 journal papers and 37 conference papers were included. Four macrocategories were considered to classify the different biosignals used for HMI control: biopotential, muscle mechanical motion, body motion, and their combinations (hybrid systems). The HMIs were also classified according to their target application by considering six categories: prosthetic control, robotic control, virtual reality control, gesture recognition, communication, and smart environment control. An ever‐growing number of publications has been observed over the last years. Most of the studies (about 67%) pertain to the assistive field, while 20% relate to rehabilitation and 13% to assistance and rehabilitation. A moderate increase can be observed in studies focusing on robotic control, prosthetic control, and gesture recognition in the last decade. In contrast, studies on the other targets experienced only a small increase. Biopotentials are no longer the leading control signals, and the use of muscle mechanical motion signals has experienced a considerable rise, especially in prosthetic control. Hybrid technologies are promising, as they could lead to higher performances. However, they also increase HMIs’ complex-ity, so their usefulness should be carefully evaluated for the specific application

Prerequisites for Affective Signal Processing (ASP) - Part III

This is the third part in a series on prerequisites for affective signal processing (ASP). So far, six prerequisites were identified: validation (e.g., mapping of constructs on signals), triangulation, a physiology-driven approach, and contributions of the signal processing community (van den Broek et al., 2009) and identification of users and theoretical specification (van den Broek et al., 2010). Here, two additional prerequisites are identified: integration of biosignals, and physical characteristics

Personality assessment based on biosignals during a decision-making task

Due to the emergence of novel acquisition devices and signal processing techniques, the study of electrophysiology and its applications has assumed an important role on the Biomedical Engineering community. Recently, research on this area has expanded to several domains, with the psychophysiology being a proeminent one, more specifically in the field of personality psychology. In this thesis, participants were asked to perform a wildly known decision-making task, the Iowa Gambling Task (IGT), and their biosignals were recorded during this performance with the objective of determining whether changes in biosignals could be related to personality. This project was composed by 71 participants and their biosignals were used to extract meaningful features that together could create a predictive model of personality. For this, all biosignals were processed prior to the feature extraction step and the features were extracted from the entire signals, recorded during the performance of the IGT, and also dividing the task in five blocks. After the extraction, a machine learning algorithm was used to compute the best predictive models for the Five Factor Model (FFM) personality dimensions and for the Maximization and Regret scales, using each biosignal individually and in the end all features from all biosignals. The results showed that the predictive models which use features from all biosignals perform better than the models which use only one biosignal. The Openness to Experience, Agreeableness and Maximization scales are well predicted with features from Electrocardiogram (ECG), the Agreeableness, Maximization and Extraversion scales with Electrodermal Activity (EDA) features and the Extraversion and Openness to Experience scales with features from Blood Volume Pulse (BVP). The hypothesis that personality traits is more expressed in the start of IGT was confirmed since the highest number of features is extracted from the Block 1 of the IGT. The results should be further validated for other populations