Emotions in context: examining pervasive affective sensing systems, applications, and analyses

Pervasive sensing has opened up new opportunities for measuring our feelings and understanding our behavior by monitoring our affective states while mobile. This review paper surveys pervasive affect sensing by examining and considering three major elements of affective pervasive systems, namely; “sensing”, “analysis”, and “application”. Sensing investigates the different sensing modalities that are used in existing real-time affective applications, Analysis explores different approaches to emotion recognition and visualization based on different types of collected data, and Application investigates different leading areas of affective applications. For each of the three aspects, the paper includes an extensive survey of the literature and finally outlines some of challenges and future research opportunities of affective sensing in the context of pervasive computing

Automatic Driver Fatigue Monitoring Using Hidden Markov Models and Bayesian Networks

The automotive industry is growing bigger each year. The central concern for any automotive company is driver and passenger safety. Many automotive companies have developed driver assistance systems, to help the driver and to ensure driver safety. These systems include adaptive cruise control, lane departure warning, lane change assistance, collision avoidance, night vision, automatic parking, traffic sign recognition, and driver fatigue detection. In this thesis, we aim to build a driver fatigue detection system that advances the research in this area. Using vision in detecting driver fatigue is commonly the key part for driver fatigue detection systems. We have decided to investigate different direction. We examine the driver's voice, heart rate, and driving performance to assess fatigue level. The system consists of three main modules: the audio module, the heart rate and other signals module, and the Bayesian network module. The audio module analyzes an audio recording of a driver and tries to estimate the level of fatigue for the driver. A Voice Activity Detection (VAD) module is used to extract driver speech from the audio recording. Mel-Frequency Cepstrum Coefficients, (MFCC) features are extracted from the speech signal, and then Support Vector Machines (SVM) and Hidden Markov Models (HMM) classifiers are used to detect driver fatigue. Both classifiers are tuned for best performance, and the performance of both classifiers is reported and compared. The heart rate and other signals module uses heart rate, steering wheel position, and the positions of the accelerator, brake, and clutch pedals to detect the level of fatigue. These signals' sample rates are then adjusted to match, allowing simple features to be extracted from the signals, and SVM and HMM classifiers are used to detect fatigue level. The performance of both classifiers is reported and compared. Bayesian networks' abilities to capture dependencies and uncertainty make them a sound choice to perform the data fusion. Prior information (Day/Night driving and previous decision) is also incorporated into the network to improve the final decision. The accuracies of the audio and heart rate and other signals modules are used to calculate certain CPTs for the Bayesian network, while the rest of the CPTs are calculated subjectively. The inference queries are calculated using the variable elimination algorithm. For those time steps where the audio module decision is absent, a window is defined and the last decision within this window is used as a current decision. The performance of the system is assessed based on the average accuracy per second. A dataset was built to train and test the system. The experimental results show that the system is very promising. The performance of the system was assessed based on the average accuracy per second; the total accuracy of the system is 90.5%. The system design can be easily improved by easily integrating more modules into the Bayesian network

Recognising Complex Mental States from Naturalistic Human-Computer Interactions

New advances in computer vision techniques will revolutionize the way we interact with computers, as they, together with other improvements, will help us build machines that understand us better. The face is the main non-verbal channel for human-human communication and contains valuable information about emotion, mood, and mental state. Affective computing researchers have investigated widely how facial expressions can be used for automatically recognizing affect and mental states. Nowadays, physiological signals can be measured by video-based techniques, which can also be utilised for emotion detection. Physiological signals, are an important indicator of internal feelings, and are more robust against social masking. This thesis focuses on computer vision techniques to detect facial expression and physiological changes for recognizing non-basic and natural emotions during human-computer interaction. It covers all stages of the research process from data acquisition, integration and application. Most previous studies focused on acquiring data from prototypic basic emotions acted out under laboratory conditions. To evaluate the proposed method under more practical conditions, two different scenarios were used for data collection. In the first scenario, a set of controlled stimulus was used to trigger the user’s emotion. The second scenario aimed at capturing more naturalistic emotions that might occur during a writing activity. In the second scenario, the engagement level of the participants with other affective states was the target of the system. For the first time this thesis explores how video-based physiological measures can be used in affect detection. Video-based measuring of physiological signals is a new technique that needs more improvement to be used in practical applications. A machine learning approach is proposed and evaluated to improve the accuracy of heart rate (HR) measurement using an ordinary camera during a naturalistic interaction with computer

A machine learning framework for automatic human activity classification from wearable sensors

Wearable sensors are becoming increasingly common and they permit the capture of physiological data during exercise, recuperation and everyday activities. This work investigated and advanced the current state-of-the-art in machine learning technology for the automatic classification of captured physiological data from wearable sensors. The overall goal of the work presented here is to research and investigate every aspect of the technology and methods involved in this field and to create a framework of technology that can be utilised on low-cost platforms across a wide range of activities. Both rudimentary and advanced techniques were compared, including those that allowed for both real-time processing on an android platform and highly accurate postprocessing on a desktop computer. State-of-the-art feature extraction methods such as Fourier and Wavelet analysis were also researched to ascertain how well they could extract discriminative physiological information. Various classifiers were investigated in terms of their ability to work with different feature extraction methods. Consequently, complex classification fusion models were created to increase the overall accuracy of the activity recognition process. Genetic algorithms were also employed to optimise classifier parameter selection in the multidimensional search space. Large annotated sporting activity datasets were created for a range of sports that allowed different classification models to be compared. This allowed for a machine learning framework to be constructed that could potentially create accurate models when applied to any unknown dataset. This framework was also successfully applied to medical and everyday-activity datasets confirming that the approach could be deployed in different application settings

Computer audition for emotional wellbeing

This thesis is focused on the application of computer audition (i. e., machine listening) methodologies for monitoring states of emotional wellbeing. Computer audition is a growing field and has been successfully applied to an array of use cases in recent years. There are several advantages to audio-based computational analysis; for example, audio can be recorded non-invasively, stored economically, and can capture rich information on happenings in a given environment, e. g., human behaviour. With this in mind, maintaining emotional wellbeing is a challenge for humans and emotion-altering conditions, including stress and anxiety, have become increasingly common in recent years. Such conditions manifest in the body, inherently changing how we express ourselves. Research shows these alterations are perceivable within vocalisation, suggesting that speech-based audio monitoring may be valuable for developing artificially intelligent systems that target improved wellbeing. Furthermore, computer audition applies machine learning and other computational techniques to audio understanding, and so by combining computer audition with applications in the domain of computational paralinguistics and emotional wellbeing, this research concerns the broader field of empathy for Artificial Intelligence (AI). To this end, speech-based audio modelling that incorporates and understands paralinguistic wellbeing-related states may be a vital cornerstone for improving the degree of empathy that an artificial intelligence has. To summarise, this thesis investigates the extent to which speech-based computer audition methodologies can be utilised to understand human emotional wellbeing. A fundamental background on the fields in question as they pertain to emotional wellbeing is first presented, followed by an outline of the applied audio-based methodologies. Next, detail is provided for several machine learning experiments focused on emotional wellbeing applications, including analysis and recognition of under-researched phenomena in speech, e. g., anxiety, and markers of stress. Core contributions from this thesis include the collection of several related datasets, hybrid fusion strategies for an emotional gold standard, novel machine learning strategies for data interpretation, and an in-depth acoustic-based computational evaluation of several human states. All of these contributions focus on ascertaining the advantage of audio in the context of modelling emotional wellbeing. Given the sensitive nature of human wellbeing, the ethical implications involved with developing and applying such systems are discussed throughout

Wireless Sensors for Brain Activity—A Survey

Over the last decade, the area of electroencephalography (EEG) witnessed a progressive move from high-end large measurement devices, relying on accurate construction and providing high sensitivity, to miniature hardware, more specifically wireless wearable EEG devices. While accurate, traditional EEG systems need a complex structure and long periods of application time, unwittingly causing discomfort and distress on the users. Given their size and price, aside from their lower sensitivity and narrower spectrum band(s), wearable EEG devices may be used regularly by individuals for continuous collection of user data from non-medical environments. This allows their usage for diverse, nontraditional, non-medical applications, including cognition, BCI, education, and gaming. Given the reduced need for standardization or accuracy, the area remains a rather incipient one, mostly driven by the emergence of new devices that represent the critical link of the innovation chain. In this context, the aim of this study is to provide a holistic assessment of the consumer-grade EEG devices for cognition, BCI, education, and gaming, based on the existing products, the success of their underlying technologies, as benchmarked by the undertaken studies, and their integration with current applications across the four areas. Beyond establishing a reference point, this review also provides the critical and necessary systematic guidance for non-medical EEG research and development efforts at the start of their investigation.</jats:p