Recognising Complex Mental States from Naturalistic Human-Computer Interactions

New advances in computer vision techniques will revolutionize the way we interact with computers, as they, together with other improvements, will help us build machines that understand us better. The face is the main non-verbal channel for human-human communication and contains valuable information about emotion, mood, and mental state. Affective computing researchers have investigated widely how facial expressions can be used for automatically recognizing affect and mental states. Nowadays, physiological signals can be measured by video-based techniques, which can also be utilised for emotion detection. Physiological signals, are an important indicator of internal feelings, and are more robust against social masking. This thesis focuses on computer vision techniques to detect facial expression and physiological changes for recognizing non-basic and natural emotions during human-computer interaction. It covers all stages of the research process from data acquisition, integration and application. Most previous studies focused on acquiring data from prototypic basic emotions acted out under laboratory conditions. To evaluate the proposed method under more practical conditions, two different scenarios were used for data collection. In the first scenario, a set of controlled stimulus was used to trigger the user’s emotion. The second scenario aimed at capturing more naturalistic emotions that might occur during a writing activity. In the second scenario, the engagement level of the participants with other affective states was the target of the system. For the first time this thesis explores how video-based physiological measures can be used in affect detection. Video-based measuring of physiological signals is a new technique that needs more improvement to be used in practical applications. A machine learning approach is proposed and evaluated to improve the accuracy of heart rate (HR) measurement using an ordinary camera during a naturalistic interaction with computer

Recognising Complex Mental States from Naturalistic Human-Computer Interactions

New advances in computer vision techniques will revolutionize the way we interact with computers, as they, together with other improvements, will help us build machines that understand us better. The face is the main non-verbal channel for human-human communication and contains valuable information about emotion, mood, and mental state. Affective computing researchers have investigated widely how facial expressions can be used for automatically recognizing affect and mental states. Nowadays, physiological signals can be measured by video-based techniques, which can also be utilised for emotion detection. Physiological signals, are an important indicator of internal feelings, and are more robust against social masking. This thesis focuses on computer vision techniques to detect facial expression and physiological changes for recognizing non-basic and natural emotions during human-computer interaction. It covers all stages of the research process from data acquisition, integration and application. Most previous studies focused on acquiring data from prototypic basic emotions acted out under laboratory conditions. To evaluate the proposed method under more practical conditions, two different scenarios were used for data collection. In the first scenario, a set of controlled stimulus was used to trigger the user’s emotion. The second scenario aimed at capturing more naturalistic emotions that might occur during a writing activity. In the second scenario, the engagement level of the participants with other affective states was the target of the system. For the first time this thesis explores how video-based physiological measures can be used in affect detection. Video-based measuring of physiological signals is a new technique that needs more improvement to be used in practical applications. A machine learning approach is proposed and evaluated to improve the accuracy of heart rate (HR) measurement using an ordinary camera during a naturalistic interaction with computer

Simulation of emotional behaviours for virtual agents with personalities

We have proposed an improved emotion model based on the well-known PAD model in affective computing and the NEO PI-R personality model in psychology. A novel parameter Emotion Intensity (EI) is proposed to represent different strength of anger, disgust, fear, happiness and sadness. Another novel Resistant formulation is also proposed to effectively simulate the complicated negative emotions. Eight experiments are conducted to simulate different emotional responses under different stimuli with different personality traits

The Effect of GED Candidate Race and Motivation Factors on Exam Outcomes

Earning a General Educational Development (GED) credential can have positive results in a student\u27s life including higher wages and better job opportunities. The 2014 version of the GED exam changed the format from a paper-based test to a computer-based test. This change coincided with a 35% decline in the pass rate indicating not all students are prepared to pass the new computer-based test (CBT). The purpose of this quantitative study was to evaluate the influence of a candidate\u27s race and reason for taking the exam on the pass or fail outcome of the new computer-based GED exam. The study used Vroom\u27s expectancy theory as the theoretical framework. The guiding question was to examine the relationship between a candidate\u27s motivation and pass or fail outcome of the CBT. This study used a quantitative approach to analyze available archival data from The Technical College System of Georgia in 2014 and 2015. Two chi-square analyses were conducted on data from 21,641 participants using candidate\u27s race, reason for taking the exam, and GED pass or fail outcome. Results suggested that individually, both a candidate\u27s race and reason for taking the test have a statistically significant effect on the participant\u27s pass or fail outcome. Results from this study may help GED educators and students better understand factors that can influence student success. Integrating career development orientations and remedial computer based technology classes into the GED preparation process were recommended. Implications for positive social change include the potential to increase student motivation, improve the preparedness of both students and educators and subsequently increase the number of people who pass the GED exam

Simulation of emotional behaviours for virtual agents with personalities

We have proposed an improved emotion model based on the well-known PAD model in affective computing and the NEO PI-R personality model in psychology. A novel parameter Emotion Intensity (EI) is proposed to represent different strength of anger, disgust, fear, happiness and sadness. Another novel Resistant formulation is also proposed to effectively simulate the complicated negative emotions. Eight experiments are conducted to simulate different emotional responses under different stimuli with different personality traits