Robust Modeling of Epistemic Mental States

This work identifies and advances some research challenges in the analysis of facial features and their temporal dynamics with epistemic mental states in dyadic conversations. Epistemic states are: Agreement, Concentration, Thoughtful, Certain, and Interest. In this paper, we perform a number of statistical analyses and simulations to identify the relationship between facial features and epistemic states. Non-linear relations are found to be more prevalent, while temporal features derived from original facial features have demonstrated a strong correlation with intensity changes. Then, we propose a novel prediction framework that takes facial features and their nonlinear relation scores as input and predict different epistemic states in videos. The prediction of epistemic states is boosted when the classification of emotion changing regions such as rising, falling, or steady-state are incorporated with the temporal features. The proposed predictive models can predict the epistemic states with significantly improved accuracy: correlation coefficient (CoERR) for Agreement is 0.827, for Concentration 0.901, for Thoughtful 0.794, for Certain 0.854, and for Interest 0.913.Comment: Accepted for Publication in Multimedia Tools and Application, Special Issue: Socio-Affective Technologie

Expression

Limited access to non-verbal cues hinders the dyadic conversation or social interaction of people who are blind or visually impaired. This paper presents Expression-an integrated assistive solution using Google Glass. The key function of the system is to enable the user to perceive social signals during a natural face-to-face conversation. Empirical evaluation of the system is presented with qualitative (Likert score: 4.383/5) and quantitative results (overall F-measure of the nonverbal expression recognition: 0.768). Copyright 2014 978-1-4503-3047-3/14/09

IMAPS: A smart phone based real-time framework for prediction of affect in natural dyadic conversation

The lack of ability to perceive emotions and affective states is a setback for people who are blind or visually impaired in professional and social communications. Towards developing assistive technology solution in facilitating natural dyadic conversations for people with such disability, this paper describes the development of a smart phone based system called interactive mobile affect perception system (iMAPS) for prediction of affective dimensions (valence-arousal-dominance). The proposed solution utilizes an Android platform in conjunction with a wireless network to build a fully integrated iMAPS. Empirical analyses were conducted to measure the efficacy and utility of the proposed solution. It was found that the proposed framework can predict affect dimensions with good accuracy (Maximum Correlation Coefficient for valence: 0.68, arousal: 0.71, and dominance: 0.67) in natural dyadic conversation. The overall minimum and maximum response times are (181 milliseconds) and (500 milliseconds), respectively. © 2012 IEEE

Robust modeling of epistemic mental states

This work identifies and advances some research challenges in the analysis of facial features and their temporal dynamics with epistemic mental states in dyadic conversations. Epistemic states are: Agreement, Concentration, Thoughtful, Certain, and Interest. In this paper, we perform a number of statistical analyses and simulations to identify the relationship between facial features and epistemic states. Non-linear relations are found to be more prevalent, while temporal features derived from original facial features have demonstrated a strong correlation with intensity changes. Then, we propose a novel prediction framework that takes facial features and their nonlinear relation scores as input and predict different epistemic states in videos. The prediction of epistemic states is boosted when the classification of emotion changing regions such as rising, falling, or steady-state are incorporated with the temporal features. The proposed predictive models can predict the epistemic states with significantly improved accuracy: correlation coefficient (CoERR) for Agreement is 0.827, for Concentration 0.901, for Thoughtful 0.794, for Certain 0.854, and for Interest 0.913

E m o A s s i s t: emotion enabled assistive tool to enhance dyadic conversation for the blind

This paper presents the design and implementation of EmoAssist: a smart-phone based system to assist in dyadic conversations. The main goal of the system is to provide access to more non-verbal communication options to people who are blind or visually impaired. The key functionalities of the system are to predict behavioral expressions (such a yawn, a closed lip smile, a open lip smile, looking away, sleepy, etc.) and 3-D affective dimensions (valence, arousal, and dominance) from visual cues in order to provide the correct auditory feedback or response. A number of challenges related to the data communication protocols, efficient tracking of the face, modeling of behavioral expressions/affective dimensions, feedback mechanism and system integration were addressed to build an effective and functional system. In addition, orientation-sensor information from the smart-phone was used to correct image alignment to improve the robustness for real world application. Empirical studies show that the EmoAssist can predict affective dimensions with acceptable accuracy (Maximum Correlation-Coefficient for valence: 0.76, arousal: 0.78, and dominance: 0.76) in natural dyadic conversation. The overall minimum and maximum response-times are (64.61 milliseconds) and (128.22 milliseconds), respectively. The integration of sensor information for correcting the orientation improved (16 % in average) the accuracy in recognizing behavioralexpressions. A usability study with ten blind people in social interaction shows that the EmoAssist is highly acceptable with an Average acceptability rating using of 6.0 in Likert scale (where 1 and 7 are the lowest and highest possible ratings, respectively)