Affective games:a multimodal classification system

Affective gaming is a relatively new field of research that exploits human emotions to influence gameplay for an enhanced player experience. Changes in player’s psychology reflect on their behaviour and physiology, hence recognition of such variation is a core element in affective games. Complementary sources of affect offer more reliable recognition, especially in contexts where one modality is partial or unavailable. As a multimodal recognition system, affect-aware games are subject to the practical difficulties met by traditional trained classifiers. In addition, inherited game-related challenges in terms of data collection and performance arise while attempting to sustain an acceptable level of immersion. Most existing scenarios employ sensors that offer limited freedom of movement resulting in less realistic experiences. Recent advances now offer technology that allows players to communicate more freely and naturally with the game, and furthermore, control it without the use of input devices. However, the affective game industry is still in its infancy and definitely needs to catch up with the current life-like level of adaptation provided by graphics and animation

Using fundamental frequency of cancer survivors’ speech to investigate emotional distress in out-patient visits

Objective Emotions, are in part conveyed by varying levels of fundamental frequency of voice pitch (f0). This study tests the hypothesis that patients display heightened levels of emotional arousal (f0) during Verona Coding Definitions of Emotional Sequences (VR-CoDES) cues and concerns versus during neutral statements. Methods The audio recordings of sixteen head and neck cancer survivors’ follow-up consultations were coded for patients’ emotional distress. Pitch (f0) of coded cues and concerns, including neutral statements was extracted. These were compared using a hierarchical linear model, nested for patient and pitch range, controlling for statement speech length. Utterance content was also explored. Results Clustering by patient explained 30% of the variance in utterances f0. Cues and concerns were on average 13.07 Hz higher than neutral statements (p = 0.02). Cues and concerns in these consultations contained content with a high proportion of recurrence fears. Conclusion The present study highlights the benefits and challenges of adding f0 and potential other prosodic features to the toolkit of coding emotional distress in the health communication setting. Practice implications The assessment of f0 during clinical conversations can provide additional information for research into emotional expression.PostprintPeer reviewe

Pemodelan CNN Untuk Deteksi Emosi Berbasis Speech Bahasa Indonesia

Perkembangan teknologi menunjukkan semakin banyak kebutuhan perangkat yang mampu menjalankan interaksi antara manusia dengan computer secara cerdas. Satu contohnya adalah sistem pengenalan emosi dengan computer. Di dalamnya diperlukan kemampuan untuk melakukan pengenalan, penafsiran, dan memberikan respons emosi yang diekspresikan dalam ucapan. Tetapi sampai saat ini penelilitan speech emotion recognition (SER) yang berbasis bahasa Indonesia masih sangat sedikit. Hal ini disebabkan keterbatasan korpus data berbahasa Indonesia untuk SER. Pada penelitian ini dibuat sistem SER dengan mengambil dataset dari TV series berbahasa Indonesia. Sistem dirancang dengan kemampuan untuk  melakukan proses klasifikasi emosi, yaitu empat kelas label emosi  marah, senang, netral dan sedih. Untuk implementasinya digunakan metode deep learning, yang dalam hal ini dipilih metode CNN. Pada sistem ini input berupa kombinasi dari tiga fitur, yaitu MFCC, frekuensi fundamental, dan RMSE. Dari eksperimen yang telah dijalankan telah diperoleh hasil terbaik untuk sistem SER berbahasa Indonesia dengan menggunakan input MFCC + frekuensi fundamental, yang menunjukkan tingkat akurasi sebesar 85%. Sedangkan ketika hanya menggunakan input MFCC memiliki tingkat akurasi sampai 83%. Sementara itu ketika dipaksakan dengan kombinasi ketiga input MFCC+ F0+ RMSE mengalami penurunan kinerja dan hanya mencapai akurasi 78% ,dan akurasi terendah menggunakan fitur MFCC + RMSE yaitu 72%. Dari study awal ini diharapkan mampu memberikan gambaran bagi para peneliti di bidang SER, tentang  bagaimana memilih fitur sinyal wicara sebagai input di dalam pengujian dan mempermudah untuk langkah pengembangan penelitiannya

I feel you: the design and evaluation of a domotic affect-sensitive spoken conversational agent

We describe the work on infusion of emotion into a limited-task autonomous spoken conversational agent situated in the domestic environment, using a need-inspired task-independent emotion model (NEMO). In order to demonstrate the generation of affect through the use of the model, we describe the work of integrating it with a natural-language mixed-initiative HiFi-control spoken conversational agent (SCA). NEMO and the host system communicate externally, removing the need for the Dialog Manager to be modified, as is done in most existing dialog systems, in order to be adaptive. The first part of the paper concerns the integration between NEMO and the host agent. The second part summarizes the work on automatic affect prediction, namely, frustration and contentment, from dialog features, a non-conventional source, in the attempt of moving towards a more user-centric approach. The final part reports the evaluation results obtained from a user study, in which both versions of the agent (non-adaptive and emotionally-adaptive) were compared. The results provide substantial evidences with respect to the benefits of adding emotion in a spoken conversational agent, especially in mitigating users' frustrations and, ultimately, improving their satisfaction

Psychophysiology in games

Psychophysiology is the study of the relationship between psychology and its physiological manifestations. That relationship is of particular importance for both game design and ultimately gameplaying. Players’ psychophysiology offers a gateway towards a better understanding of playing behavior and experience. That knowledge can, in turn, be beneficial for the player as it allows designers to make better games for them; either explicitly by altering the game during play or implicitly during the game design process. This chapter argues for the importance of physiology for the investigation of player affect in games, reviews the current state of the art in sensor technology and outlines the key phases for the application of psychophysiology in games.The work is supported, in part, by the EU-funded FP7 ICT iLearnRWproject (project no: 318803).peer-reviewe

Gaming variables in linguistic research. Italian scale validation and a Minecraft pilot study

This paper deals with the concept of gamified science and its recent applications to the linguistic field. We argue that, albeit promising, this paradigm still lacks analytical tools to model the effects of the peculiar experimental setting on the results obtained. After a theoretical introduction to the User Engagement and Gaming Literacy constructs, we present two validated Italian translations of scales representing them. Lastly, we test these two gaming variables in a pilot study on the postvocalic realizations of /k t/ in the Florentine variety. Results show that both variables positively condition the production of non-continuants (i.e., emphasized words) but through different underlying mechanisms

On automatic emotion classification using acoustic features

In this thesis, we describe extensive experiments on the classification of emotions from speech using acoustic features. This area of research has important applications in human computer interaction. We have thoroughly reviewed the current literature and present our results on some of the contemporary emotional speech databases. The principal focus is on creating a large set of acoustic features, descriptive of different emotional states and finding methods for selecting a subset of best performing features by using feature selection methods. In this thesis we have looked at several traditional feature selection methods and propose a novel scheme which employs a preferential Borda voting strategy for ranking features. The comparative results show that our proposed scheme can strike a balance between accurate but computationally intensive wrapper methods and less accurate but computationally less intensive filter methods for feature selection. By using the selected features, several schemes for extending the binary classifiers to multiclass classification are tested. Some of these classifiers form serial combinations of binary classifiers while others use a hierarchical structure to perform this task. We describe a new hierarchical classification scheme, which we call Data-Driven Dimensional Emotion Classification (3DEC), whose decision hierarchy is based on non-metric multidimensional scaling (NMDS) of the data. This method of creating a hierarchical structure for the classification of emotion classes gives significant improvements over other methods tested. The NMDS representation of emotional speech data can be interpreted in terms of the well-known valence-arousal model of emotion. We find that this model does not givea particularly good fit to the data: although the arousal dimension can be identified easily, valence is not well represented in the transformed data. From the recognitionresults on these two dimensions, we conclude that valence and arousal dimensions are not orthogonal to each other. In the last part of this thesis, we deal with the very difficult but important topic of improving the generalisation capabilities of speech emotion recognition (SER) systems over different speakers and recording environments. This topic has been generally overlooked in the current research in this area. First we try the traditional methods used in automatic speech recognition (ASR) systems for improving the generalisation of SER in intra– and inter–database emotion classification. These traditional methods do improve the average accuracy of the emotion classifier. In this thesis, we identify these differences in the training and test data, due to speakers and acoustic environments, as a covariate shift. This shift is minimised by using importance weighting algorithms from the emerging field of transfer learning to guide the learning algorithm towards that training data which gives better representation of testing data. Our results show that importance weighting algorithms can be used to minimise the differences between the training and testing data. We also test the effectiveness of importance weighting algorithms on inter–database and cross-lingual emotion recognition. From these results, we draw conclusions about the universal nature of emotions across different languages

Development of the multimodal system of educational game for partially sighted and blind children

У дисертацији су приказани развој мултимодалног система за образовну игру Луграм и истраживања везана за његову примену у области редовног и специјализованог основношколског образовања. Извршено је испитивање утицаја Луграма на повишење ефеката учења геометрије у редовној разредној настави, испитивање да ли његову прилагођену мултимодалну верзију могу да користе слабовида и слепа деца и да ли се она може користити за њихову обуку за самосталну употребу рачунарске аудио верзије Луграма. Утврђено је да Луграм има утицаја на пораст успеха у учењу геометрије и да се прилагођена аудио- тактилна верзија може користити у сврху обуке слабовиде и слепе деце за самостално коришћење аудио верзије игре. Резултати истраживања су показали оправданост приступа развоју Луграма као мултимодалног система за игру и усмерили његов даљи развој ка асистивном мултимодалном систему који поседује способност говорне интеракције са корисником и могућност прилагођења различитим категоријама корисника.U disertaciji su prikazani razvoj multimodalnog sistema za obrazovnu igru Lugram i istraživanja vezana za njegovu primenu u oblasti redovnog i specijalizovanog osnovnoškolskog obrazovanja. Izvršeno je ispitivanje uticaja Lugrama na povišenje efekata učenja geometrije u redovnoj razrednoj nastavi, ispitivanje da li njegovu prilagođenu multimodalnu verziju mogu da koriste slabovida i slepa deca i da li se ona može koristiti za njihovu obuku za samostalnu upotrebu računarske audio verzije Lugrama. Utvrđeno je da Lugram ima uticaja na porast uspeha u učenju geometrije i da se prilagođena audio- taktilna verzija može koristiti u svrhu obuke slabovide i slepe dece za samostalno korišćenje audio verzije igre. Rezultati istraživanja su pokazali opravdanost pristupa razvoju Lugrama kao multimodalnog sistema za igru i usmerili njegov dalji razvoj ka asistivnom multimodalnom sistemu koji poseduje sposobnost govorne interakcije sa korisnikom i mogućnost prilagođenja različitim kategorijama korisnika.The dissertation presents the development of the multimodal system for the educational game Lugram and the research related to its application in the regular and specialized primary education. The examination of the effect which Lugram has on the improvement in learning geometry in the primary education is done, as well as the testing weather its adapted multimodal version can be used for the training of partially sighted and blind children for the independent use of the computer audio version of Lugram. It is established that Lugram has an influence on the improvement in learning geometry and that adapted audio-tactile version can be used for the training purposes for partially sighted and blind children. Results of the research have justified the approach in the development of Lugram as a multimodal system for the game. Moreover, they focus further development toward assistive multimodal system which has ability of the voice interaction with the user and also ability of adaptation to the different kind of users