de Vries, Jan

English

Emoties en ook stress maken een zeer belangrijk deel uit van ons dagelijks leven. Ons drukke bestaan kan ons echter emotioneel uit balans brengen. Slimme apparaten kunnen ons helpen in balans te blijven, bijvoorbeeld door op tijd te waarschuwen voor overmatige stress. Essentieel voor zulke apparaten is dat zij emoties en stress moeten kunnen meten. Dit proefschrift bestudeert hoe een bepaald type zelflerende computersystemen kan worden ingezet in dit zogeheten affectieve domein en verkent daarin herkenning van emoties op basis van drie verschillende meetmethoden, gemeten aan: het lijf (fysiologie), het gelaat (gezichtsuitdrukkingen) en het brein (cognitieve processen). Daarmee probeert dit proefschrift theorie en toepassing op het kruisvlak van informatica en psychologie, ook wel affectieve informatica genoemd, te verbinden.Het onderzoek toont aan dat computers, op basis van deze zelflerende systemen, goed in staat zijn om emoties te herkennen uit foto’s van gelaatsuitdrukkingen en ook goed stress kunnen herkennen uit hartslagsignalen, beide met grote precisie. De gebruikte methoden leveren bovendien modellen op waaruit nieuwe kennis geput kan worden. Zo blijkt dat de mond en de ogen erg belangrijk zijn bij het herkennen van de gezichtsexpressies en dat de emotieherkenning uit fysiologie kan worden verbeterd door toevoeging van metingen van bepaalde hartfrequenties.Emotions and stress are vital for our day to day functioning. The busy life we live, can however, cause emotional unbalance. Smart devices can help us remain balanced by providing early warnings, for example, in case of high levels of stress. A key enabler for such devices is to be able to measure emotions and stress. This thesis studies how a certain type of self-learning computer system can be used in this so-called affective domain and explores emotion recognition based upon three different methods of measuring: the body (physiology), the face (facial expressions) and the brain (cognitive processes). By doing so, this thesis combines theory and application at the boundary of computing science and psychology, also termed affective computing.The research performed shows that computers, based upon these self-learning systems, can detect emotions from photos of facial expressions and can detect stress from cardiac signals, both with high accuracy. The methods used, also provide models based upon which new knowledge can be gained. As an example, the mouth and eyes were found most vital for recognizing facial expressions and it was found that emotion recognition from physiology can be improved by adding measurements of specific heart frequencies

Dissertations of the University of Groningen

   University of GroningenModeling Affective State using Learning Vector Quantizationde Vries, JanIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2014Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):de Vries, J. (2014). Modeling Affective State using Learning Vector Quantization. [Thesis fully internal(DIV), University of Groningen]. [S.n.].CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 21-01-2023Hoofdstuk 8SUMMARYSummarizing, this thesis focusses on advancing the knowledge of LVQ methods,in particular in the application to domain of affective computing. In this the-sis various LVQ variants were studied in a controlled environment characterizedby high dimensional isotropic clusters. This data model yields a simplified repre-sentation of reality that allows studying learning dynamics in detail. The findingsdemonstrated that while LVQ 2.1 can obtain optimal asymptotic performance it suf-fers from stability issues due to divergent behavior of the prototypes. The introduc-tion of a window, in LFM-W, shows convergence for well chosen window sizes, butintroduced large dependency on parameter optimization, with respect to reachingoptimal performance. While GLVQ suffered from both divergence and parameterdependency, RSLVQ combined the strengths of both methods and, as the name sug-gests, is more robust to the parameter settings. It required only limited parameteroptimization to reach close to optimal asymptotic performance. Optimization of thewindow size and softness parameter of RSLVQ, however was critical for efficientlearning.Affective computing comprises the application of computers to the study ofemotions, moods, stress and other affective phenomena. Although affect has beenstudied for a long time in psychology, the research field of affective computing isstill in its relative infancy and poses interesting challenges to computer science andin particular machine learning. In order to research the performance of LVQ va-riants outside of the controlled simulation environment, we applied various LVQtechniques to various classification tasks in the affective domain. To the best of ourknowledge, this work comprises a first application of LVQ to the particular affectivechallenges at hand. Applied to these three real world scenarios, RSLVQ showed itsrobustness and very competitive predictive performance as affective classifier, com-peting well with GMLVQ and linear SVM. To that end, three datasets were collectedusing different input modalities, that approached affective computing from threeperspectives:146 8. SUMMARYBodily A large-scale study was performed in which skin conductance, respirationand electrocardiogram were measured in semi-controlled conditions. A large va-riety of features was derived from each of these modalities. Classifiers were builtusing uni- and multi-modal input. Using LVQ techniques, we obtained up to 86.7%accuracy and AUC of 0.95 in the two-class classification task to separate stress fromrelaxation. Relevance learning was used to identify the most informative features,indicating that most information was embedded in the cardiac signals primarily intime-domain HRV measures. In addition to commonly used features, we also ex-plored various novel features, of which the very-high frequency band of the powerspectrum was found to be a very relevant addition. Best performing algorithmswere GMLVQ, RSLVQ and linear SVM, of which the performance differences werewithin percentage points.Facial A benchmark dataset, the Cohn-Kanade database, was used to build clas-sifiers for facial expression recognition, based upon LBP features that reflect localtextures from still images. The feature space was characterized by its large dimen-sionality. Linear SVM performed best with accuracies up to 94.5% for 6-class and93.2% for 7-class classification, closely followed by RSLVQ. Implicit relevances ob-tained from RSLVQ provided insight into the most prominent features, which origi-nated, primarily, from the mouth region and eye regions. The specific LBP featuresthat were found most influential within these regions confirmed that mouth ope-ning/closing is a primary differentiator picked up by the classifier.Cognitive The cognitive processes involved in the emotional interpretation of sti-muli is termed appraisal. Several appraisal models have been coined that defineseveral dimensions of emotional appraisal. We set up a study in which emotionswere elicited and accompanying appraisal values determined. In a 5-class classifi-cation task, RSLVQ and SVM performed similarly well with accuracies up to 54.5%and Cohen’s Kappa of 0.42. Compared to the earlier applied technique of class con-ditional means, this showed a gain of 5 percentage point in accuracy and 0.05 inKappa. Applied to this noisy dataset GMLVQ performed relatively poorly at a levelsimilar to that of GLVQ.Observations made over the three experiments include that GLVQ and GRLVQsuffered from stability issues similar to the simulated environment. RSLVQ and li-near SVM performed similarly well at performance levels close to or beyond thestate of art. GMLVQ performed poorly in a very noisy environment, but very wellin the relatively easier classification tasks. The computational costs involved in re-levance learning, however did not allow for the application of GMLVQ to the facial8.1. Outlook 147emotion dataset.For each of the three affective domains we outlined potential applications anddemonstrators. These include the Vitality Bracelet, which monitors daily stress le-vels through skin conductance measurements, triggers in case of high stress andprovides a paced breathing exercise to let the user calm down; the facial expressionrecognition was applied to various iconic faces to reveal the emotions displayed inthe Mona Lisa, by Bill Clinton and Albert Einstein; and usage of the cognitive ap-praisal model was explored in the Empathic Photo Frame that displays pictures tomatch a desired emotion while using representations of both pictures and emotionsin the space spanned by the appraisal dimensions.8.1 OutlookThe following topics have been identified for future work:• Although in our experiments we made steps to move the affective researchfrom the lab to more daily life situations, the conditions in our experimentswere still semi-controlled. It would be interesting to see how the classifierstrained and the results found translate to even less controlled, daily life situa-tions.• Very high dimensional spaces such as the dataset for facial emotion recog-nition pose their challenges for the more computational expensive relevancelearning LVQ. Despite our efforts in optimization of the implementation ofGMLVQ and GRLVQ these techniques took significant time for such very highdimensional classification problems. Further code optimization or the use ofmore powerful computers would enable the application of relevance learningin LVQ on very high dimensional datasets.• We have outlined demonstrator systems for the three affective subdomainsresearched in this thesis. Further validation and optimization of these systemsin a daily life setting could bring these applications closer to productization.In sum this thesis provides useful results for further investigations of the appli-cation of LVQ methods to the affective domain and brings us one step closer to theapplication of affective computing in daily life.

Modeling Affective State using Learning Vector Quantization

https://pure.rug.nl/ws/files/14216026/Chapter_8.pdf

Modeling Affective State using Learning Vector Quantization

Abstract

Similar works

Full text

Available Versions

Dissertations of the University of Groningen