This is the third part in a series on prerequisites for affective signal processing (ASP). So far, six prerequisites were identified: validation (e.g., mapping of constructs on signals), triangulation, a physiology-driven approach, and contributions of the signal processing community (van den Broek et al., 2009) and identification of users and theoretical specification (van den Broek et al., 2010). Here, two additional prerequisites are identified: integration of biosignals, and physical characteristics

Healey, Jennifer A.

Janssen, Joris H.

van den Broek, Egon

van der Zwaag, Marjolein D.

Westerink, Joyce H.D.M.

English

University of Twente Research Information

PREREQUISITES FOR AFFECTIVE SIGNAL PROCESSING (ASP) –PART IIIEgon L. van den Broekhttp://www.human-centeredcomputing.com/vandenbroek@acm.orgJoris H. Janssen, Marjolein D. van der ZwaagUser Experience Group, Philips Research Europe, High Tech Campus 34, 5656 AE Eindhoven, The Netherlands{joris.h.janssen,marjolein.van.der.zwaag}@philips.comJennifer A. HealeyFuture Technology Research, Intel Labs Santa Clara, Juliette Lane SC12-319 Santa Clara CA 95054, USAjennifer.healey@intel.comKeywords: affective signal processing, emotion, integration of biosignals, physical characteristicsAbstract: This is the third part in a series on prerequisites for affective signal processing (ASP). So far, six prerequi-sites were identified: validation (e.g., mapping of constructs on signals), triangulation, a physiology-drivenapproach, and contributions of the signal processing community (van den Broek et al., 2009) and identifica-tion of users and theoretical specification (van den Broek et al., 2010). Here, two additional prerequisites areidentified: integration of biosignals, and physical characteristics.1 INTRODUCTIONThis paper is the third paper in a series that addressesprerequisites for Affective Signal Processing (ASP).Let us start with what Rosalind W. Picard statesin the preface of her book Affective Computing(AC) (Picard, 1997, p. x):But what I ran into, in trying to understandhow our brains accomplish vision, was emotion. Notas a corollary, tacked on to how humans see, but asa direct component, an integral part of perception.. . . The role of emotions in “being emotional” isa small part of their story. The rest is largelyuntold, and has profound consequences – not just forunderstanding human thinking, but specifically forcomputing.Subsequently, she continues denoting the impor-tance of emotions through stating that . . . emotionsplay an essential role in rational decision making,perception, learning, and a variety of other cog-nitive functions (Picard, 1997, p. x). She evenposes that . . . too little emotion can impair decisionmaking (Picard, 1997, p. x). The former notionwas already embraced by cognitive sciences andis now generally accepted. The latter notion wasa shocking conclusion for the artificial intelligence(AI) community since their traditional foundation isone of reasoning and logic.In a nutshell, AC aims to model or classify humanemotions, using the ‘affective signals’ they transmit.These can be either facial characteristics (e.g., ob-tained through computer vision), movements, speechprocessing, biosignals, or a combination of these sig-nals (van den Broek et al., 2009). This paper ad-dresses the use of biosignals for affective signal pro-cessing (ASP), a notion which originates from the late19th century (James, 1894). Biosignals are especiallypromising as they can now be measured unobtrusivelyand in real-time through wearable devices.More than a decade after Picard’s seminal book,more than anything else, it has become apparent thatAC is incredibly hard and complex (Boehner et al.,2007; Chanel et al., 2009). Despite the vast effortstoward AC, results are still disappointing. Although itshould be noted that some results, should be markedas good; e.g., Healey and Picard (1998); Rani et al.(2003); Hönig et al. (2007); Benovoy et al. (2008).Although sometimes promising results are re-ported in literature, often follow-up research failed toreplicate those results; cf. Chanel et al. (2009). More-over, in the cases recognition rates of 90% or higherare achieved, three out of the four best results arebased one the results on one participant. Only in theresearch of Hönig et al. (2007) a group of (24) partici-pants participated; however, this study only differenti-ated between two levels of stress. Taken together, AChas a long way to go, before large-scale real-worldapplications using AC are feasible.Taken together, we pose that AC is a bridge toofar and ASP is what should have our attention first.To enable a breakthrough in results on ASP, we pro-pose to adopt a set of prerequisites, with which westarted in van den Broek et al. (2009) and continuedin van den Broek et al. (2010). After this, AC can bebrought to practice.This paper complements the two other pa-pers van den Broek et al. (2009) and van den Broeket al. (2010), which already introduced 6 prerequi-sites. Together, the three papers should form the foun-dation for more successful ASP and, in the future,successful AC.In the next section, the two new prerequisites willbe introduced, namely: integration of biosignals andphysiological characteristics. We end this paper inSection 3, with a brief conclusion.2 PREREQUISITES – PART IIIIn van den Broek et al. (2009) and van den Broeket al. (2010), the following prerequisites for ASP wereintroduced: validity, triangulation, a physiology-driven approach, contributions from signal process-ing, user identification, and theoretical specification.While each of these is still of the utmost importancefor ASP, we will now denote two additional pre-requisites: physical characteristics and integration ofbiosignals.2.1 Integration of biosignalsAlthough not frequently discussed, biosignals are in-fluenced by other factors besides affect (Cacioppo andTassinary, 1990). For instance, Figure 1 illustrateshow pervasive motion artifacts can be for ASP in realworld settings. Both HR and EDA are elevated dur-ing the period of high activity from 27 to 30 minutes.Moreover, the signal graphs also show that changes inHR follow changes in activity much more rapidly thanEDA, in onset and especially in terms of recovery. Forlevel 4 (walking) in this graph, it even seems that thephysical effects are so dominant that ASP should notbe attempted. In contrast, with level 1 (lying down),2 (sitting), and 3 (standing/strolling) this is possible.In van den Broek et al. (2009), we discussed deal-ing with these issues through triangulation; i.e., usingmultiple signals to describe or measure one construct.As a particular and highly effective instance of tri-angulation, we now discuss the integration of two ormore biosignals into one feature that can be used asinput to a classifier. This idea stems from the fact thatadditional biosignals can often explain noise that ispresent through other influences.The integration of biosignals takes three steps: (1)identifying a theoretical relationship between mul-tiple biosignals, (2) selecting an appropriate modelthat integrates both, and (3) data gathering and modeltraining.In the first step, a noisy biosignal is selected andtheoretical relationships with other variables are iden-tified. As we saw, HR is also influenced by physicalactivity and respiration. Hence, we should gather res-piration data and accelerometer data to correct the HRsignal for this noise. Other such relationships exist,for instance, between skin conductance and skin tem-perature, skin conductance and physical activity, orHR and skin temperature. Often this correction re-lates to a theoretical concept as well; e.g., correctingthe high frequency (HF) power obtained from the in-ter beat intervals (IBI) for respiration gives a reliablemeasure for activity of the parasympathetic nervoussystem, which is involved in relaxation and recoveryprocesses (Grossman and Taylor, 2007).The second step consists of selecting a procedureto integrate multiple signals. A popular approach isusing regression to describe the relationship betweenthe variables. Correcting observed values then con-sists of computing its residualized value (i.e., the dis-tance to the regression line) on the dimension of inter-est. In most cases, this is done using a linear regres-sion line, but it can just as well be done with morecomplex non-linear relationships. A second popularmethod for the integration of such signals is throughconditional probabilities. In that case, a Bayesianmodel is built in which the corrected value is con-ditional on the observed value and the values of theother influencing variables. When a new value is ob-served, the corrected value with the maximum a pos-teriori probability can be selected through this model.In the third step, when the theoretical relationshipsand integration procedures are established, data hasto be gathered that can be used to train the selectedmodel. Each instance of the data should contain avalue for of the features of the model. In the caseof regression, there are many algorithms that deter-mine a line or plane of best fit through the data. Aparticularly popular approach is using maximum like-lihood approach by minimizing the least squares er-ror. For Bayesian models, the data is modeled through(mostly continuous) probability distributions. The nu-merous methods for parameterizing a probability dis-tribution from data are beyond the scope of this paper;see Bishop (2006) and Korb and Nicholson (2004) formore info.0 5 10 15 20 25 301000200030004000GSRGSR and HR with Activity0 5 10 15 20 25 306080100120HR0 5 10 15 20 25 30234MinutesActivityFigure 1: Recordings of Heart rate (HR), electrodermal activity (EDA) (or galvanic skin response, GSR), and a person’sactivity for a period of 30 minutes, in a real world setting. See Section 2.1 for the legend of the activities.2.2 Physical characteristicsIn this section, we discuss the implications of physi-cal characteristics of sensors and the environment foraffective signal processing. There are a number of dif-ferent sensors. For respiration measurements, a gaugeband can be placed around the chest. Thermistor sen-sors placed on the surface of the skin can be used tomeasure skin temperature (Kistler et al., 1998). HRcan be measured through surface electrodes (ECG) orthrough a photoplethysmograph (BVP). Skin conduc-tance and muscle tension (EMG) are also measuredthrough surface electrodes.The choice of surface electrodes depends on thekind of measurement, the aim of the measurement,and the application in which it is used. On the onehand, in the lab one opts for the most sensitive andreliable electrodes, which are wet electrodes that usea gel for better conductivity. On the other hand, forwearable affective measurements a better option isdry electrodes, as these are more practical and easierto attach and incorporate in devices.The kind of gel used in wet electrodes dependson the measurement type. For skin conductancemeasurements, a salt less gel should be used as saltchanges the composition of the skin which influencesthe measurement (Boucsein, 1992). For EMG andECG, gels with high electric conductance are better.The location of the surface electrodes is importantas improperly placed can cause noise in the signal.However, in the case of ASP, the wearable devicesand setting will put constraints on the location of thesensors. For example, the upper phalanx of the fingertips conventionally used for skin conductance mea-surements cannot be used while driving a car. Otherparts of the hands or the sole of the foot should beused instead. For HR, instead of using electrodes onthe chest (ECG) one can use a BVP sensor on the ear,hand, or foot. Skin temperature can also be measuredon the foot instead of the hand.For ASP, the physical characteristics of the en-vironment like humidity and temperature also playan important role. This predominantly influencesthe skin conductance and temperature measurements.This point is of special interest for longer periodsof continuous measurements and is also different inmedical experiments which require a controlled labsituation in where humidity and temperature of theroom can be kept constant.To deal with the issues of different sensor posi-tions and changes in environmental temperature andhumidity one should standardize the measurementsusing z-scores for each session. During continuouslonger term measurements one can use a sliding timewindow for a set period, e.g. one or two hours, whichis used for standardization.To conclude, due to the large amount of differ-ences in the aim of physiological measurements, dif-ferent sensor positions, and different or even changingenvironmental conditions, one should always care-fully puzzle to find the best combination of electrodetypes and locations. Furthermore, standardizing thesignals will also reduce a lot of the otherwise unex-plained variance in the signal. In the end, this willprovide cleaner signals to the machine learning algo-rithms and will lead to a much more successful ASP.3 CONCLUSIONThis paper provided the third set of prerequisitesfor ASP. It comprises the prerequisites integrationof biosignals and physical characteristics, which arecomplementary to the six previously introduced pre-requisites: identification of users and theoretical spec-ification (van den Broek et al., 2010) and validity, tri-angulation, the physiology-driven approach, and con-tributions of signal processing (van den Broek et al.,2009).Perhaps the conclusion should be that, for now,AC is too complex (Boehner et al., 2007); cf. Chanelet al. (2009). We pose that it would be wise to takea step back, and study ASP, using the prerequisitesprovided. Then, time will learn whether AC will befuture or remain fiction.ACKNOWLEDGMENTSThe authors would like to thank both Joyce H.D.M.Westerink (Philips Research, Eindhoven, The Nether-lands) and the anonymous reviewers for their com-ments.REFERENCESBenovoy, M., Cooperstock, J. R., and Deitcher, J. (2008).Biosignals analysis and its application in a performancesetting: Towards the development of an emotional-imaging generator. In Biosignals 2008: Proceedings ofthe first International Conference on Biomedical Elec-tronics and Devices, volume 1, pages 253–258, Funchal,Madeira, Portugal. INSTICC.Bishop, C. M. (2006). Pattern Recognition and MachineLearning. Information Science and Statistics. New York,NY, USA: Springer Science+Business Media, LLC.Boehner, K., DePaula, R., Dourish, P., and Sengers, P.(2007). How emotion is made and measured. Interna-tional Journal of Human-Computer Studies, 65(4):275–291.Boucsein, W. (1992). Electrodermal activity. New York,NY, USA: Plenum Press.Cacioppo, J. and Tassinary, L. (1990). Inferring psycholog-ical significance from physiological signals. AmericanPsychologist, 45(1):16–28.Chanel, G., Kierkels, J. J. M., Soleymani, M., and Pun,T. (2009). Short-term emotion assessment in a recallparadigm. International Journal of Human-ComputerStudies, 67(8):607–627.Grossman, P. and Taylor, E. W. (2007). Toward understand-ing respiratory sinus arrhythmia: Relations to cardiac va-gal tone, evolution and biobehavioral functions. Biolog-ical Psychology, 74(2):263–285.Healey, J. A. and Picard, R. W. (1998). Digital processingof affective signals. In Proceedings of the IEEE Interna-tional Conference on Acoustics, Speech, and Signal Pro-cessing (ICASSP98), volume 6, pages 3749–3752, Seat-tle, WA, USA. IEEE.Hönig, F., Batliner, A., and Nöth, E. (2007). Real-timerecognition of the affective user state with physiologicalsignals. In Proceedings of the Doctoral Consortium ofAffective Computing and Intelligent Interaction (ACII),pages 1–8, Lisbon, Portugal.James, W. (1894). The physical basis of emotion. Psycho-logical Review, 1:526–529.Kistler, A., Mariauzouls, C., and von Berlepsch, K. (1998).Fingertip temperature as an indicator for sympatheticresponses. International Journal of Psychophysiology,29(1):35–41.Korb, K. B. and Nicholson, A. E. (2004). Bayesian ArtificialIntelligence. Boca Raton, FL, USA: Chapman & Hall /CRC Press.Picard, R. W. (1997). Affective Computing. Boston, MA,USA: MIT Press.Rani, P., Sarkar, N., Smith, C. A., and Adams, J. A. (2003).Affective communication for implicit human-machineinteraction. In IEEE International Conference on Sys-tems, Man, and Cybernetics, volume 5, pages 4896–4903.van den Broek, E. L., Janssen, J. H., van der Zwaag, M. D.,and Healey, J. A. (2010). Prerequisits for Affective Sig-nal Processing (ASP) – Part II. In Biosignals 2010: Pro-ceedings of the International Conference on Bio-InspiredSystems and Signal Processing, page [submitted], Valen-cia – Spain.van den Broek, E. L., Janssen, J. H., Westerink, J. H. D. M.,and Healey, J. A. (2009). Prerequisits for Affective Sig-nal Processing (ASP). In Encarnação, P. and Veloso,A., editors, Biosignals 2009: Proceedings of the Inter-national Conference on Bio-Inspired Systems and SignalProcessing, pages 426–433, Porto – Portugal.

Prerequisites for Affective Signal Processing (ASP) - Part III

In [1–3], a series of prerequisites for affective signal processing (ASP) was defined: validation (e.g., mapping of constructs on signals), triangulation, a physiology-driven approach, contributions of the signal processing community, identification of users, theoretical specification, integration of biosignals, and physical characteristics. This paper defines three additional prerequisites: historical perspective, temporal construction, and real-world baselines

Prerequisites for Affective Signal Processing (ASP) - Part IV

Last year, in van den Broek et al. (2009a), a start was made with defining prerequisites for affective signal processing (ASP). Four prerequisites were identified: validation (e.g., mapping of constructs on signals), triangulation, a physiology-driven approach, and contributions of the signal processing community. In parallel with this paper, in van den Broek et al. (2010) another set of two prerequisites is presented: integration of biosignals and physical characteristic. This paper continues this quest and defines two additional prerequisites: identification of users and theoretical specification. In addition, the second part of a review on the classification of emotions through ASP is presented; the first part can be found in van den Broek et al. (2009a)

van der Zwaag, Marjolein

Prerequisites for Affective Signal Processing (ASP) - Part II

A theoretical study on six classiﬁer fusion techniques.

Affect detection in the real world: Recording and processing physiological signals.

Affective Computing.

Affective intelligent car interfaces with emotion recognition.

Affective Man-Machine Interface: Unveiling human emotionsthroughbiosignals, page [inpress]. Biomedical Engineering Systems and Technologies: BIOSTEC2009 Selected Revised papers.

An empirical study of machine learning techniques for affect recognition in human-robot interaction.

Anxiety detecting robotic systems - towards implicit humanrobot collaboration.

ART: A hybrid classiﬁcation model.

Automatic temporal segment detection and affect recognition from face and body display.

Basic emotions are associated with distinct patterns of cardiorespiratory activity.

Bimodal emotion recognition using speech and physiological changes, chapter 15,

Biometric valence and arousal recognition.

Comparing two emotion models for deriving affective states from physiological data.

Considerations for emotion-aware consumer products.

Detecting stress during real-world driving tasks using physiological sensors.

Developing innovativetechnologyforfuturepersonalizedautismresearch and treatment.

Emotion recognition from surface EMG signal using wavelet transform and neural network.

Emotion recognition system using short-term monitoring of physiological signals.

Emotional speech recognition: Resources, features, and methods.

Empathy as an interpersonal phenomenon.

Ethnic differences among housewives in psychophysical and skin potential responsestoelectricshock.

Human-robot interaction using affective cues.

Inferring psychological signiﬁcance from physiological signals.

Integrating information from speech and physiological signals to achieve emotional sensitivity.

MAUI: a multimodal affective user interface.

Music emotion classiﬁcation: A regression approach.

Optimal combinations of pattern classiﬁers.

Personalized affective music player.

Physiological sensing and feature extraction for emotion recognition by exploiting acupuncture spots.

Prerequisits for Affective Signal Processing

Prerequisits for Affective Signal Processing (ASP) – Part III.

Short-term emotion assessment in a recall paradigm.

Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine – belief network architecture.

The cognitive regulation of emotions: The role of success versus failure experience and coping dispositions.

The imitation of man by machine – the view that machines will think as man does reveals misunderstanding of the nature of human thought.

The role of criminal proﬁling in the computer forensics process.

Toward machine emotional intelligence: Analysis of affective physiological state.

Unobtrusive Sensing of Emotions (USE).

https://ris.utwente.nl/ws/files/5370190/Prerequisites_IV_final2r.pdf

Prerequisites for Affective Signal Processing (ASP) - Part III

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information

University of Twente Research Information

University of Twente Research Information