van den Broek, Egon

English

This commentary unveils over a century of literature that touches the core of affective computing but appears to be unknown to its community. To enable affective computing to flourish, knowledge on human’s physiology, on concepts such as emotions and moods, and on methods and techniques from signal processing and machine learning need to blend together. Two books that appeared approximately 50 years ago were selected, which originate from completely distinct branches of science, but both mark branches of research crucial to affective computing: i) Flanders Dunbar’s “Emotions and bodily changes: A survey of literature on psychosomatic interrelationships 1910-1953‿ (1954) and ii) Satosi Watanabe’s “Methodologies of Pattern Recognition (1969). When the vast and rich body of available relevant scientific literature will be cherished and the lessons learned so long ago will be embraced, affective computing can (finally) make its leap forward and probably will have a bright future

University of Twente Research Information

On the bodily expressions of emotion, be aware:More than a century of research!A commentary on Affective Computing byKristina HöökEgon L. van den BroekMedia and Network Services,TNO Technical SciencesP.O. Box 5050, 2600 GB Delft, The NetherlandsHuman Media Interaction,University of TwenteP.O. Box 217, 7500 AE Enschede, The NetherlandsKarakter University Center,Radboud University Medical Center NijmegenP.O. Box 9101, 6500 HB Nijmegen, The Netherlandsvandenbroek@acm.orgAbstract. This commentary unveils over a century of literature thattouches the core of affective computing but appears to be unknown toits community. To enable affective computing to flourish, knowledge onhuman’s physiology, on concepts such as emotions and moods, and onmethods and techniques from signal processing and machine learningneed to blend together. Two books that appeared approximately 50 yearsago were selected, which originate from completely distinct branches ofscience, but both mark branches of research crucial to affective com-puting: i) Flanders Dunbar’s “Emotions and bodily changes: A surveyof literature on psychosomatic interrelationships 1910-1953” (1954) andii) Satosi Watanabe’s “Methodologies of Pattern Recognition” (1969).When the vast and rich body of available relevant scientific literaturewill be cherished and the lessons learned so long ago will be embraced,affective computing can (finally) make its leap forward and probably willhave a bright future!Please refer to this article as follows:Broek, E.L. van den (2012). On the bodily expressions of emotion, be aware:More than a century of research! A commentary on Affective Computing byKristina Höök. In M. Soegaard (Ed.), The Interaction-Design.org Encyclope-dia of Human-Computer Interaction, Chapter 12. Aarhus C., Denmark: TheInteraction-Design.org foundation. URL: http://www.interaction-design.org/encyclopedia/affective computing.html#egon+l.+van+den+broek2 Egon L. van den Broek1 IntroductionWhen thinking about this commentary, ideas popped up and emotions emerged.What to comment on? Kia Höök delivered an excellent chapter (Höök, 2012).She mentions three angles to approach emotion in technology from (cf. van denBroek, 2011), namely: affective computing, affective interaction, and technologyas experience. In this commentary, I will narrow the focus to affective computingsolely. Furthermore, I have also chosen to take a step back and be so bold as totake a methodological perspective with a historical flavor. Why? Well, through-out the years I have discovered more and more literature that touches the coreof affective computing but appears to be unknown (e.g., Arnold, 1968; Cand-land, 1962; Dunbar, 1954). This commentary is founded on two books from atime long before the term affective computing was coined, the 50s and 60s ofthe previous century. Both books are taken from completely distinct branches ofscience. Knowledge on science’s history can prevent us, both practitioners andscientists, from repeating mistakes. As such, this commentary touches upon theessence of science itself.Kia Höök provides a concise overview of emotion in technology (Höök, 2012).She embraced affective interaction instead of affective computing. In contrast, inthis commentary, I have taken the affective computing standpoint. Moreover, KiaHöök has taken a design perspective, where this commentary touches upon andquestions the fundaments of emotions in technology. Lessons had been learnedbut have already been forgotten (Arnold, 1968; Candland, 1962; Dunbar, 1954).Consequently, affective computing tends to reinvent the wheel, at least to someextent. Yes, this is a bold claim, a very bold claim but I hope that after readingthis commentary, you as a reader may share my concerns.2 Emotions and bodily changesIn 1954, 5 years before her death, Flanders Dunbar delivered the fourth editionof “Emotions and bodily changes: A survey of literature on psychosomatic in-terrelationships 1910-1953” (Dunbar, 1954). With this impressive volume, sheprovides an exhaustive and structured review of scientific literature of (roughly)the first half of the previous century on emotions and bodily changes. The vol-ume’s title is well chosen and reflects its content nicely. This makes this bookundoubtedly valuable for the community of affective computing. However, as faras I know, outside my own work (e.g., Van den Broek, 2011), not a single refer-ence is made to this book in any affective computing article, report, or book. Ican only hope that I have missed quite a few ...Flanders Dunbar starts her book (1954) with: Nearly half a millennium B.C.,Socrates came back from army service to report to his Greek countrymen that inone respect the barbarian Thracians were in advance of Greek civilization: Theyknew that the body could not be cured without the mind. “This,” he continued,“is the reason why the cure of many diseases is unknown to the physicians ofHellas, because they are ignorant of the whole.” It was Hippocrates, the FatherOn the bodily expressions of emotion 3of Medicine, who said: “In order to cure the human body it is necessary to haveknowledge of the whole of things.” And Paracelsus wrote: “True medicine onlyarises from the creative knowledge of the last and deepest powers of the wholeuniverse; only he who grasps the innermost nature of man, can cure him inearnest.” To us today this seems rather an impossible demand. (p. 3). Wherethe work of Dunbar illustrates that the origins of affective computing can betraced back to more than a century ago, this quote illustrates that knowledgeon the interaction between body and mind was already known more than 25centuries ago! Let us now identify some core concepts as mentioned in the quotefrom Dunbar (1954), which are crucial for affective computing.The old Greek already noted that “the body could not be cured without themind” (cf. Höök, 2012). So, both are indisputably related and, hence, in prin-ciple, the measurement of emotions should be feasible. This is well illustratedby the remark that “the cure of many diseases is unknown to the physicians ofHellas”, as the Greek culture was devoted to the body and not to the mind.Recent work confirmed this relation. For example, when chronic stress is experi-enced, similar physiological responses emerge as were present during the stressfulevents from which the stress originates. If such physiological responses persist,they can cause pervasive and structural chemical imbalances in people’s phys-iological systems, including their autonomic and central nervous system, theirneuroendocrine system, their immune system, and even in their brain (Brosschot,2010). This brings us to the need for the “knowledge of the whole of things”, aholistic view, perhaps closely related to what Kia Höök denotes as Technologyas Experience (Höök, 2012).Although the previous enumeration of people’s physiological systems can givethe impression that we are close to a holistic model, it should be noted that thisis in sharp contrast with the current level of science. For example, with (chronic)stress, a thorough understanding is still missing. This can be explained by thecomplexity of human’s physiological systems, the continuous interaction of allsystems, and their integral dynamic nature. However, Brosschot (2010) considersemotions as if these can be isolated and attributed to bodily processes only. Ifirmly agree with Höök (2012) that dynamics beyond the body should also betaken into account. Moreover, as Höök (2012) also notes, in relation to computingentities, the interaction consists of much more than emotions; however, the sameis true when no computing is involved at all.3 Methodologies of Statistics and Pattern Recognition25 centuries ago scientists did not apply modern statistics; however, 1 centuryago, scientists did already apply statistics; for example, Fisher invented theANOVA class of statistical models in 1918. This provided the means to testand generalize findings on emotions and bodily changes and boosted the devel-opment of behavioral sciences in general (Dunbar, 1954). Moreover, this workfits into Rosalind W. Picard’s definition of affective computing: “. . . a set of ideason what I call ‘affective computing,’ computing that relates to, arises from, or4 Egon L. van den Broekinfluences emotions.” (Picard, 1995, p. 1) At least it fits when taken as the tradi-tional interpretation of computing (i.e., to determine by mathematical means).However, the added value of affective computing would be its engineering com-ponent, in particular, signal processing and pattern recognition (van den Broek,2011). This would enable machines to sense emotions, reason about them, andperhaps develop them themselves. This would mark a new era of computing.With the invention of computing machinery, shortly after World War II, anew type of statistics was developed: pattern recognition. In his edited volume“Methodologies of Pattern Recognition” (1969), Satosi Watanabe collected a setof papers that were presented or meant to be presented at the InternationalConference on Methodologies of Pattern Recognition in 1968. Watanabe startedhis book with defining pattern recognition: ’‘To the layman’s ear, the term pat-tern recognition sounds like a very narrow esoteric field of electronic computerapplications. But, actually, it is a vast and explicit endeavor at mechanizationof the most fundamental human function of perception and concept formation.”(p. vii).Watanabe denotes pattern recognition by computers as the “mechanizationof the most fundamental human function of (i) perception and (ii) concept forma-tion.” Up to this date human pattern recognition in general is largely unsolved.We do not understand how we, as humans, process affective signals (van denBroek, 2011). Moreover, the perception of signals and, subsequently, patternsis one thing; their interpretation in terms of emotions is something completelydifferent. This issue refers to content validity; that is, (i) the agreement amongexperts on the domain of emotions; (ii) the degree to which a (low level) per-cept adequately represents an emotion; and (iii) the degree to which (a set of)percepts adequately represents all aspects of the emotions under investigation.4 Concept formation in contextThe issue of concept formation relates to the process of construct validation,which aims to develop a ground truth (or an ontology or semantic network),constructed around the emotions investigated. Such a framework requires theo-retically grounded, observable, operational definitions of all constructs and therelations between them. Such a network aims to provide a verifiable theoreticalframework. The lack of such a network is one of the most pregnant problemsaffective computing is coping with. Kia Höök describes emotions as if we canpinpoint them. Although intuitively this is indeed the case, in practice it provesto be very hard to define emotions (Duffy, 1941; Kleinginna & Kleinginna, 1981).Par excellence, humans can recognize patterns in noisy environments. More-over, the ease with which humans adapt to new situations, to new patternsremains striking. Moreover, this is in sharp contrast with the performance ofsignal processing and pattern recognition algorithms. Often, these perform wellin a controlled environment; however, in the “real world” their performance de-teriorates (Healey, 2008). This problem refers to the influence of the context onmeasurements, which is also denoted as ecological validity. Due to a lack of realOn the bodily expressions of emotion 5world research, in general, the ecological validity of research on affective com-puting is limited and its use often still has to be shown in “real world” practice.However, as Kia Höök illustrates, some nice exceptions to this statement havebeen presented throughout the last decade.In 1941, Elizabeth Duffy published her article “An explanation of ‘emotional’phenomena without the use of the concept ‘emotion”’ in which she starts bystating that she considers “. . . ‘emotion’, as a scientific concept, is worse thanuseless. . . . ‘Emotion’ apparently did not represent a separate and distinguishablecondition.” Although this statement is 60 years old it is still (or, again) up todate, perhaps even more than ever (cf. Kleinginna & Kleinginna, 1981). Almostfifty years later, in 1990, John T. Cacioppo and Louis G. Tassinary expressed asimilar concern; however, they more generally addressed the complexity of psy-chophysiological relations. These “are conceptualized in terms of their specificity(e.g., one-to-one versus many-to-one) and their generality (e.g., situation or per-son specific versus cross-situational and pancultural).” (Cacioppo & Tassinary,1990). They proposed a model, which “yields four classes of psychophysiologi-cal relations: (a) outcomes, (b) concomitants, (c) markers, and (d) invariants.”Although Cacioppo and Tassinary (1990) discuss the influence of context, theydo not operationalize it; hence, this discussion’s value for affective computingis limited. Nevertheless, articles such as this are food for thought. Regrettably,attempts such as this are rare in the community of affective computing; conse-quently, the field’s research methods are fragile and a solid theoretical frameworkis missing (Van den Broek, 2011).5 DiscussionTo ensure sufficient advancement, it has been proposed to develop comput-ing entities that respond on their user(s) physiological response(s), withoutthe use of any interpretation of them in terms of emotions or cognitive pro-cesses (Tractinsky, 2004). This approach has been shown to be feasible for sev-eral areas of application. However, this approach also undermines the position ofaffective computing itself as a field of research. It suggests that emotion researchhas to mature further before affective computing can be brought to practice.This would be an honest conclusion but a crude one for the field of affectivecomputing. It implies that affective computing should take a few steps back be-fore making its leap forward. A good starting point for this process would bethe hot topics on emotion research that Gross (2010, p. 215) summarized in hisarticle “The future’s so bright, I gotta wear shades” (see also van den Broek,2011).Taken together, Kia Höök should be acknowledged for her concise overviewof emotion in technology. In her chapter she takes the affective interaction stand-point (Höök, 2012). In contrast, with this commentary, I have taken an affec-tive computing standpoint. Moreover, Kia Höök has taken a design perspective,where this commentary touches upon the fundaments of emotions in technology.I pose that if anything, affective computing has to learn more about its roots6 Egon L. van den Broek(e.g., Arnold, 1968; Candland, 1962; Dunbar, 1954); then, affective computingcan and probably will have a bright future!AcknowledgmentsI would like to thank Mads Soegaards (the Interaction-Design.org foundation)for his invitation to write this commentary and for his support in and patienceduring the whole process. In addition, Frans van der Sluis (Human Media Inter-action, HMI, University of Twente) is gratefully acknowledged for his commentson an early draft of this commentary. Last, I would like to thank Lynn Pack-wood (Human Media Interaction, HMI, University of Twente) for her carefulproof reading.ReferencesArnold, M. B. (1968). The nature of emotion: Selected readings. Har-mondsworth, Middlesex, England: Penguin Books Ltd.Brosschot, J. F. (2010). Markers of chronic stress: Prolonged physiological acti-vation and (un)conscious perseverative cognition. Neuroscience & Biobe-havioral Reviews, 35(1), 46–50.Cacioppo, J. T., & Tassinary, L. G. (1990). Inferring psychological significancefrom physiological signals. American Psychologist, 45(1), 16–28.Candland, D. K. (1962). Emotion: Bodily change – An enduring problem inpsychology, selected readings. Princeton, NJ, USA: D. van Nostrand Com-pany, Inc.Duffy, E. (1941). An explanation of “emotional” phenomena without the use ofthe concept “emotion”. Journal of General Psychology , 25 , 283–293.Dunbar, H. F. (1954). Emotions and bodily changes: A survey of literature onpsychosomatic interrelationships 1910–1953 (Fourth ed.). New York, NY,USA: Columbia University Press.Fisher, R. A. (1918). The correlation between relatives on the supposition ofMendelian inheritance. Transactions of the Royal Society of Edinburgh,52(2), 399–433.Gross, J. J. (2010). The future’s so bright, I gotta wear shades. Emotion Review ,2(3), 212–216.Healey, J. A. (2008). Sensing affective experience. In J. H. D. M. Westerink,M. Ouwerkerk, T. overbeek, W. F. Pasveer, & B. de Ruyter (Eds.), Prob-ing experience: From assessment of user emotions and behaviour to de-velopment of products (Vol. 8, pp. 91–100). Dordrecht, The Netherlands:Springer Science + Business Media B.V.Höök, K. (2012). Affective computing: Affective computing, affective interac-tion and technology as experience. In M. Soegaard (Ed.), The interaction-design.org encyclopedia of human-computer interaction. Aarhus C., Den-mark: The Interaction-Design.org Foundation.References 7Kleinginna, P. R., & Kleinginna, A. M. (1981). A categorized list of emotiondefinitions, with a suggestion for a consensual definition. Motivation andEmotion, 5(4), 345–379.Picard, R. W. (1995). Affective computing (Technical Report No. 321). Cam-bridge, MA, USA: M.I.T. Media Laboratory Perceptual Computing Sec-tion.Tractinsky, N. (2004). Tools over solutions? comments on interacting with com-puters special issue on affective computing. Interacting with Computers,16(4), 751–757.van den Broek, E. L. (2011). Affective Signal Processing (ASP): Unraveling themystery of emotions. Unpublished doctoral dissertation, Human MediaInteraction (HMI), Faculty of Electrical Engineering, Mathematics, andComputer Science, University of Twente, Enschede, The Netherlands.Watanabe, S. (1969). Methodologies of pattern recognition. New York, NY,USA: Academic Press, Inc.

On the bodily expressions of emotion, be aware: More than a century of research! : A commentary on Affective Computing by Kristina Höök

https://ris.utwente.nl/ws/files/5585513/VandenBroek12-On_the_bodily_expressions_of_emotion.pdf

On the bodily expressions of emotion, be aware: More than a century of research! : A commentary on Affective Computing by Kristina Höök

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information