It is certainly not the case that wen we consider research on the role of human characteristics in the user interface of computers that no attention has been paid to the role of humor. However, when we compare efforts in this area with efforts and experiments that attempt to demonstrate the positive role of general emotion modelling in the user interface, then we must conclude that this attention is still low. As we all know, sometimes the computer is a source of frustration rather than a source of enjoyment. And indeed we see research projects that aim at recognizing a user’s frustration, rather than his enjoyment. However, rather than detecting frustration, and maybe reacting on it in a humorous way, we would like to prevent frustration by making interaction with a computer more natural and more enjoyable. For that reason we are working on multimodal interaction and embodied conversational agents. In the interaction with embodied conversational agents verbal and nonverbal communication are equally important. Multimodal emotion display and detection are among our advanced research issues, and investigations in the role of humor in human-computer interaction is one of them

Nijholt, A.

English

Nijholt, Anton

University of Twente Research Information

Computers that  Smile: Humor in the Interface  Anton Nijholt University of Twente Department of Computer Science PO Box 217, 7500 AE Enschede The Netherlands anijholt@cs.utwente.nl  Introduction It is certainly not the case that wen we consider research on the role of human characteristics in the user interface of computers that no attention has been paid to the role of humor. However, when we compare efforts in this area with efforts and experiments that attempt to demonstrate the positive role of general emotion modelling in the user interface, then we must conclude that this attention is still low. As we all know, sometimes the computer is a source of frustration rather than a source of enjoyment. And indeed we see research projects that aim at recognizing a user’s frustration, rather than his enjoyment. However, rather than detecting frustration, and maybe reacting on it in a humorous way, we would like to prevent frustration by making interaction with a computer more natural and more enjoyable. For that reason we are working on multimodal interaction and embodied conversational agents. In the interaction with embodied conversational agents verbal and nonverbal communication are equally important. Multimodal emotion display and detection are among our advanced research issues, and investigations in the role of humor in human-computer interaction is one of them.  The Role of Humor in Interpersonal Interaction In interpersonal interactions, either at work or at home, humans use humor, humans smile and humans laugh. Humor can be spontaneous, but it can also serve a social role and be used deliberately. A smile can be the effect of appreciating a humorous event, but it can also be used to regulate the conversation. A laugh can be spontaneous but can also mask disagreement or be cynical. In an educational situation humor can be used by the teacher to catch students' attention but also to foster critical thinking. Humor allows criticism to be smoothed, stress can be relieved and students can become more involved in joint class room activities by the use of humor. In an (E-)commerce situation we have negotiators that use humor to induce trust [1]. The relation between humor and trust has been studied in [11]. Hampes also studied the relationship between intimacy and humor. Humor can be the right answer to frustration. Clearly, and especially of interest when we design embodied conversational agents, humor plays an important role in interpersonal attraction [7]. One reason to look at this role, and in particular the role of similar humor appreciation, is that sense of humor is generally considered a highly valued characteristic of self and others. Nearly everybody claims to have average to above average senses of humor. Perceived similarity in humor appreciation can therefore be an important dimension when designing for interpersonal attraction. In the experiments reported by Cann et al. [7] participants had to interact with an unseen stranger. Before the interaction ratings were made of the attitudes of the participants and they were led to believe that the stranger had similar or dissimilar attitudes. The stranger responded either positively or neutrally to a participant’s attempt to humor. As a main result it was shown that similarity in humor appreciation was able to negate the negative effects of dissimilarity for other attitudes when looking at interpersonal attraction. In our group we have studied how similarity in attitudes is related to the development of a friendship relationship. We investigated how to integrate social psychology and other results of friendship research in the design and implementation of embodied conversational agents. The development of a friendship relationship requires time, but especially in the initiation phase the kinds of similarities mentioned above can be exploited.  Computers as Social Actors In the research on the ‘computers are social actors’ (CASA) paradigm (see e.g. Reeves & Nass [21]) it has been convincingly demonstrated that people interact with computers as if they were social actors. Due to the way we can let a computer interact, people may find the computer polite, dominant, extrovert, introvert, or whatever attitudes or personality (traits) we can display in a computer. Moreover, they react to these attitudes and traits as if they were displayed by a human being. As an example, consider the situation where a person interacts with the computer in order to perform a certain task. When, after completing the task, the person is asked by the same computer about its (i.e., the computer’s) behavior, the user is much more positive than when asked this question while sitting behind an other computer. From these CASA experiments we conclude that it is possible, at least in principle, to design systems that are perceived as social actors and that can display characteristics that elicit positive feelings about an interaction, even though the interaction is not considered as perfect from the user’s point of view. As mentioned above, humor plays an important role in interpersonal interactions. And so do smiles. We will return to the role of smiles later. Will humor in the interface have similar effects as in interpersonal interactions? In [17], experiments are reported that have been performed to examine the effects of humor in task-oriented computer-mediated communication and in human-computer interaction. It was shown that humor can have many positive effects. For example, participants who had received jokes during the interaction rated a system as more likable and competent They smiled and laughed more, they responded in a more sociable manner and reported greater cooperation. The study provides strong evidence that humor should be incorporated in computer mediated communication and human-computer interaction systems.  About Humor in the Interface In [5] Binsted discusses how humour can make user interfaces friendlier. Humans use humour to ease communication problems. In a same way humour can be used to solve communication problems that arise with human-computer interaction using natural language interfaces. Binsted explains that the kinds of humour to be used do not have to be very sophisticated. Suitable humour that can be used is self-deprecating humour. In some cases deprecating the user of a third party can be appropriate, but this type of humour is very risky. Humour can make a computer more human when it fails and can ease the interaction. Inappropriate humour, however, is irritating and humour should be tailored to the user. When a certain user regularly works with a system, the system can adapt the use of humor to the user’s taste. She concludes that humour that is sparingly and carefully used can make natural language interfaces much friendlier. There are other who have reported about the (expected) role of humour in the interface, e.g. Stock [22]. However, there are not that may applications, but see Loehr [15]. As observed above, rather than looking for ways to increase enjoyment, researchers prefer to look at handling frustration. For example, there have been experiments in (text-based) human-computer interaction where users’emotions were ignored and where users were allowed to vent their feelings and thoughts to the computer. In the latter case users continued to interact with the computer much longer than when did not have this opportunity. As an other example, and again rather in contrast to what we would like to research, in Marsala et al. [16], event appraisal is used to recognize and process feelings of guilt and anger in a setting where an embodied conversational agent talks with a mother of children with leukemia.  Embodied Conversational Agents In our research on natural interactivity between humans and computers we adhere to the CASA paradigm. Embodied conversational agents (ECA’s) have become a well-established research area. Embodied agents are agents that are visible in the interface as animated cartoon characters or resembling human beings. Sometimes they just consist of an animated talking face, displaying facial expressions and, when using speech synthesis, having lip synchronization. These agents are used to inform and explain or even to demonstrate products or sequences of activities in educational, e-commerce or entertainment settings. Experiments have shown that ECA’s can increase the motivation of a student or a user interacting with the system. Lester and others [14] showed that a display of involvement by an embodied conversational agent motivates a student in doing (and continuing) his or her learning task. Current research deals with improving intelligent behaviour of these ECA’s, but also with adding emotional behaviour and personality. Trust is another issue. Do we trust an ECA that plays the role of a doctor or a salesperson? The role of small talk for, among others, inducing trust in an embodied real estate agent is discussed in [3]. Having properties like these help to improve the 'believability' of an embodied agent [2]. Clearly, humor is very much related to many of the natural interaction issues mentioned above: emotions, personality, attraction, and trust. Until now we have not seen much research going on into embodied agents that interpret or generate humor in the interface. Neither have we seen much going on in the area of agents that smile at appropriate moments, agents that laugh because of a ‘decision’ made on an appraisal of events, etc. The face has been mentioned as a primary source for obtaining information of the affective state of an interactant. That makes it important to be able to display the right kinds of smiles at the right time on the face of an embodied conversational agent. Different kinds of smiles have been studied by Frank & Ekman [10]. In applications using embodied conversational agents we have to decide which smiles and laughs to use while interacting with a human converstional partner. Is the embodied agent really amused or does it only display a polite smile because it does not really like the joke made by its human conversational partner. Or shouldn’t it laugh or smile at all because of this politically incorrect joke?  Nonverbal and Affective Interaction In previous years we have seen the emergence of affective computing. Although many research results on modelling of affect are available, it is certainly not the case that a comprehensive theory of affect modelling is available. Reasons to include emotion modeling in intelligent systems are, among others, to enable decision-making in situations where it is difficult, if not impossible, to make rational decisions, to afford recognition of a user's emotions in order to give better and more natural feedback, and to provide display of emotions. Especially when the interface includes an embodied conversational agent, it seems rather obvious that the user expects a display of emotions and some recognition of emotions by the embodied agent. In some recent papers we investigated the possibilities to detect and generate multimodal emotion display. Emotions show in speech, gestures, posture and facial expressions. There are also possibilities to measure the physiological appearances of emotions. There are ways to measure enjoyment by looking at these modalities. However, rather than measuring we would like to understand situations from which enjoyment emerges.  Appraisal of Humorous Events? It is clear from the observations we had in the previous sections that there is a need for models that allow generation, prediction, detection and interpreting of humorous events. There is also a need to be able to generate, display and interpret smiles and laughs in a context that is not necessarily found humorous by an embodied conversational agent, but that neverheless may lead to smiles and laughs, for example to keep a conversation going. We think that it seems to be quite natural to make a step from event appraisal theories for emotion (Ortony et al. [20]) to appraisal theories for humourous events, in order to try to realize that ECA’s smile or even laugh at the right moment, making them more believable. This requires some measurement of humor appreciation. Several scales have been introduced to measure sense of humor and appreciation of humor. Especially important are scales that have been designed to measure appreciation of humorous events. Can we annotate sufficiently humorous events such that a machine learning program can develop its own snese of humor?  Conclusions This is a position paper. We are able to touch upon the state of the art of the above mentioned research areas (affective computing, facial expressions, embodied conversational agents, humorous interfaces), to touch upon the current state of the art of computational humour research, as presented in a recent deliverable we wrote for a European IST project on humour research [18], and to present some views on the role of humor in future interfaces, inhabited by embodied conversational agents. We think that from these observations it may become clear that current research on affective computing, research on generating and interpreting facial expressions and research on embodied (and intelligent) agents can and should converge and that modest attempts on humour research can be brought inside this framework in order to design new and interesting applications in human-computer interaction using embodied agents  References [1] V. Adelswärd & B.-M. Öberg. The function of laughter and joking in negotiation activities. Humor. Intern. Journal of Humor Research (11) 4, Mouton de Gruyter, Berlin, 1998, 411-429. [2] J. Bates. Virtual reality, art, and entertainment. Presence: Teleoperators and Virtual Environments 1 (1), 133-138. [3] T. Bickmore & J. Cassell. Relational agents: A model and implementation of building user trust. Proc. ACM CHI 2001, Seattle, Washington, 2001. [4] N. Bischof. The varieties of smiling and the problem of motivational acclimatization. Abstract. In: Proceedings 8th European Conference on Facial Expression – Measurement and Meaning, University of the Saarland, Saarbrücken, Germany, 1999. [5] K. Binsted. Using humour to make natural language interfaces more friendly. In: Proceedings of the AI, ALife and Entertainment Workshop, Intern. Joint Conf. on Artificial Intelligence, 1995. [6] Bui The Duy, D. Heylen, M. Poel & A. Nijholt. Generation of facial expression from emotion using a fuzzy rule based system. In: Proceedings 14th Australian Joint Conf. on Artificial Intelligence (AI'01), Springer LNCS, December 2001, Adelaide, Australia, to appear. [7] A. Cann, L.G. Calhoun & J.S. Banks. On the role of humor appreciation in interpersonal attraction: It’s no joking matter. Humor. Intern. J. of Humor Research (10) 1, 1997, 77-89. [8] P. Ekman. Facial expressions and emotion. American Psychologist, Vol. 48, No. 4, 384-392. [9] C. Elliott. Why boys like motorcycles: using emotion theory to find structure in humorous stories. Unpublished paper, School of Computer Science, DePaul University, Chicago, 1999.  [10] M.G. Frank & P. Ekman. Not all smiles are created equal: the differences between enjoyment and nonenjoyment smiles. Humor. Intern. J. of Humor Research (6) 1, 1993, 9-26. [11] W.P. Hampes. The relationship between humor and trust. Humor. Intern. J. of Humor Research (12) 3, 1999, 253-259. [12] J. Hulstijn & A. Nijholt (eds.). Computational Humor: Interpretation and generation of verbal humor. Proc. Twente Workshop on Language Technology 12, Un. of Twente, Nethelands, 1996. [13] J. Klein, Y. Moon & R.W. Picard. This computer responds to user frustration. Theory, design, results, and implications. MIT Media Lab, 1999. [14] J.C. Lester, S.A. Converse, S.E. Kahler, S.T. Barlowe, B.A. Stone & R. Bhogal. The persona effect: Affective impact of animated pedagogical agents. In: Proc. of CHI '97 Human Factors in Computing Systems, ACM, 359-356. [15] D. Loehr. An integration of a pun generator with a natural language robot. In: Proc. Intern. Workshop on Computational Humor, J. Hulstijn & A. Nijholt (eds.), Un. of Twente, Neth., 1996. [16] S.C. Marsella, W. Lewis Johnson & C. LaBore. Interactive pedagogical drama. In : Proceedings of 4th International Conference on Autonomous Agents 2000, ACM Press, 301-308. [17] J. Morkes, H.K. Kernal, and C. Nass. Humor in task-oriented computer-mediated communication and human-computer interaction. In: Proceedings of CHI 98, 1998. [18] M.P. Mulder & A. Nijholt. Humour research: State of the art. Deliverable D1, IST Programme on Future and Emerging Technologies. Action Line: IST-2000-6.1.1, HAHAcronym: Humorous Agent for Humorous Acronyms, October 2001. [19] A. Nijholt. Agents, Believability and Embodiment in Advanced Learning Environments. In: Proc. IEEE International Conf. on Advanced Learning Technologies (ICALT 2001), T. Okamto, R. Hartley, Kinshuk & J.P. Klus (eds.), Madison, Wisconsin, IEEE Computer Society, 457-459. [20] A. Ortony, G.L. Clore & A. Collins. The Cognitive Structure of Emotions. Cambridge University Press, 1988. [21] B. Reeves & C. Nass. The Media Equation: how people treat computers, televisions and new media like real people and places. Cameridge, Cambridge University Press, 1996. [22] O. Stock. Password Swordfish: Verbal Humour in the Interface. In: Proc. Intern. Workshop on Computational Humor, J. Hulstijn & A. Nijholt (eds.), University of Twente, Netherlands, 1996. View publication stats

Computers that smile: Humor in the interface

Agents, Believability and Embodiment in Advanced Learning Environments. In:

An integration of a pun generator with a natural language robot. In:

Computational Humor: Interpretation and generation of verbal humor.

Facial expressions and emotion.

Generation of facial expression from emotion using a fuzzy rule based system. In:

Humor in task-oriented computer-mediated communication and human-computer interaction. In:

Humour research: State of the art. Deliverable D1, IST Programme on Future and Emerging Technologies. Action Line: IST-2000-6.1.1, HAHAcronym: Humorous Agent for Humorous Acronyms,

Interactive pedagogical drama.

Not all smiles are created equal: the differences between enjoyment and nonenjoyment smiles.

On the role of humor appreciation in interpersonal attraction: It’s no joking matter.

Password Swordfish: Verbal Humour in the Interface. In:

Relational agents: A model and implementation of building user trust.

The Cognitive Structure of Emotions.

The function of laughter and joking in negotiation activities.

The Media Equation: how people treat computers, televisions and new media like real people and places. Cameridge,

The persona effect: Affective impact of animated pedagogical agents. In:

The relationship between humor and trust.

The varieties of smiling and the problem of motivational acclimatization. Abstract. In:

This computer responds to user frustration. Theory, design, results, and implications.

Using humour to make natural language interfaces more friendly. In:

Virtual reality, art, and entertainment.

Why boys like motorcycles: using emotion theory to find structure in humorous stories. Unpublished paper,

https://ris.utwente.nl/ws/files/282322469/Nijholt2002computers.pdf

Computers that smile: Humor in the interface

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information