Affective Computing

This book provides an overview of state of the art research in Affective Computing. It presents new ideas, original results and practical experiences in this increasingly important research field. The book consists of 23 chapters categorized into four sections. Since one of the most important means of human communication is facial expression, the first section of this book (Chapters 1 to 7) presents a research on synthesis and recognition of facial expressions. Given that we not only use the face but also body movements to express ourselves, in the second section (Chapters 8 to 11) we present a research on perception and generation of emotional expressions by using full-body motions. The third section of the book (Chapters 12 to 16) presents computational models on emotion, as well as findings from neuroscience research. In the last section of the book (Chapters 17 to 22) we present applications related to affective computing

The power of affective touch within social robotics

There have been many leaps and bounds within social robotics, especially within human-robot interaction and how to make it a more meaningful relationship. This is traditionally accomplished through communicating via vision and sound. It has been shown that humans naturally seek interaction through touch yet the implications on emotions is unknown both in human-human interaction and social human-robot interaction. This thesis unpacks the social robotics community and the research undertaken to show a significant gap in the use of touch as a form of communication. The meaning behind touch will be investigated and what implication it has on emotions. A simplistic prototype was developed focusing on texture and breathing. This was used to carry out experiments to find out which combination of texture and movement felt natural. This proved to be a combination of synthetic fur and 14 breaths per minute. For human’s touch is said to be the most natural way of communicating emotions, this is the first step in achieving successful human-robot interaction in a more natural human-like way

顔の表情に基づいた感情と行動を表出するシステムに関する研究

九州工業大学博士学位論文 学位記番号:情工博甲第320号 学位授与年月日:平成29年3月24日1 Introduction|2 Configuration of CONBE Robot System|3 Animal-like Behavior of CONBE Robot using CBA|4 Emotion Generating System of CONBE Robot|5 Experiment and discussion|6 Conclusions九州工業大学平成28年

On the Relationship between People, Objects, & Interactive Technologies: Transforming Digital & Physical experiences through the process of Realizing Empathy

La manera com les persones es relacionen amb el seu entorn, ja sigui físic o digital, és cada cop més complexa i fugaç, fent que la relació de l'usuari amb els seus objectes i eines digitals, de vegades, sigui extrema i de curta durada. Tanmateix, la propietat d'objectes i objectes tecnològics interactius no és buida de significat, són mostres de reflexió i representació per als altres i del seu paper a la societat. La clau per mantenir una relació i el significat amb aquests objectes rau en el disseny i la intenció de l'experiència interactiva creada. Inspirats en les disciplines de la psicologia, el procés de disseny, la interacció humà-ordinador i els models de negoci, aquesta tesi explora, analitza, crea i prova els fonaments teòrics sobre l'empatia i el concepte d'entaular una relació de llarga durada entre les persones i les tecnologies interactives.  Amb aquesta finalitat, aquesta tesi es divideix en 4 fases: (1) l’estudi en profunditat de les referències bibliogràfiques dins del sector HCI, amb especial atenció al rol del disseny i la psicologia amb la intenció de respondre preguntes com: “Com podem construir relacions de llarga durada entre persones i objectes intel·ligents?” (2) Recopilar i adoptar definicions, eines i terminologia de treballs relacionats que aportin a la construcció de la contribució principal d'aquesta tesi, (3) Crear i presentar un model d'interacció entre persones i tecnologia que aporti a una interacció de llarga durada, i (4) presentar un cas d’estudi on s’implementi el model proposat.  Després del treball bibliogràfic, al sector de l'HCI, s'ha identificat un buit, fruit de les principals preocupacions expressades: la manca de connexió entre la teoria i la pràctica del disseny, així com una mancança en l’àmbit de l'Empatia. El resultat fa que molts dels models d’interacció amb intenció empàtica i afectiva no se sustentin entre si. Això ens ha portat a la segona fase de la tesi on aprofitem les referències de múltiples disciplines per estudiar què és l'empatia, com s'implementa, com es percep i com evoluciona cap a l'objectiu d'una relació a llarg termini, com a punt focal cap a les principals contribucions de la tesi. .  Després de reunir i analitzar exhaustivament les referències al voltant de l'empatia, entrem a la tercera fase on presentem el model teòric d'interacció amb el potencial d'establir una interacció a llarg termini i l’anomenat Procés de realització de l'empatia (RE). Més que intentar definir què és l'empatia, aquesta proposta intenta oferir una perspectiva diferent de l'empatia i visualitza el seu abast com un procés influenciat per models de diàleg i col·laboració amb el propòsit de crear comprensió mútua i donar significat a aquest intercanvi.  Amb un model clar i una sòlida base teòrica, la fase final de la tesi cerca provar el model proposat amb l'objectiu d'observar si es poden detectar indicadors d'afecció afectiva i confiança entre una persona i el seu objecte tecnològic. En aquest cas, vam tenir l'oportunitat de treballar amb robots socials com el nostre “altre actor” per dissenyar les proves del model. Aquestes proves pretenien capturar els indicadors d'empatia entre un humà i un robot que abraça: l'aferrament afectiu, la confiança, la regulació de les expectatives i la reflexió sobre la perspectiva de l'altre dins un conjunt d'estratègies de col·laboració. Plantegem la hipòtesi que una estratègia de col·laboració activa condueix a un compromís més significatiu de generar empatia entre un humà i un robot en comparació amb una estratègia passiva. Els resultats són encoratjadors i clarament estableixen un camí per a futures investigacions sobre el disseny d'aquest model. La forma en que las personas se relacionan con su entorno, ya sea físico o digital, se vuelve cada vez más compleja y fugaz, haciendo que la relación del usuario con sus objetos y herramientas digitales, en ocasiones, sea extrema y de corta duración. Sin embargo, la propiedad de objetos y objetos tecnológicos interactivos no es vacía de significado, son muestras de reflexión y representación para los demás y de su papel en la sociedad. La clave para mantener una relación y significado con estos objetos radica en el diseño y la intención de la experiencia interactiva creada. Inspirados en las disciplinas de la psicología, el proceso de diseño, la interacción humano-ordenador y los modelos de negocio, esta tesis explora, analiza, crea y prueba los fundamentos teóricos sobre la empatía y el concepto de entablar una relación de larga duración entre las personas y las tecnologías interactivas.  Con este fin, esta tesis se divide en 4 fases: (1) estudio en profundidad de las referencias bibliográficas dentro del sector HCI, con especial atención al rol del diseño y la psicología con la intención de responder a preguntas como: “¿Cómo podemos construir relaciones de larga duración entre personas y objetos inteligentes?”(2) Recopilar y adoptar definiciones, herramientas y terminología de trabajos relacionados que aporten a la construcción de la contribución principal de esta tesis, (3) Crear y presentar un modelo de interacción entre personas y tecnología que aporte a una interacción de larga duración, y (4) presentar un caso de estudio donde se implemente el modelo propuesto.  Tras el trabajo bibliográfico en el sector del HCI se ha identificado un vacío, fruto de las principales preocupaciones expresadas: la falta de conexión entre la teoría y la práctica del diseño, así como una falta en el tema de la Empatía. El resultado hace que muchos de los modelos de interacción con intención empática y afectiva no se sustenten entre sí. Esto nos ha llevado a la segunda fase de la tesis en la que aprovechamos las referencias de múltiples disciplinas para estudiar qué es la empatía, cómo se implementa, cómo se percibe y cómo evoluciona hacia el objetivo de una relación a largo plazo, como punto focal hacia las principales contribuciones de la tesis. .  Después de una reunir y analizar exhaustivamente las referencias en torno a la empatía, entramos en la tercera fase donde presentamos el modelo teórico de interacción con el potencial de entablar una interacción a largo plazo y denominado Proceso de realización de la empatía (RE). Más que intentar definir qué es la empatía, esta propuesta trata de ofrecer una perspectiva diferente a la empatía y visualiza su alcance como un proceso influenciado por modelos de diálogo y colaboración con el propósito de crear comprensión mutua y dar significado a ese intercambio.  Con un modelo claro y una sólida base teórica, la fase final de la tesis busca probar el modelo propuesto con el objetivo de observar si el modelo puede detectar indicadores de Apego Afectivo y Confianza entre una persona y su objeto tecnológico. En el caso de este trabajo, tuvimos la oportunidad de trabajar con robots sociales como nuestro “otro actor” para diseñar las pruebas del modelo. Estas pruebas pretendían capturar los indicadores de de empatía entre un humano y un robot que abarca: el apego afectivo, la confianza, la regulación de las expectativas y la reflexión sobre la perspectiva del otro dentro de un conjunto de estrategias de colaboración. Planteamos la hipótesis de que una estrategia de colaboración activa conduce a un compromiso más significativo de generar empatía entre un humano y un robot en comparación con una estrategia pasiva. Los resultados son alentadores y claramente establecen un camino para futuras investigaciones sobre el diseño de este modelo. How people engage with their surroundings, whether physical or digital, becomes increasingly complex and rapid, making the user’s relationship with their objects and digital tools, at times, extreme and short-lived. Yet, there is still meaning in ownership of objects and interactive technological objects, they are tokens of reflection and representation to others and their role in society. The key to sustaining a relationship and sense of meaning with these objects lies in the design and intention of the interactive experience created. Inspired by disciplines of psychology, design, Human-computer interaction, and business modeling, this thesis explored, analyzed, created, and tested theoretical foundations on Empathy and the concept of initiating a long-term relationship between people and their interactive technologies.  To that end, the thesis book was managed in 4 main stages: (1) presenting a deeper dive into bibliographic references within HCI and the role of both design and psychology in the attempt to tackle questions like: “How can we build long-term relationships between people and their smart objects?” (2) Collect and adopt from related works that helped build the main contributions of the thesis book, (3) Create an interaction model between humans and their technology that lent itself for potential long-term engagement, and (4) a case study that implemented and instantiated the model designed.     After mapping the HCI bibliographical works in the first phase, a gap was revealed indicative of the main concerns expressed: a lack of connection between theory and design practice as well as a lack in the topic of Empathy. The result makes many of the models of interaction with empathetic and affective intention unsupported between each other. This has led us to the second phase of the thesis where we leveraged references across multiple disciplines to survey what empathy is, how it is implemented, perceived and evolved toward the goal of long-term relationship, as a focal point toward the main thesis contributions.   After an exhaustive gathering and analysis of the work around Empathy, we entered the third phase where we present the proposed theoretical model of interaction with the potential for long-term engagement named the Process of Realizing Empathy (RE). Rather than attempting to further define empathy, this proposal is about offering a different perspective to empathy that visualizes its scope as a process influenced by dialogue and collaborative models with the goal to reach meaning between the actors involved.  With a clear model in place and a strong theoretical foundation, the final phase of the thesis looked to test the proposed model with the goal of observing if the model can provoke its indicators of Affective Attachment and Trust between a person and their technological object. In the case of this thesis work, we had the opportunity to work with social robots as our “other actor” to design the tests for the model. This testbed meant to capture the indicators of early empathy realization between a human and a robot encompassing affective attachment, trust, expectation regulation, and reflecting on the other’s perspective within a set of collaborative strategies. We hypothesized that an active collaboration strategy is conducive to a more meaningful and purposeful engagement of realizing empathy between a human and a robot compared to a passive one. The results are encouraging and clearly establish a path for further research on this model’s design.

Virtual Reality Games for Motor Rehabilitation

This paper presents a fuzzy logic based method to track user satisfaction without the need for devices to monitor users physiological conditions. User satisfaction is the key to any product’s acceptance; computer applications and video games provide a unique opportunity to provide a tailored environment for each user to better suit their needs. We have implemented a non-adaptive fuzzy logic model of emotion, based on the emotional component of the Fuzzy Logic Adaptive Model of Emotion (FLAME) proposed by El-Nasr, to estimate player emotion in UnrealTournament 2004. In this paper we describe the implementation of this system and present the results of one of several play tests. Our research contradicts the current literature that suggests physiological measurements are needed. We show that it is possible to use a software only method to estimate user emotion

Real-time generation and adaptation of social companion robot behaviors

Social robots will be part of our future homes. They will assist us in everyday tasks, entertain us, and provide helpful advice. However, the technology still faces challenges that must be overcome to equip the machine with social competencies and make it a socially intelligent and accepted housemate. An essential skill of every social robot is verbal and non-verbal communication. In contrast to voice assistants, smartphones, and smart home technology, which are already part of many people's lives today, social robots have an embodiment that raises expectations towards the machine. Their anthropomorphic or zoomorphic appearance suggests they can communicate naturally with speech, gestures, or facial expressions and understand corresponding human behaviors. In addition, robots also need to consider individual users' preferences: everybody is shaped by their culture, social norms, and life experiences, resulting in different expectations towards communication with a robot. However, robots do not have human intuition - they must be equipped with the corresponding algorithmic solutions to these problems. This thesis investigates the use of reinforcement learning to adapt the robot's verbal and non-verbal communication to the user's needs and preferences. Such non-functional adaptation of the robot's behaviors primarily aims to improve the user experience and the robot's perceived social intelligence. The literature has not yet provided a holistic view of the overall challenge: real-time adaptation requires control over the robot's multimodal behavior generation, an understanding of human feedback, and an algorithmic basis for machine learning. Thus, this thesis develops a conceptual framework for designing real-time non-functional social robot behavior adaptation with reinforcement learning. It provides a higher-level view from the system designer's perspective and guidance from the start to the end. It illustrates the process of modeling, simulating, and evaluating such adaptation processes. Specifically, it guides the integration of human feedback and social signals to equip the machine with social awareness. The conceptual framework is put into practice for several use cases, resulting in technical proofs of concept and research prototypes. They are evaluated in the lab and in in-situ studies. These approaches address typical activities in domestic environments, focussing on the robot's expression of personality, persona, politeness, and humor. Within this scope, the robot adapts its spoken utterances, prosody, and animations based on human explicit or implicit feedback.Soziale Roboter werden Teil unseres zukünftigen Zuhauses sein. Sie werden uns bei alltäglichen Aufgaben unterstützen, uns unterhalten und uns mit hilfreichen Ratschlägen versorgen. Noch gibt es allerdings technische Herausforderungen, die zunächst überwunden werden müssen, um die Maschine mit sozialen Kompetenzen auszustatten und zu einem sozial intelligenten und akzeptierten Mitbewohner zu machen. Eine wesentliche Fähigkeit eines jeden sozialen Roboters ist die verbale und nonverbale Kommunikation. Im Gegensatz zu Sprachassistenten, Smartphones und Smart-Home-Technologien, die bereits heute Teil des Lebens vieler Menschen sind, haben soziale Roboter eine Verkörperung, die Erwartungen an die Maschine weckt. Ihr anthropomorphes oder zoomorphes Aussehen legt nahe, dass sie in der Lage sind, auf natürliche Weise mit Sprache, Gestik oder Mimik zu kommunizieren, aber auch entsprechende menschliche Kommunikation zu verstehen. Darüber hinaus müssen Roboter auch die individuellen Vorlieben der Benutzer berücksichtigen. So ist jeder Mensch von seiner Kultur, sozialen Normen und eigenen Lebenserfahrungen geprägt, was zu unterschiedlichen Erwartungen an die Kommunikation mit einem Roboter führt. Roboter haben jedoch keine menschliche Intuition - sie müssen mit entsprechenden Algorithmen für diese Probleme ausgestattet werden. In dieser Arbeit wird der Einsatz von bestärkendem Lernen untersucht, um die verbale und nonverbale Kommunikation des Roboters an die Bedürfnisse und Vorlieben des Benutzers anzupassen. Eine solche nicht-funktionale Anpassung des Roboterverhaltens zielt in erster Linie darauf ab, das Benutzererlebnis und die wahrgenommene soziale Intelligenz des Roboters zu verbessern. Die Literatur bietet bisher keine ganzheitliche Sicht auf diese Herausforderung: Echtzeitanpassung erfordert die Kontrolle über die multimodale Verhaltenserzeugung des Roboters, ein Verständnis des menschlichen Feedbacks und eine algorithmische Basis für maschinelles Lernen. Daher wird in dieser Arbeit ein konzeptioneller Rahmen für die Gestaltung von nicht-funktionaler Anpassung der Kommunikation sozialer Roboter mit bestärkendem Lernen entwickelt. Er bietet eine übergeordnete Sichtweise aus der Perspektive des Systemdesigners und eine Anleitung vom Anfang bis zum Ende. Er veranschaulicht den Prozess der Modellierung, Simulation und Evaluierung solcher Anpassungsprozesse. Insbesondere wird auf die Integration von menschlichem Feedback und sozialen Signalen eingegangen, um die Maschine mit sozialem Bewusstsein auszustatten. Der konzeptionelle Rahmen wird für mehrere Anwendungsfälle in die Praxis umgesetzt, was zu technischen Konzeptnachweisen und Forschungsprototypen führt, die in Labor- und In-situ-Studien evaluiert werden. Diese Ansätze befassen sich mit typischen Aktivitäten in häuslichen Umgebungen, wobei der Schwerpunkt auf dem Ausdruck der Persönlichkeit, dem Persona, der Höflichkeit und dem Humor des Roboters liegt. In diesem Rahmen passt der Roboter seine Sprache, Prosodie, und Animationen auf Basis expliziten oder impliziten menschlichen Feedbacks an

An Investigation into the uncanny: character design, behaviour and context.

Whilst there has been a substantial amount of research into the uncanny valley, defining research that contextualises a character as they would normally be viewed remains an unexplored area. Often previous research focused solely on realistic render styles giving characters an unfair basis that tended towards the realistic, thus facilitating only one mode of animation style: realism. Furthermore, characters were not contextualized because researchers often used footage from previous productions. These characters also differed in quality as various artists worked on different productions. This research considers characterisation as three key components, the aesthetic, the behaviour and the contextualisation. Attempts were made to develop a greater understanding of how these components contribute to the appeal of a character within the field of 3D computer animation. Research consisted of two experiments. Both experiments were conducted using an online survey method. The first experiment used five different characters ranging from realistic to abstract. Each character displayed three different behaviours and the characters were contextualized within a six panel narrative. Data obtained from the first experiment was used to refine the second experiment. A further experiment was conducted to further define how combinations of different behaviours and the context containing a character affected the subject’s perception. The second experiment used three different character types and the characters were contextualized within a video stimulus. Findings from the first experiment indicated a strong relationship between character type and context. Interest with the various characters changed depending on adaptions to either the behaviour of the said character or the contextualisation. Certain character types based on appearance where better suited to different contexts than others. An abstract character was more likely to be perceived positively by the subject in a surprising context stipulated by the behaviour of the character and form of the narrative sequence. Other characters such as one based around an inanimate object found a greater positive reception with the subjects under sad contextual constraints rather than happy or surprise. The first experiment took into account various independent variables obtained from the subject and aimed to draw parallels if found between these variables and the subjects perception of a given character be it positive or negative. However, these variables namely gender, nationality and age had no effect on the subject’s perception. In the second experiment, it was found that in order for the realistic human character to be perceived more positively, the behaviour needed to match the context. When a mismatch occurred the subjects began to perceive the character more negatively. The cartoon character was however not affected by the mismatch of behaviour and context. The experiment was further expanded when two different character types were compared committing negative actions and having negative actions inflicted upon them and what effect it had on the subjects perception. It was found that a cartoon character committing a negative action was perceived positively whilst a human character committing the same act was perceived negatively. However, when a negative action was inflicted on these same characters, subjects were more concerned for the human character than the cartoon character. Results from both experiments confirm the idea that various characters are perceived very differently by the viewers and come with predefined notions within the viewer of how they should behave. What is expected of one character type is not acceptable for another character type. Cartoon characters can get away with bizarre behaviour. A real human character may have some sort of novel unusual behaviour, whilst a realistic CG human character is assessed on how realistically (normally) it behaves. This research expands upon previous research into this area by offering a greater understanding of character types and emphasising the importance of contextualisation