To Affinity and Beyond: Interactive Digital Humans as a Human Computer Interface

The field of human computer interaction is increasingly exploring the use of more natural, human-like user interfaces to build intelligent agents to aid in everyday life. This is coupled with a move to people using ever more realistic avatars to represent themselves in their digital lives. As the ability to produce emotionally engaging digital human representations is only just now becoming technically possible, there is little research into how to approach such tasks. This is due to both technical complexity and operational implementation cost. This is now changing as we are at a nexus point with new approaches, faster graphics processing and enabling new technologies in machine learning and computer vision becoming available. I articulate the issues required for such digital humans to be considered successfully located on the other side of the phenomenon known as the Uncanny Valley. My results show that a complex mix of perceived and contextual aspects affect the sense making on digital humans and highlights previously undocumented effects of interactivity on the affinity. Users are willing to accept digital humans as a new form of user interface and they react to them emotionally in previously unanticipated ways. My research shows that it is possible to build an effective interactive digital human that crosses the Uncanny Valley. I directly explore what is required to build a visually realistic digital human as a primary research question and I explore if such a realistic face provides sufficient benefit to justify the challenges involved in building it. I conducted a Delphi study to inform the research approaches and then produced a complex digital human character based on these insights. This interactive and realistic digital human avatar represents a major technical undertaking involving multiple teams around the world. Finally, I explored a framework for examining the ethical implications and signpost future research areas

Design for Child-Robot Play The implications of Design Research within the field of Human-Robot Interaction studies for Children

This thesis investigates the intersections of three disciplines, that are Design Research, Human-Robot Interaction studies, and Child Studies. In particular, this doctoral research is focused on two research questions, namely, what is (or might be) the role of design research in HRI? And, how to design acceptable and desirable child-robot play applications? The first chapter introduces an overview of the mutual interest between robotics and design that is at the basis of the research. On the one hand, the interest of design toward robotics is documented through some exemplary projects from artists and designers that speculate on the human-robot coexistence condition. Vice versa, the robotics interest toward design is documented by referring to some tracks of robotic conferences, scienti c workshops and robotics journals which focused on the design-robotics relationship. Finally, a brief description of the background conditions that characterized this doctoral research are introduced, such as the fact of being a research founded by a company. The second chapter provides an overview of the state of the art of the intersections between three multidisciplinary disciplines. First, a de nition of Design Research is provided, together with its main trends and open issues. Then, the review focuses on the contribution of Design Research to the HRI eld, which can be summed up in actions focused on three aspects: artefacts, stakeholders, and contexts. This is followed by a focus on the role of Design Research within the context of children studies, in which it is possible to identify two main design-child relationships: design as a method for developing children’s learning experiences; and children as part of the design process for developing novel interactive systems. The third chapter introduces the Research through Design (RtD) approach and its relevance in conducting design research in HRI. The proposed methodology, based on this approach, is particularly characterized by the presence of design explorations as study methods. These, in turn, are developed through a common project’s methodology, also reported in this chapter. The fourth chapter is dedicated to the analysis of the scenario in which the child-robot interaction takes place. This was aimed at understanding what is edutainment robotics for children, its common features, how it relates to existing children play types, and where the interaction takes place. The chapter provides also a focus on the relationship between children and technology on a more general level, through which two themes and relative design opportunities were identi ed: physically active play and objects-to-think-with. These were respectively addressed in the two design explorations presented in this thesis: Phygital Play and Shybo. The Phygital Play project consists of an exploration of natural interaction modalities with robots, through mixed-reality, for fostering children’s active behaviours. To this end, a game platform was developed for allowing children to play with or against a robot, through body movement. Shybo, instead, is a low-anthropomorphic robot for playful learning activities with children that can be carried out in educational contexts. The robot, which reacts to properties of the physical environment, is designed to support different kinds of experiences. Then, the chapter eight is dedicated to the research outcomes, that were de ned through a process of reflection. The contribution of the research was analysed and documented by focusing on three main levels, namely: artefact, knowledge and theory. The artefact level corresponds to the situated implementations developed through the projects. The knowledge level consists of a set of actionable principles, emerged from the results and lessons learned from the projects. At the theory level, a theoretical framework was proposed with the aim of informing the future design of child- robot play applications. Thelastchapterprovidesa naloverviewofthe doctoral research, a series of limitations regarding the research, its process and its outcomes, and some indications for future research

Real-time generation and adaptation of social companion robot behaviors

Social robots will be part of our future homes. They will assist us in everyday tasks, entertain us, and provide helpful advice. However, the technology still faces challenges that must be overcome to equip the machine with social competencies and make it a socially intelligent and accepted housemate. An essential skill of every social robot is verbal and non-verbal communication. In contrast to voice assistants, smartphones, and smart home technology, which are already part of many people's lives today, social robots have an embodiment that raises expectations towards the machine. Their anthropomorphic or zoomorphic appearance suggests they can communicate naturally with speech, gestures, or facial expressions and understand corresponding human behaviors. In addition, robots also need to consider individual users' preferences: everybody is shaped by their culture, social norms, and life experiences, resulting in different expectations towards communication with a robot. However, robots do not have human intuition - they must be equipped with the corresponding algorithmic solutions to these problems. This thesis investigates the use of reinforcement learning to adapt the robot's verbal and non-verbal communication to the user's needs and preferences. Such non-functional adaptation of the robot's behaviors primarily aims to improve the user experience and the robot's perceived social intelligence. The literature has not yet provided a holistic view of the overall challenge: real-time adaptation requires control over the robot's multimodal behavior generation, an understanding of human feedback, and an algorithmic basis for machine learning. Thus, this thesis develops a conceptual framework for designing real-time non-functional social robot behavior adaptation with reinforcement learning. It provides a higher-level view from the system designer's perspective and guidance from the start to the end. It illustrates the process of modeling, simulating, and evaluating such adaptation processes. Specifically, it guides the integration of human feedback and social signals to equip the machine with social awareness. The conceptual framework is put into practice for several use cases, resulting in technical proofs of concept and research prototypes. They are evaluated in the lab and in in-situ studies. These approaches address typical activities in domestic environments, focussing on the robot's expression of personality, persona, politeness, and humor. Within this scope, the robot adapts its spoken utterances, prosody, and animations based on human explicit or implicit feedback.Soziale Roboter werden Teil unseres zukünftigen Zuhauses sein. Sie werden uns bei alltäglichen Aufgaben unterstützen, uns unterhalten und uns mit hilfreichen Ratschlägen versorgen. Noch gibt es allerdings technische Herausforderungen, die zunächst überwunden werden müssen, um die Maschine mit sozialen Kompetenzen auszustatten und zu einem sozial intelligenten und akzeptierten Mitbewohner zu machen. Eine wesentliche Fähigkeit eines jeden sozialen Roboters ist die verbale und nonverbale Kommunikation. Im Gegensatz zu Sprachassistenten, Smartphones und Smart-Home-Technologien, die bereits heute Teil des Lebens vieler Menschen sind, haben soziale Roboter eine Verkörperung, die Erwartungen an die Maschine weckt. Ihr anthropomorphes oder zoomorphes Aussehen legt nahe, dass sie in der Lage sind, auf natürliche Weise mit Sprache, Gestik oder Mimik zu kommunizieren, aber auch entsprechende menschliche Kommunikation zu verstehen. Darüber hinaus müssen Roboter auch die individuellen Vorlieben der Benutzer berücksichtigen. So ist jeder Mensch von seiner Kultur, sozialen Normen und eigenen Lebenserfahrungen geprägt, was zu unterschiedlichen Erwartungen an die Kommunikation mit einem Roboter führt. Roboter haben jedoch keine menschliche Intuition - sie müssen mit entsprechenden Algorithmen für diese Probleme ausgestattet werden. In dieser Arbeit wird der Einsatz von bestärkendem Lernen untersucht, um die verbale und nonverbale Kommunikation des Roboters an die Bedürfnisse und Vorlieben des Benutzers anzupassen. Eine solche nicht-funktionale Anpassung des Roboterverhaltens zielt in erster Linie darauf ab, das Benutzererlebnis und die wahrgenommene soziale Intelligenz des Roboters zu verbessern. Die Literatur bietet bisher keine ganzheitliche Sicht auf diese Herausforderung: Echtzeitanpassung erfordert die Kontrolle über die multimodale Verhaltenserzeugung des Roboters, ein Verständnis des menschlichen Feedbacks und eine algorithmische Basis für maschinelles Lernen. Daher wird in dieser Arbeit ein konzeptioneller Rahmen für die Gestaltung von nicht-funktionaler Anpassung der Kommunikation sozialer Roboter mit bestärkendem Lernen entwickelt. Er bietet eine übergeordnete Sichtweise aus der Perspektive des Systemdesigners und eine Anleitung vom Anfang bis zum Ende. Er veranschaulicht den Prozess der Modellierung, Simulation und Evaluierung solcher Anpassungsprozesse. Insbesondere wird auf die Integration von menschlichem Feedback und sozialen Signalen eingegangen, um die Maschine mit sozialem Bewusstsein auszustatten. Der konzeptionelle Rahmen wird für mehrere Anwendungsfälle in die Praxis umgesetzt, was zu technischen Konzeptnachweisen und Forschungsprototypen führt, die in Labor- und In-situ-Studien evaluiert werden. Diese Ansätze befassen sich mit typischen Aktivitäten in häuslichen Umgebungen, wobei der Schwerpunkt auf dem Ausdruck der Persönlichkeit, dem Persona, der Höflichkeit und dem Humor des Roboters liegt. In diesem Rahmen passt der Roboter seine Sprache, Prosodie, und Animationen auf Basis expliziten oder impliziten menschlichen Feedbacks an

Designing companions, designing tools : social robots, developers, and the elderly in Japan

Ce mémoire de maîtrise trace la généalogie d’un robot social, de sa conception à ses différentes utilisations et la manière dont les utilisateurs interagissent avec. A partir d’un terrain de six mois dans une start-up et deux maisons de retraite au Japon, j’interroge la création de Pepper, un robot social crée par la compagnie japonais SoftBank. Pepper a été créé de façon à être humanoïde mais pas trop, ainsi que perçu comme adorable et charmant. Par la suite, je décris comment Pepper et d’autres robots sociaux sont utilisés, à la fois par des développeurs, mais aussi par des personnes âgées, et je souligne une tension existante entre leur utilisation comme des compagnons et des outils. En me basant sur l’anthropologie ontologique et la phénoménologie, j’examine la construction du robot comme une entité avec laquelle il est possible d’interagir, notamment à cause de sa conception en tant qu’acteur social, ontologiquement ambigu, et qui peut exprimer de l’affect. En m’intéressant aux interactions multimodales, et en particulier le toucher, je classifie trois fonctions remplies par l’interaction : découverte, contrôle, et l’expression de l’affect. Par la suite, je questionne ces actes d’agir vers et s’ils peuvent être compris comme une interaction, puisqu’ils n’impliquent pas que le robot soit engagé. J’argumente qu’une interaction est un échange de sens entre des agents engagés et incarnés. Il y a effectivement parfois un échange de sens entre le robot et son utilisateur, et le robot est un artefact incarné. Cependant, seule l’impression d’intersubjectivité est nécessaire à l’interaction, plutôt que sa réelle présence.This master’s thesis traces a genealogy of a social robot through its conception to its various uses and the ways users interact with it. Drawing on six months of fieldwork in a start-up and two nursing homes in Japan, I first investigate the genesis of a social robot created by SoftBank, a Japanese multinational telecommunications company. This social robot is quite humanlike, made to be cute and have an adorable personality. While developers constitute one of the user populations, this robot, along with several others, is also used by elderly residents in nursing homes. By analyzing the uses of these populations, I underline the tension between the social robot as a companion and a tool. Drawing on ontological anthropology and phenomenology I look at how the robot is constructed as an entity that can be interacted with, through its conception as an ontologically ambiguous, social actor, that can express affect. Looking at multimodal interaction, and especially touch, I then classify three functions they fulfill: discovery, control, and the expression of affect, before questioning whether this acting towards the robot that does not imply acting from the robot, can be considered a form of interaction. I argue that interaction is the exchange of meaning between embodied, engaged participants. Meaning can be exchanged between robots and humans and the robot can be seen as embodied, but only the appearance of intersubjectivity is enough, rather than its actual presence