Information acquisition using eye-gaze tracking for person-following with mobile robots

In the effort of developing natural means for human-robot interaction (HRI), signifcant amount of research has been focusing on Person-Following (PF) for mobile robots. PF, which generally consists of detecting, recognizing and following people, is believed to be one of the required functionalities for most future robots that share their environments with their human companions. Research in this field is mostly directed towards fully automating this functionality, which makes the challenge even more tedious. Focusing on this challenge leads research to divert from other challenges that coexist in any PF system. A natural PF functionality consists of a number of tasks that are required to be implemented in the system. However, in more realistic life scenarios, not all the tasks required for PF need to be automated. Instead, some of these tasks can be operated by human operators and therefore require natural means of interaction and information acquisition. In order to highlight all the tasks that are believed to exist in any PF system, this paper introduces a novel taxonomy for PF. Also, in order to provide a natural means for HRI, TeleGaze is used for information acquisition in the implementation of the taxonomy. TeleGaze was previously developed by the authors as a means of natural HRI for teleoperation through eye-gaze tracking. Using TeleGaze in the aid of developing PF systems is believed to show the feasibility of achieving a realistic information acquisition in a natural way

Who am I talking with? A face memory for social robots

In order to provide personalized services and to develop human-like interaction capabilities robots need to rec- ognize their human partner. Face recognition has been studied in the past decade exhaustively in the context of security systems and with significant progress on huge datasets. However, these capabilities are not in focus when it comes to social interaction situations. Humans are able to remember people seen for a short moment in time and apply this knowledge directly in their engagement in conversation. In order to equip a robot with capabilities to recall human interlocutors and to provide user- aware services, we adopt human-human interaction schemes to propose a face memory on the basis of active appearance models integrated with the active memory architecture. This paper presents the concept of the interactive face memory, the applied recognition algorithms, and their embedding into the robot’s system architecture. Performance measures are discussed for general face databases as well as scenario-specific datasets

Shared Perception in Human-Robot Interaction

Interaction can be seen as a composition of perspectives: the integration of perceptions, intentions, and actions on the environment two or more agents share. For an interaction to be effective, each agent must be prone to “sharedness”: being situated in a common environment, able to read what others express about their perspective, and ready to adjust one’s own perspective accordingly. In this sense, effective interaction is supported by perceiving the environment jointly with others, a capability that in this research is called Shared Perception. Nonetheless, perception is a complex process that brings the observer receiving sensory inputs from the external world and interpreting them based on its own, previous experiences, predictions, and intentions. In addition, social interaction itself contributes to shaping what is perceived: others’ attention, perspective, actions, and internal states may also be incorporated into perception. Thus, Shared perception reflects the observer's ability to integrate these three sources of information: the environment, the self, and other agents. If Shared Perception is essential among humans, it is equally crucial for interaction with robots, which need social and cognitive abilities to interact with humans naturally and successfully. This research deals with Shared Perception within the context of Social Human-Robot Interaction (HRI) and involves an interdisciplinary approach. The two general axes of the thesis are the investigation of human perception while interacting with robots and the modeling of robot’s perception while interacting with humans. Such two directions are outlined through three specific Research Objectives, whose achievements represent the contribution of this work. i) The formulation of a theoretical framework of Shared Perception in HRI valid for interpreting and developing different socio-perceptual mechanisms and abilities. ii) The investigation of Shared Perception in humans focusing on the perceptual mechanism of Context Dependency, and therefore exploring how social interaction affects the use of previous experience in human spatial perception. iii) The implementation of a deep-learning model for Addressee Estimation to foster robots’ socio-perceptual skills through the awareness of others’ behavior, as suggested in the Shared Perception framework. To achieve the first Research Objective, several human socio-perceptual mechanisms are presented and interpreted in a unified account. This exposition parallels mechanisms elicited by interaction with humans and humanoid robots and aims to build a framework valid to investigate human perception in the context of HRI. Based on the thought of D. Davidson and conceived as the integration of information coming from the environment, the self, and other agents, the idea of "triangulation" expresses the critical dynamics of Shared Perception. Also, it is proposed as the functional structure to support the implementation of socio-perceptual skills in robots. This general framework serves as a reference to fulfill the other two Research Objectives, which explore specific aspects of Shared Perception. For what concerns the second Research Objective, the human perceptual mechanism of Context Dependency is investigated, for the first time, within social interaction. Human perception is based on unconscious inference, where sensory inputs integrate with prior information. This phenomenon helps in facing the uncertainty of the external world with predictions built upon previous experience. To investigate the effect of social interaction on such a mechanism, the iCub robot has been used as an experimental tool to create an interactive scenario with a controlled setting. A user study based on psychophysical methods, Bayesian modeling, and a neural network analysis of human results demonstrated that social interaction influenced Context Dependency so that when interacting with a social agent, humans rely less on their internal models and more on external stimuli. Such results are framed in Shared Perception and contribute to revealing the integration dynamics of the three sources of Shared Perception. The others’ presence and social behavior (other agents) affect the balance between sensory inputs (environment) and personal history (self) in favor of the information shared with others, that is, the environment. The third Research Objective consists of tackling the Addressee Estimation problem, i.e., understanding to whom a speaker is talking, to improve the iCub social behavior in multi-party interactions. Addressee Estimation can be considered a Shared Perception ability because it is achieved by using sensory information from the environment, internal representations of the agents’ position, and, more importantly, the understanding of others’ behavior. An architecture for Addressee Estimation is thus designed considering the integration process of Shared Perception (environment, self, other agents) and partially implemented with respect to the third element: the awareness of others’ behavior. To achieve this, a hybrid deep-learning (CNN+LSTM) model is developed to estimate the speaker-robot relative placement of the addressee based on the non-verbal behavior of the speaker. Addressee Estimation abilities based on Shared Perception dynamics are aimed at improving multi-party HRI. Making robots aware of other agents’ behavior towards the environment is the first crucial step for incorporating such information into the robot’s perception and modeling Shared Perception

Autonomous decision-making for socially interactive robots

Mención Internacional en el título de doctorThe aim of this thesis is to present a novel decision-making system based on bio-inspired concepts to decide the actions to make during the interaction between humans and robots. We use concepts from nature to make the robot may behave analogously to a living being for a better acceptance by people. The system is applied to autonomous Socially Interactive Robots that works in environments with users. These objectives are motivated by the need of having robots collaborating, entertaining or helping in educational tasks for real situations with children or elder people where the robot has to behave socially. Moreover, the decision-making system can be integrated into this kind of robots in order to learn how to act depending on the user profile the robot is interacting with. The decision-making system proposed in this thesis is a solution to all these issues in addition to a complement for interactive learning in HRI. We also show real applications of the system proposed applying it in an educational scenario, a situation where the robot can learn and interact with different kinds of people. The last goal of this thesis is to develop a robotic architecture that is able to learn how to behave in different contexts where humans and robots coexist. For that purpose, we design a modular and portable robotic architecture that is included in several robots. Including well-known software engineering techniques together with innovative agile software development procedures that produces an easily extensible architecture.El objetivo de esta tesis es presentar un novedoso sistema de toma de decisiones basado en conceptos bioinspirados para decidir las acciones a realizar durante la interacción entre personas y robots. Usamos conceptos de la naturaleza para hacer que el robot pueda comportarse análogamente a un ser vivo para una mejor aceptación por las personas. El sistema está desarrollado para que se pueda aplicar a los llamados Robots Socialmente Interactivos que están destinados a entornos con usuarios. Estos objetivos están motivados por la necesidad de tener robots en tareas de colaboración, entretenimiento o en educación en situaciones reales con niños o personas mayores en las cuales el robot debe comportarse siguiendo las normas sociales. Además, el sistema de toma de decisiones es integrado en estos tipos de robots con el fin de que pueda aprender a actuar dependiendo del perfil de usuario con el que el robot está interactuando. El sistema de toma de decisiones que proponemos en esta tesis es una solución a todos estos desafíos además de un complemento para el aprendizaje interactivo en la interacción humano-robot. También mostramos aplicaciones reales del sistema propuesto aplicándolo en un escenario educativo, una situación en la que el robot puede aprender e interaccionar con diferentes tipos de personas. El último objetivo de esta tesis es desarrollar un arquitectura robótica que sea capaz de aprender a comportarse en diferentes contextos donde las personas y los robots coexistan. Con ese propósito, diseñamos una arquitectura robótica modular y portable que está incluida en varios robots. Incluyendo técnicas bien conocidas de ingeniería del software junto con procedimientos innovadores de desarrollo de sofware ágil que producen una arquitectura fácilmente extensible.Programa Oficial de Doctorado en Ingeniería Eléctrica, Electrónica y AutomáticaPresidente: Fabio Bonsignorio.- Secretario: María Dolores Blanco Rojas.- Vocal: Martin Stoele

Neural Network Based Reinforcement Learning for Audio-Visual Gaze Control in Human-Robot Interaction

This paper introduces a novel neural network-based reinforcement learning approach for robot gaze control. Our approach enables a robot to learn and to adapt its gaze control strategy for human-robot interaction neither with the use of external sensors nor with human supervision. The robot learns to focus its attention onto groups of people from its own audio-visual experiences, independently of the number of people, of their positions and of their physical appearances. In particular, we use a recurrent neural network architecture in combination with Q-learning to find an optimal action-selection policy; we pre-train the network using a simulated environment that mimics realistic scenarios that involve speaking/silent participants, thus avoiding the need of tedious sessions of a robot interacting with people. Our experimental evaluation suggests that the proposed method is robust against parameter estimation, i.e. the parameter values yielded by the method do not have a decisive impact on the performance. The best results are obtained when both audio and visual information is jointly used. Experiments with the Nao robot indicate that our framework is a step forward towards the autonomous learning of socially acceptable gaze behavior.Comment: Paper submitted to Pattern Recognition Letter

A Review of Verbal and Non-Verbal Human-Robot Interactive Communication

In this paper, an overview of human-robot interactive communication is presented, covering verbal as well as non-verbal aspects of human-robot interaction. Following a historical introduction, and motivation towards fluid human-robot communication, ten desiderata are proposed, which provide an organizational axis both of recent as well as of future research on human-robot communication. Then, the ten desiderata are examined in detail, culminating to a unifying discussion, and a forward-looking conclusion

DAC-h3: A Proactive Robot Cognitive Architecture to Acquire and Express Knowledge About the World and the Self

This paper introduces a cognitive architecture for a humanoid robot to engage in a proactive, mixed-initiative exploration and manipulation of its environment, where the initiative can originate from both the human and the robot. The framework, based on a biologically-grounded theory of the brain and mind, integrates a reactive interaction engine, a number of state-of-the art perceptual and motor learning algorithms, as well as planning abilities and an autobiographical memory. The architecture as a whole drives the robot behavior to solve the symbol grounding problem, acquire language capabilities, execute goal-oriented behavior, and express a verbal narrative of its own experience in the world. We validate our approach in human-robot interaction experiments with the iCub humanoid robot, showing that the proposed cognitive architecture can be applied in real time within a realistic scenario and that it can be used with naive users

On the Relationship between People, Objects, & Interactive Technologies: Transforming Digital & Physical experiences through the process of Realizing Empathy

La manera com les persones es relacionen amb el seu entorn, ja sigui físic o digital, és cada cop més complexa i fugaç, fent que la relació de l'usuari amb els seus objectes i eines digitals, de vegades, sigui extrema i de curta durada. Tanmateix, la propietat d'objectes i objectes tecnològics interactius no és buida de significat, són mostres de reflexió i representació per als altres i del seu paper a la societat. La clau per mantenir una relació i el significat amb aquests objectes rau en el disseny i la intenció de l'experiència interactiva creada. Inspirats en les disciplines de la psicologia, el procés de disseny, la interacció humà-ordinador i els models de negoci, aquesta tesi explora, analitza, crea i prova els fonaments teòrics sobre l'empatia i el concepte d'entaular una relació de llarga durada entre les persones i les tecnologies interactives.  Amb aquesta finalitat, aquesta tesi es divideix en 4 fases: (1) l’estudi en profunditat de les referències bibliogràfiques dins del sector HCI, amb especial atenció al rol del disseny i la psicologia amb la intenció de respondre preguntes com: “Com podem construir relacions de llarga durada entre persones i objectes intel·ligents?” (2) Recopilar i adoptar definicions, eines i terminologia de treballs relacionats que aportin a la construcció de la contribució principal d'aquesta tesi, (3) Crear i presentar un model d'interacció entre persones i tecnologia que aporti a una interacció de llarga durada, i (4) presentar un cas d’estudi on s’implementi el model proposat.  Després del treball bibliogràfic, al sector de l'HCI, s'ha identificat un buit, fruit de les principals preocupacions expressades: la manca de connexió entre la teoria i la pràctica del disseny, així com una mancança en l’àmbit de l'Empatia. El resultat fa que molts dels models d’interacció amb intenció empàtica i afectiva no se sustentin entre si. Això ens ha portat a la segona fase de la tesi on aprofitem les referències de múltiples disciplines per estudiar què és l'empatia, com s'implementa, com es percep i com evoluciona cap a l'objectiu d'una relació a llarg termini, com a punt focal cap a les principals contribucions de la tesi. .  Després de reunir i analitzar exhaustivament les referències al voltant de l'empatia, entrem a la tercera fase on presentem el model teòric d'interacció amb el potencial d'establir una interacció a llarg termini i l’anomenat Procés de realització de l'empatia (RE). Més que intentar definir què és l'empatia, aquesta proposta intenta oferir una perspectiva diferent de l'empatia i visualitza el seu abast com un procés influenciat per models de diàleg i col·laboració amb el propòsit de crear comprensió mútua i donar significat a aquest intercanvi.  Amb un model clar i una sòlida base teòrica, la fase final de la tesi cerca provar el model proposat amb l'objectiu d'observar si es poden detectar indicadors d'afecció afectiva i confiança entre una persona i el seu objecte tecnològic. En aquest cas, vam tenir l'oportunitat de treballar amb robots socials com el nostre “altre actor” per dissenyar les proves del model. Aquestes proves pretenien capturar els indicadors d'empatia entre un humà i un robot que abraça: l'aferrament afectiu, la confiança, la regulació de les expectatives i la reflexió sobre la perspectiva de l'altre dins un conjunt d'estratègies de col·laboració. Plantegem la hipòtesi que una estratègia de col·laboració activa condueix a un compromís més significatiu de generar empatia entre un humà i un robot en comparació amb una estratègia passiva. Els resultats són encoratjadors i clarament estableixen un camí per a futures investigacions sobre el disseny d'aquest model. La forma en que las personas se relacionan con su entorno, ya sea físico o digital, se vuelve cada vez más compleja y fugaz, haciendo que la relación del usuario con sus objetos y herramientas digitales, en ocasiones, sea extrema y de corta duración. Sin embargo, la propiedad de objetos y objetos tecnológicos interactivos no es vacía de significado, son muestras de reflexión y representación para los demás y de su papel en la sociedad. La clave para mantener una relación y significado con estos objetos radica en el diseño y la intención de la experiencia interactiva creada. Inspirados en las disciplinas de la psicología, el proceso de diseño, la interacción humano-ordenador y los modelos de negocio, esta tesis explora, analiza, crea y prueba los fundamentos teóricos sobre la empatía y el concepto de entablar una relación de larga duración entre las personas y las tecnologías interactivas.  Con este fin, esta tesis se divide en 4 fases: (1) estudio en profundidad de las referencias bibliográficas dentro del sector HCI, con especial atención al rol del diseño y la psicología con la intención de responder a preguntas como: “¿Cómo podemos construir relaciones de larga duración entre personas y objetos inteligentes?”(2) Recopilar y adoptar definiciones, herramientas y terminología de trabajos relacionados que aporten a la construcción de la contribución principal de esta tesis, (3) Crear y presentar un modelo de interacción entre personas y tecnología que aporte a una interacción de larga duración, y (4) presentar un caso de estudio donde se implemente el modelo propuesto.  Tras el trabajo bibliográfico en el sector del HCI se ha identificado un vacío, fruto de las principales preocupaciones expresadas: la falta de conexión entre la teoría y la práctica del diseño, así como una falta en el tema de la Empatía. El resultado hace que muchos de los modelos de interacción con intención empática y afectiva no se sustenten entre sí. Esto nos ha llevado a la segunda fase de la tesis en la que aprovechamos las referencias de múltiples disciplinas para estudiar qué es la empatía, cómo se implementa, cómo se percibe y cómo evoluciona hacia el objetivo de una relación a largo plazo, como punto focal hacia las principales contribuciones de la tesis. .  Después de una reunir y analizar exhaustivamente las referencias en torno a la empatía, entramos en la tercera fase donde presentamos el modelo teórico de interacción con el potencial de entablar una interacción a largo plazo y denominado Proceso de realización de la empatía (RE). Más que intentar definir qué es la empatía, esta propuesta trata de ofrecer una perspectiva diferente a la empatía y visualiza su alcance como un proceso influenciado por modelos de diálogo y colaboración con el propósito de crear comprensión mutua y dar significado a ese intercambio.  Con un modelo claro y una sólida base teórica, la fase final de la tesis busca probar el modelo propuesto con el objetivo de observar si el modelo puede detectar indicadores de Apego Afectivo y Confianza entre una persona y su objeto tecnológico. En el caso de este trabajo, tuvimos la oportunidad de trabajar con robots sociales como nuestro “otro actor” para diseñar las pruebas del modelo. Estas pruebas pretendían capturar los indicadores de de empatía entre un humano y un robot que abarca: el apego afectivo, la confianza, la regulación de las expectativas y la reflexión sobre la perspectiva del otro dentro de un conjunto de estrategias de colaboración. Planteamos la hipótesis de que una estrategia de colaboración activa conduce a un compromiso más significativo de generar empatía entre un humano y un robot en comparación con una estrategia pasiva. Los resultados son alentadores y claramente establecen un camino para futuras investigaciones sobre el diseño de este modelo. How people engage with their surroundings, whether physical or digital, becomes increasingly complex and rapid, making the user’s relationship with their objects and digital tools, at times, extreme and short-lived. Yet, there is still meaning in ownership of objects and interactive technological objects, they are tokens of reflection and representation to others and their role in society. The key to sustaining a relationship and sense of meaning with these objects lies in the design and intention of the interactive experience created. Inspired by disciplines of psychology, design, Human-computer interaction, and business modeling, this thesis explored, analyzed, created, and tested theoretical foundations on Empathy and the concept of initiating a long-term relationship between people and their interactive technologies.  To that end, the thesis book was managed in 4 main stages: (1) presenting a deeper dive into bibliographic references within HCI and the role of both design and psychology in the attempt to tackle questions like: “How can we build long-term relationships between people and their smart objects?” (2) Collect and adopt from related works that helped build the main contributions of the thesis book, (3) Create an interaction model between humans and their technology that lent itself for potential long-term engagement, and (4) a case study that implemented and instantiated the model designed.     After mapping the HCI bibliographical works in the first phase, a gap was revealed indicative of the main concerns expressed: a lack of connection between theory and design practice as well as a lack in the topic of Empathy. The result makes many of the models of interaction with empathetic and affective intention unsupported between each other. This has led us to the second phase of the thesis where we leveraged references across multiple disciplines to survey what empathy is, how it is implemented, perceived and evolved toward the goal of long-term relationship, as a focal point toward the main thesis contributions.   After an exhaustive gathering and analysis of the work around Empathy, we entered the third phase where we present the proposed theoretical model of interaction with the potential for long-term engagement named the Process of Realizing Empathy (RE). Rather than attempting to further define empathy, this proposal is about offering a different perspective to empathy that visualizes its scope as a process influenced by dialogue and collaborative models with the goal to reach meaning between the actors involved.  With a clear model in place and a strong theoretical foundation, the final phase of the thesis looked to test the proposed model with the goal of observing if the model can provoke its indicators of Affective Attachment and Trust between a person and their technological object. In the case of this thesis work, we had the opportunity to work with social robots as our “other actor” to design the tests for the model. This testbed meant to capture the indicators of early empathy realization between a human and a robot encompassing affective attachment, trust, expectation regulation, and reflecting on the other’s perspective within a set of collaborative strategies. We hypothesized that an active collaboration strategy is conducive to a more meaningful and purposeful engagement of realizing empathy between a human and a robot compared to a passive one. The results are encouraging and clearly establish a path for further research on this model’s design.