On the Development of Adaptive and User-Centred Interactive Multimodal Interfaces

Multimodal systems have attained increased attention in recent years, which has made possible important improvements in the technologies for recognition, processing, and generation of multimodal information. However, there are still many issues related to multimodality which are not clear, for example, the principles that make it possible to resemble human-human multimodal communication. This chapter focuses on some of the most important challenges that researchers have recently envisioned for future multimodal interfaces. It also describes current efforts to develop intelligent, adaptive, proactive, portable and affective multimodal interfaces

EMERALD—Exercise Monitoring Emotional Assistant

The increase in the elderly population in today’s society entails the need for new policies to maintain an adequate level of care without excessively increasing social spending. One of the possible options is to promote home care for the elderly. In this sense, this paper introduces a personal assistant designed to help elderly people in their activities of daily living. This system, called EMERALD, is comprised of a sensing platform and different mechanisms for emotion detection and decision-making that combined produces a cognitive assistant that engages users in Active Aging. The contribution of the paper is twofold—on the one hand, the integration of low-cost sensors that among other characteristics allows for detecting the emotional state of the user at an affordable cost; on the other hand, an automatic activity suggestion module that engages the users, mainly oriented to the elderly, in a healthy lifestyle. Moreover, by continuously correcting the system using the on-line monitoring carried out through the sensors integrated in the system, the system is personalized, and, in broad terms, emotionally intelligent. A functional prototype is being currently tested in a daycare centre in the northern area of Portugal where preliminary tests show positive results.This research was partially funded by the Fundacao para a Ciencia e Tecnologia (FCT) within the projects UID/CEC/00319/2019 and Post-Doc Grant SFRH/BPD/102696/2014 (Angelo Costa). This work is also partially funded by the MINECO/FEDER TIN2015-65515-C4-1-R and RISEWISE (RISEWomen with disabilities In Social Engagement) EU project under Agreement No. 690874.info:eu-repo/semantics/publishedVersio

Real-time generation and adaptation of social companion robot behaviors

Social robots will be part of our future homes. They will assist us in everyday tasks, entertain us, and provide helpful advice. However, the technology still faces challenges that must be overcome to equip the machine with social competencies and make it a socially intelligent and accepted housemate. An essential skill of every social robot is verbal and non-verbal communication. In contrast to voice assistants, smartphones, and smart home technology, which are already part of many people's lives today, social robots have an embodiment that raises expectations towards the machine. Their anthropomorphic or zoomorphic appearance suggests they can communicate naturally with speech, gestures, or facial expressions and understand corresponding human behaviors. In addition, robots also need to consider individual users' preferences: everybody is shaped by their culture, social norms, and life experiences, resulting in different expectations towards communication with a robot. However, robots do not have human intuition - they must be equipped with the corresponding algorithmic solutions to these problems. This thesis investigates the use of reinforcement learning to adapt the robot's verbal and non-verbal communication to the user's needs and preferences. Such non-functional adaptation of the robot's behaviors primarily aims to improve the user experience and the robot's perceived social intelligence. The literature has not yet provided a holistic view of the overall challenge: real-time adaptation requires control over the robot's multimodal behavior generation, an understanding of human feedback, and an algorithmic basis for machine learning. Thus, this thesis develops a conceptual framework for designing real-time non-functional social robot behavior adaptation with reinforcement learning. It provides a higher-level view from the system designer's perspective and guidance from the start to the end. It illustrates the process of modeling, simulating, and evaluating such adaptation processes. Specifically, it guides the integration of human feedback and social signals to equip the machine with social awareness. The conceptual framework is put into practice for several use cases, resulting in technical proofs of concept and research prototypes. They are evaluated in the lab and in in-situ studies. These approaches address typical activities in domestic environments, focussing on the robot's expression of personality, persona, politeness, and humor. Within this scope, the robot adapts its spoken utterances, prosody, and animations based on human explicit or implicit feedback.Soziale Roboter werden Teil unseres zukünftigen Zuhauses sein. Sie werden uns bei alltäglichen Aufgaben unterstützen, uns unterhalten und uns mit hilfreichen Ratschlägen versorgen. Noch gibt es allerdings technische Herausforderungen, die zunächst überwunden werden müssen, um die Maschine mit sozialen Kompetenzen auszustatten und zu einem sozial intelligenten und akzeptierten Mitbewohner zu machen. Eine wesentliche Fähigkeit eines jeden sozialen Roboters ist die verbale und nonverbale Kommunikation. Im Gegensatz zu Sprachassistenten, Smartphones und Smart-Home-Technologien, die bereits heute Teil des Lebens vieler Menschen sind, haben soziale Roboter eine Verkörperung, die Erwartungen an die Maschine weckt. Ihr anthropomorphes oder zoomorphes Aussehen legt nahe, dass sie in der Lage sind, auf natürliche Weise mit Sprache, Gestik oder Mimik zu kommunizieren, aber auch entsprechende menschliche Kommunikation zu verstehen. Darüber hinaus müssen Roboter auch die individuellen Vorlieben der Benutzer berücksichtigen. So ist jeder Mensch von seiner Kultur, sozialen Normen und eigenen Lebenserfahrungen geprägt, was zu unterschiedlichen Erwartungen an die Kommunikation mit einem Roboter führt. Roboter haben jedoch keine menschliche Intuition - sie müssen mit entsprechenden Algorithmen für diese Probleme ausgestattet werden. In dieser Arbeit wird der Einsatz von bestärkendem Lernen untersucht, um die verbale und nonverbale Kommunikation des Roboters an die Bedürfnisse und Vorlieben des Benutzers anzupassen. Eine solche nicht-funktionale Anpassung des Roboterverhaltens zielt in erster Linie darauf ab, das Benutzererlebnis und die wahrgenommene soziale Intelligenz des Roboters zu verbessern. Die Literatur bietet bisher keine ganzheitliche Sicht auf diese Herausforderung: Echtzeitanpassung erfordert die Kontrolle über die multimodale Verhaltenserzeugung des Roboters, ein Verständnis des menschlichen Feedbacks und eine algorithmische Basis für maschinelles Lernen. Daher wird in dieser Arbeit ein konzeptioneller Rahmen für die Gestaltung von nicht-funktionaler Anpassung der Kommunikation sozialer Roboter mit bestärkendem Lernen entwickelt. Er bietet eine übergeordnete Sichtweise aus der Perspektive des Systemdesigners und eine Anleitung vom Anfang bis zum Ende. Er veranschaulicht den Prozess der Modellierung, Simulation und Evaluierung solcher Anpassungsprozesse. Insbesondere wird auf die Integration von menschlichem Feedback und sozialen Signalen eingegangen, um die Maschine mit sozialem Bewusstsein auszustatten. Der konzeptionelle Rahmen wird für mehrere Anwendungsfälle in die Praxis umgesetzt, was zu technischen Konzeptnachweisen und Forschungsprototypen führt, die in Labor- und In-situ-Studien evaluiert werden. Diese Ansätze befassen sich mit typischen Aktivitäten in häuslichen Umgebungen, wobei der Schwerpunkt auf dem Ausdruck der Persönlichkeit, dem Persona, der Höflichkeit und dem Humor des Roboters liegt. In diesem Rahmen passt der Roboter seine Sprache, Prosodie, und Animationen auf Basis expliziten oder impliziten menschlichen Feedbacks an

C-EMO: A Modeling Framework for Collaborative Network Emotions

Recent research in the area of collaborative networks is focusing on the social and organizational complexity of collaboration environments as a way to prevent technological failures and consequently contribute for the collaborative network’s sustainability. One direction is moving towards the need to provide “human-tech” friendly systems with cognitive models of human factors such as stress, emotion, trust, leadership, expertise or decision-making ability. In this context, an emotion-based system is being proposed with this thesis in order to bring another approach to avoid collaboration network’s failures and help in the management of conflicts. This approach, which is expected to improve the performance of existing CNs, adopts some of the models developed in the human psychology, sociology and affective computing areas. The underlying idea is to “borrow” the concept of human-emotion and apply it into the context of CNs, giving the CN players the ability to “feel emotions”. Therefore, this thesis contributes with a modeling framework that conceptualizes the notion of “emotion” in CNs and a methodology approach based on system dynamics and agent-based techniques that estimates the CN player’s “emotional states” giving support to decision-making processes. Aiming at demonstrating the appropriateness of the proposed framework a simulation prototype was implemented and a validation approach was proposed consisting of simulation of scenarios, qualitative assessment and validation by research community peers.Recentemente a área de investigação das redes colaborativas tem vindo a debruçar-se na complexidade social e organizacional em ambientes colaborativos e como pode ser usada para prevenir falhas tecnológicas e consequentemente contribuir para redes colaborativas sustentáveis. Uma das direcções de estudo assenta na necessidade de fornecer sistemas amigáveis “humano-tecnológicos” com modelos cognitivos de factores humanos como o stress, emoção, confiança, liderança ou capacidade de tomada de decisão. É neste contexto que esta tese propõe um sistema baseado em emoções com o objectivo de oferecer outra aproximação para a gestão de conflitos e falhas da rede de colaboração. Esta abordagem, que pressupõe melhorar o desempenho das redes existentes, adopta alguns dos modelos desenvolvidos nas áreas da psicologia humana, sociologia e affective computing. A ideia que está subjacente é a de “pedir emprestado” o conceito de emoção humana e aplicá-lo no contexto das redes colaborativas, dando aos seus intervenientes a capacidade de “sentir emoções”. Assim, esta tese contribui com uma framework de modelação que conceptualiza a noção de “emoção” em redes colaborativas e com uma aproximação de metodologia sustentada em sistemas dinâmicos e baseada em agentes que estimam os “estados emocionais” dos participantes e da própria rede colaborativa. De forma a demonstrar o nível de adequabilidade da framework de modelação proposta, foi implementado um protótipo de simulação e foi proposta uma abordagem de validação consistindo em simulação de cenários, avaliação qualitativa e validação pelos pares da comunidade científica

Socially assistive robots : the specific case of the NAO

Numerous researches have studied the development of robotics, especially socially assistive robots (SAR), including the NAO robot. This small humanoid robot has a great potential in social assistance. The NAO robot’s features and capabilities, such as motricity, functionality, and affective capacities, have been studied in various contexts. The principal aim of this study is to gather every research that has been done using this robot to see how the NAO can be used and what could be its potential as a SAR. Articles using the NAO in any situation were found searching PSYCHINFO, Computer and Applied Sciences Complete and ACM Digital Library databases. The main inclusion criterion was that studies had to use the NAO robot. Studies comparing it with other robots or intervention programs were also included. Articles about technical improvements were excluded since they did not involve concrete utilisation of the NAO. Also, duplicates and articles with an important lack of information on sample were excluded. A total of 51 publications (1895 participants) were included in the review. Six categories were defined: social interactions, affectivity, intervention, assisted teaching, mild cognitive impairment/dementia, and autism/intellectual disability. A great majority of the findings are positive concerning the NAO robot. Its multimodality makes it a SAR with potential

Human-Computer Interaction

In this book the reader will find a collection of 31 papers presenting different facets of Human Computer Interaction, the result of research projects and experiments as well as new approaches to design user interfaces. The book is organized according to the following main topics in a sequential order: new interaction paradigms, multimodality, usability studies on several interaction mechanisms, human factors, universal design and development methodologies and tools

Design and semantics of form and movement (DeSForM 2006)

Design and Semantics of Form and Movement (DeSForM) grew from applied research exploring emerging design methods and practices to support new generation product and interface design. The products and interfaces are concerned with: the context of ubiquitous computing and ambient technologies and the need for greater empathy in the pre-programmed behaviour of the ‘machines’ that populate our lives. Such explorative research in the CfDR has been led by Young, supported by Kyffin, Visiting Professor from Philips Design and sponsored by Philips Design over a period of four years (research funding £87k). DeSForM1 was the first of a series of three conferences that enable the presentation and debate of international work within this field: • 1st European conference on Design and Semantics of Form and Movement (DeSForM1), Baltic, Gateshead, 2005, Feijs L., Kyffin S. & Young R.A. eds. • 2nd European conference on Design and Semantics of Form and Movement (DeSForM2), Evoluon, Eindhoven, 2006, Feijs L., Kyffin S. & Young R.A. eds. • 3rd European conference on Design and Semantics of Form and Movement (DeSForM3), New Design School Building, Newcastle, 2007, Feijs L., Kyffin S. & Young R.A. eds. Philips sponsorship of practice-based enquiry led to research by three teams of research students over three years and on-going sponsorship of research through the Northumbria University Design and Innovation Laboratory (nuDIL). Young has been invited on the steering panel of the UK Thinking Digital Conference concerning the latest developments in digital and media technologies. Informed by this research is the work of PhD student Yukie Nakano who examines new technologies in relation to eco-design textiles

Advances in Human-Robot Interaction

Rapid advances in the field of robotics have made it possible to use robots not just in industrial automation but also in entertainment, rehabilitation, and home service. Since robots will likely affect many aspects of human existence, fundamental questions of human-robot interaction must be formulated and, if at all possible, resolved. Some of these questions are addressed in this collection of papers by leading HRI researchers