I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots

In the rapidly evolving landscape of human-computer interaction, the integration of vision capabilities into conversational agents stands as a crucial advancement. This paper presents an initial implementation of a dialogue manager that leverages the latest progress in Large Language Models (e.g., GPT-4, IDEFICS) to enhance the traditional text-based prompts with real-time visual input. LLMs are used to interpret both textual prompts and visual stimuli, creating a more contextually aware conversational agent. The system's prompt engineering, incorporating dialogue with summarisation of the images, ensures a balance between context preservation and computational efficiency. Six interactions with a Furhat robot powered by this system are reported, illustrating and discussing the results obtained. By implementing this vision-enabled dialogue system, the paper envisions a future where conversational agents seamlessly blend textual and visual modalities, enabling richer, more context-aware dialogues.Comment: 8 pages, 3 figure

Towards a Cognitive Architecture for Socially Adaptive Human-Robot Interaction

People have a natural predisposition to interact in an adaptive manner with others, by instinctively changing their actions, tones and speech according to the perceived needs of their peers. Moreover, we are not only capable of registering the affective and cognitive state of our partners, but over a prolonged period of interaction we also learn which behaviours are the most appropriate and well-suited for each one of them individually. This universal trait that we share regardless of our different personalities is referred to as social adaptation (adaptability). Humans are always capable of adapting to the others although our personalities may influence the speed and efficacy of the adaptation. This means that in our everyday lives we are accustomed to partake in complex and personalized interactions with our peers. Carrying this ability to personalize to human-robot interaction (HRI) is highly desirable since it would provide user-personalized interaction, a crucial element in many HRI scenarios - interactions with older adults, assistive or rehabilitative robotics, child-robot interaction (CRI), and many others. For a social robot to be able to recreate this same kind of rich, human-like interaction, it should be aware of our needs and affective states and be capable of continuously adapting its behaviour to them. Equipping a robot with these functionalities however is not a straightforward task. A robust approach for solving this is implementing a framework for the robot supporting social awareness and adaptation. In other words, the robot needs to be equipped with the basic cognitive functionalities, which would allow the robot to learn how to select the behaviours that would maximize the pleasantness of the interaction for its peers, while being guided by an internal motivation system that would provide autonomy to its decision-making process. The goal of this research was threefold: attempt to design a cognitive architecture supporting social HRI and implement it on a robotic platform; study how an adaptive framework of this kind would function when tested in HRI studies with users; and explore how including the element of adaptability and personalization in a cognitive framework would in reality affect the users - would it bring an additional richness to the human-robot interaction as hypothesized, or would it instead only add uncertainty and unpredictability that would not be accepted by the robot`s human peers? This thesis covers the work done on developing a cognitive framework for human-robot interaction; analyzes the various challenges of implementing the cognitive functionalities, porting the framework on several robotic platforms and testing potential validation scenarios; and finally presents the user studies performed with the robotic platforms of iCub and MiRo, focused on understanding how a cognitive framework behaves in a free-form HRI context and if humans can be aware and appreciate the adaptivity of the robot. In summary, this thesis had the task of approaching the complex field of cognitive HRI and attempt to shed some light on how cognition and adaptation develop from both the human and the robot side in an HRI scenario

Automation from lean perspective-potentials and challenges

The competitive climate of production and high labour cost, motivate western companies to use technologies like automation as a mean to increase manufacturing competitiveness. On the other hand companies are aware about cost reductive policies like lean production which has shown noticeable achievement, consequently some manufacturers tend to follow such system. In this situation, in order to have lean enterprise, it is vital to find a clear picture of challenges and potentials of implementing automation within a lean environment. So, finding the right level and type of automation becomes vital for companies, and achieving this is not possible without a lean development of automation. The paper presents an overview of automation development from a lean perspective. The focus is on manufacturing and a case study in the automotive industry is presented. Challenges and potentials of automation are pinpointed and some suggestions regarding automation development is given

Managing a Fleet of Autonomous Mobile Robots (AMR) using Cloud Robotics Platform

In this paper, we provide details of implementing a system for managing a fleet of autonomous mobile robots (AMR) operating in a factory or a warehouse premise. While the robots are themselves autonomous in its motion and obstacle avoidance capability, the target destination for each robot is provided by a global planner. The global planner and the ground vehicles (robots) constitute a multi agent system (MAS) which communicate with each other over a wireless network. Three different approaches are explored for implementation. The first two approaches make use of the distributed computing based Networked Robotics architecture and communication framework of Robot Operating System (ROS) itself while the third approach uses Rapyuta Cloud Robotics framework for this implementation. The comparative performance of these approaches are analyzed through simulation as well as real world experiment with actual robots. These analyses provide an in-depth understanding of the inner working of the Cloud Robotics Platform in contrast to the usual ROS framework. The insight gained through this exercise will be valuable for students as well as practicing engineers interested in implementing similar systems else where. In the process, we also identify few critical limitations of the current Rapyuta platform and provide suggestions to overcome them.Comment: 14 pages, 15 figures, journal pape

The CARESSES study protocol: testing and evaluating culturally competent socially assistive robots among older adults residing in long term care homes through a controlled experimental trial

Background : This article describes the design of an intervention study that focuses on whether and to what degree culturally competent social robots can improve health and well-being related outcomes among older adults residing long-term care homes. The trial forms the final stage of the international, multidisciplinary CARESSES project aimed at designing, developing and evaluating culturally competent robots that can assist older people according to the culture of the individual they are supporting. The importance of cultural competence has been demonstrated in previous nursing literature to be key towards improving health outcomes among patients. Method : This study employed a mixed-method, single-blind, parallel-group controlled before-and-after experimental trial design that took place in England and Japan. It aimed to recruit 45 residents of long-term care homes aged ≥65 years, possess sufficient cognitive and physical health and who self-identify with the English, Indian or Japanese culture (n = 15 each). Participants were allocated to either the experimental group, control group 1 or control group 2 (all n = 15). Those allocated to the experimental group or control group 1 received a Pepper robot programmed with the CARESSES culturally competent artificial intelligence (experimental group) or a limited version of this software (control group 1) for 18 h across 2 weeks. Participants in control group 2 did not receive a robot and continued to receive care as usual. Participants could also nominate their informal carer(s) to participate. Quantitative data collection occurred at baseline, after 1 week of use, and after 2 weeks of use with the latter time-point also including qualitative semi-structured interviews that explored their experience and perceptions further. Quantitative outcomes of interest included perceptions of robotic cultural competence, health-related quality of life, loneliness, user satisfaction, attitudes towards robots and caregiver burden. Discussion : This trial adds to the current preliminary and limited pool of evidence regarding the benefits of socially assistive robots for older adults which to date indicates considerable potential for improving outcomes. It is the first to assess whether and to what extent cultural competence carries importance in generating improvements to well-being