Multi-Modal Human-Machine Communication for Instructing Robot Grasping Tasks

A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One way of such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable to establish a common focus of attention and be able to use and integrate spoken instructions, visual perceptions, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and an modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects.Comment: 7 pages, 8 figure

A Robot Calligraphy System: From Simple to Complex Writing by Human Gestures

Robotic writing is a very challenging task and involves complicated kinematic control algorithms and image processing work. This paper, alternatively, proposes a robot calligraphy system that firstly applies human arm gestures to establish a font database of Chinese character elementary strokes and English letters, then uses the created database and human gestures to write Chinese characters and English words. A three-dimensional motion sensing input device is deployed to capture the human arm trajectories, which are used to build the font database and to train a classifier ensemble. 26 types of human gesture are used for writing English letters, and 5 types of gesture are used to generate 5 elementary strokes for writing Chinese characters. By using the font database, the robot calligraphy system acquires a basic writing ability to write simple strokes and letters. Then, the robot can develop to write complex Chinese characters and English words by following human body movements. The classifier ensemble, which is used to identify each gesture, is implemented through using feature selection techniques and the harmony search algorithm, thereby achieving better classification performance. The experimental evaluations are carried out to demonstrate the feasibility and performance of the proposed method. By following the motion trajectories of the human right arm, the end-effector of the robot can successfully write the English words or Chinese characters that correspond to the arm trajectories

A Posture Sequence Learning System for an Anthropomorphic Robotic Hand

The paper presents a cognitive architecture for posture learning of an anthropomorphic robotic hand. Our approach is aimed to allow the robotic system to perform complex perceptual operations, to interact with a human user and to integrate the perceptions by a cognitive representation of the scene and the observed actions. The anthropomorphic robotic hand imitates the gestures acquired by the vision system in order to learn meaningful movements, to build its knowledge by different conceptual spaces and to perform complex interaction with the human operator

Human-Machine Interface for Remote Training of Robot Tasks

Regardless of their industrial or research application, the streamlining of robot operations is limited by the proximity of experienced users to the actual hardware. Be it massive open online robotics courses, crowd-sourcing of robot task training, or remote research on massive robot farms for machine learning, the need to create an apt remote Human-Machine Interface is quite prevalent. The paper at hand proposes a novel solution to the programming/training of remote robots employing an intuitive and accurate user-interface which offers all the benefits of working with real robots without imposing delays and inefficiency. The system includes: a vision-based 3D hand detection and gesture recognition subsystem, a simulated digital twin of a robot as visual feedback, and the "remote" robot learning/executing trajectories using dynamic motion primitives. Our results indicate that the system is a promising solution to the problem of remote training of robot tasks.Comment: Accepted in IEEE International Conference on Imaging Systems and Techniques - IST201

Social Situatedness: Vygotsky and Beyond

The concept of ‘social situatedness’, i.e. the idea that the development of individual intelligence requires a social (and cultural) embedding, has recently received much attention in cognitive science and artificial intelligence research. The work of Lev Vygotsky who put forward this view already in the 1920s has influenced the discussion to some degree, but still remains far from well known. This paper therefore aims to give an overview of his cognitive development theory and discuss its relation to more recent work in primatology and socially situated artificial intelligence, in particular humanoid robotics

Developmental learning of internal models for robotics

Abstract: Robots that operate in human environments can learn motor skills asocially, from selfexploration, or socially, from imitating their peers. A robot capable of doing both can be more ~daptiveand autonomous. Learning by imitation, however, requires the ability to understand the actions ofothers in terms ofyour own motor system: this information can come from a robot's own exploration. This thesis investigates the minimal requirements for a robotic system than learns from both self-exploration and imitation of others. .Through self.exploration and computer vision techniques, a robot can develop forward 'models: internal mo'dels of its own motor system that enable it to predict the consequences of its actions. Multiple forward models are learnt that give the robot a distributed, causal representation of its motor system. It is demon~trated how a controlled increase in the complexity of these forward models speeds up the robot's learning. The robot can determine the uncertainty of its forward models, enabling it to explore so as to improve the accuracy of its???????predictions. Paying attention fO the forward models according to how their uncertainty is changing leads to a development in the robot's exploration: its interventions focus on increasingly difficult situations, adapting to the complexity of its motor system. A robot can invert forward models, creating inverse models, in order to estimate the actions that will achieve a desired goal. Switching to socialleaming. the robot uses these inverse model~ to imitate both a demonstrator's gestures and the underlying goals of their movement.Imperial Users onl