Interactive Robot Learning of Gestures, Language and Affordances

A growing field in robotics and Artificial Intelligence (AI) research is human-robot collaboration, whose target is to enable effective teamwork between humans and robots. However, in many situations human teams are still superior to human-robot teams, primarily because human teams can easily agree on a common goal with language, and the individual members observe each other effectively, leveraging their shared motor repertoire and sensorimotor resources. This paper shows that for cognitive robots it is possible, and indeed fruitful, to combine knowledge acquired from interacting with elements of the environment (affordance exploration) with the probabilistic observation of another agent's actions. We propose a model that unites (i) learning robot affordances and word descriptions with (ii) statistical recognition of human gestures with vision sensors. We discuss theoretical motivations, possible implementations, and we show initial results which highlight that, after having acquired knowledge of its surrounding environment, a humanoid robot can generalize this knowledge to the case when it observes another agent (human partner) performing the same motor actions previously executed during training.Comment: code available at https://github.com/gsaponaro/glu-gesture

Action Observation for Neurorehabilitation in Apraxia

Neurorehabilitation and brain stimulation studies of post-stroke patients suggest that action-observation effects can lead to rapid improvements in the recovery of motor functions and long-term motor cortical reorganization. Apraxia is a clinically important disorder characterized by marked impairment in representing and performing skillful movements [gestures], which limits many daily activities and impedes independent functioning. Recent clinical research has revealed errors of visuo-motor integration in patients with apraxia. This paper presents a rehabilitative perspective focusing on the possibility of action observation as a therapeutic treatment for patients with apraxia. This perspective also outlines impacts on neurorehabilitation and brain repair following the reinforcement of the perceptual-motor coupling. To date, interventions based primarily on action observation in apraxia have not been undertaken

Recognizing Speech in a Novel Accent: The Motor Theory of Speech Perception Reframed

The motor theory of speech perception holds that we perceive the speech of another in terms of a motor representation of that speech. However, when we have learned to recognize a foreign accent, it seems plausible that recognition of a word rarely involves reconstruction of the speech gestures of the speaker rather than the listener. To better assess the motor theory and this observation, we proceed in three stages. Part 1 places the motor theory of speech perception in a larger framework based on our earlier models of the adaptive formation of mirror neurons for grasping, and for viewing extensions of that mirror system as part of a larger system for neuro-linguistic processing, augmented by the present consideration of recognizing speech in a novel accent. Part 2 then offers a novel computational model of how a listener comes to understand the speech of someone speaking the listener's native language with a foreign accent. The core tenet of the model is that the listener uses hypotheses about the word the speaker is currently uttering to update probabilities linking the sound produced by the speaker to phonemes in the native language repertoire of the listener. This, on average, improves the recognition of later words. This model is neutral regarding the nature of the representations it uses (motor vs. auditory). It serve as a reference point for the discussion in Part 3, which proposes a dual-stream neuro-linguistic architecture to revisits claims for and against the motor theory of speech perception and the relevance of mirror neurons, and extracts some implications for the reframing of the motor theory

Complementary Actions

Human beings come into the world wired for social interaction. At the fourteenth week of gestation, twin fetuses already display interactive movements specifically directed towards their co- twin. Readiness for social interaction is also clearly expressed by the newborn who imitate facial gestures, suggesting that there is a common representation mediating action observation and execution. While actions that are observed and those that are planned seem to be functionally equivalent, it is unclear if the visual representation of an observed action inevitably leads to its motor representation. This is particularly true with regard to complementary actions (from the Latin complementum ; i.e. that fills up), a specific class of movements which differ, while interacting, with observed ones. In geometry, angles are defined as complementary if they form a right angle. In art and design, complementary colors are color pairs that, when combined in the right proportions, produce white or black. As a working definition, complementary actions refer here to any form of social interaction wherein two (or more) individuals complete each other\u2019s actions in a balanced way. Successful complementary interactions are founded on the abilities:\ua0 (1)\ua0 to simulate another person\u2019s movements; (2)\ua0 to predict another person\u2019s future action/ s; (3)\ua0to produce an appropriate congruent/ incongruent response that completes the other person\u2019s action/ s; and (4)\ua0to integrate the predicted effects of one\u2019s own and another person\u2019s actions. It is the neurophysiological mechanism that underlies this process which forms the main theme of this chapte

Directional adposition use in English, Swedish and Finnish

Directional adpositions such as to the left of describe where a Figure is in relation to a Ground. English and Swedish directional adpositions refer to the location of a Figure in relation to a Ground, whether both are static or in motion. In contrast, the Finnish directional adpositions edellä (in front of) and jäljessä (behind) solely describe the location of a moving Figure in relation to a moving Ground (Nikanne, 2003). When using directional adpositions, a frame of reference must be assumed for interpreting the meaning of directional adpositions. For example, the meaning of to the left of in English can be based on a relative (speaker or listener based) reference frame or an intrinsic (object based) reference frame (Levinson, 1996). When a Figure and a Ground are both in motion, it is possible for a Figure to be described as being behind or in front of the Ground, even if neither have intrinsic features. As shown by Walker (in preparation), there are good reasons to assume that in the latter case a motion based reference frame is involved. This means that if Finnish speakers would use edellä (in front of) and jäljessä (behind) more frequently in situations where both the Figure and Ground are in motion, a difference in reference frame use between Finnish on one hand and English and Swedish on the other could be expected. We asked native English, Swedish and Finnish speakers’ to select adpositions from a language specific list to describe the location of a Figure relative to a Ground when both were shown to be moving on a computer screen. We were interested in any differences between Finnish, English and Swedish speakers. All languages showed a predominant use of directional spatial adpositions referring to the lexical concepts TO THE LEFT OF, TO THE RIGHT OF, ABOVE and BELOW. There were no differences between the languages in directional adpositions use or reference frame use, including reference frame use based on motion. We conclude that despite differences in the grammars of the languages involved, and potential differences in reference frame system use, the three languages investigated encode Figure location in relation to Ground location in a similar way when both are in motion. Levinson, S. C. (1996). Frames of reference and Molyneux’s question: Crosslingiuistic evidence. In P. Bloom, M.A. Peterson, L. Nadel & M.F. Garrett (Eds.) Language and Space (pp.109-170). Massachusetts: MIT Press. Nikanne, U. (2003). How Finnish postpositions see the axis system. In E. van der Zee & J. Slack (Eds.), Representing direction in language and space. Oxford, UK: Oxford University Press. Walker, C. (in preparation). Motion encoding in language, the use of spatial locatives in a motion context. Unpublished doctoral dissertation, University of Lincoln, Lincoln. United Kingdo

Embodied Gesture Processing: Motor-Based Integration of Perception and Action in Social Artificial Agents

A close coupling of perception and action processes is assumed to play an important role in basic capabilities of social interaction, such as guiding attention and observation of others’ behavior, coordinating the form and functions of behavior, or grounding the understanding of others’ behavior in one’s own experiences. In the attempt to endow artificial embodied agents with similar abilities, we present a probabilistic model for the integration of perception and generation of hand-arm gestures via a hierarchy of shared motor representations, allowing for combined bottom-up and top-down processing. Results from human-agent interactions are reported demonstrating the model’s performance in learning, observation, imitation, and generation of gestures

Extrinsic Methods for Coding and Dictionary Learning on Grassmann Manifolds

Sparsity-based representations have recently led to notable results in various visual recognition tasks. In a separate line of research, Riemannian manifolds have been shown useful for dealing with features and models that do not lie in Euclidean spaces. With the aim of building a bridge between the two realms, we address the problem of sparse coding and dictionary learning over the space of linear subspaces, which form Riemannian structures known as Grassmann manifolds. To this end, we propose to embed Grassmann manifolds into the space of symmetric matrices by an isometric mapping. This in turn enables us to extend two sparse coding schemes to Grassmann manifolds. Furthermore, we propose closed-form solutions for learning a Grassmann dictionary, atom by atom. Lastly, to handle non-linearity in data, we extend the proposed Grassmann sparse coding and dictionary learning algorithms through embedding into Hilbert spaces. Experiments on several classification tasks (gender recognition, gesture classification, scene analysis, face recognition, action recognition and dynamic texture classification) show that the proposed approaches achieve considerable improvements in discrimination accuracy, in comparison to state-of-the-art methods such as kernelized Affine Hull Method and graph-embedding Grassmann discriminant analysis.Comment: Appearing in International Journal of Computer Visio

Pass it on: towards a political economy of propensity

The paper argues that the work of Gabriel Tarde on imitation provides a fertile means of understanding how capitalism is forging a new affective technology which conforms to a logic of propensity rather than to means-end reasoning. This it does by drawing together a biological understanding of semiconscious cognition with various practical geometric arts so as to re-stage the world as a series of susceptible situations which can be ridden rather than rigidly controlled. The paper examines the advent of technologies which attend to the variable geometry of so-called animal spirits in the realm of business and then, using Tarde's work as a springboard, considers some alternative means of understanding imitative rays which have less instrumental undertones. The paper is an illustration of the way in which biology and culture have increasingly become intertwined