Extracting Motion Primitives from Natural Handwriting Data

Institute for Adaptive and Neural ComputationHumans and animals can plan and execute movements much more adaptably and reliably than current computers can calculate robotic limb trajectories. Over recent decades, it has been suggested that our brains use motor primitives as blocks to build up movements. In broad terms a primitive is a segment of pre-optimised movement allowing a simplified movement planning solution. This thesis explores a generative model of handwriting based upon the concept of motor primitives. Unlike most primitive extraction studies, the primitives here are time extended blocks that are superimposed with character specific offsets to create a pen trajectory. This thesis shows how handwriting can be represented using a simple fixed function superposition model, where the variation in the handwriting arises from timing variation in the onset of the functions. Furthermore, it is shown how handwriting style variations could be due to primitive function differences between individuals, and how the timing code could provide a style invariant representation of the handwriting. The spike timing representation of the pen movements provides an extremely compact code, which could resemble internal spiking neural representations in the brain. The model proposes an novel way to infer primitives in data, and the proposed formalised probabilistic model allows informative priors to be introduced providing a more accurate inference of primitive shape and timing

Understanding of Object Manipulation Actions Using Human Multi-Modal Sensory Data

Object manipulation actions represent an important share of the Activities of Daily Living (ADLs). In this work, we study how to enable service robots to use human multi-modal data to understand object manipulation actions, and how they can recognize such actions when humans perform them during human-robot collaboration tasks. The multi-modal data in this study consists of videos, hand motion data, applied forces as represented by the pressure patterns on the hand, and measurements of the bending of the fingers, collected as human subjects performed manipulation actions. We investigate two different approaches. In the first one, we show that multi-modal signal (motion, finger bending and hand pressure) generated by the action can be decomposed into a set of primitives that can be seen as its building blocks. These primitives are used to define 24 multi-modal primitive features. The primitive features can in turn be used as an abstract representation of the multi-modal signal and employed for action recognition. In the latter approach, the visual features are extracted from the data using a pre-trained image classification deep convolutional neural network. The visual features are subsequently used to train the classifier. We also investigate whether adding data from other modalities produces a statistically significant improvement in the classifier performance. We show that both approaches produce a comparable performance. This implies that image-based methods can successfully recognize human actions during human-robot collaboration. On the other hand, in order to provide training data for the robot so it can learn how to perform object manipulation actions, multi-modal data provides a better alternative

Signature features with the visibility transformation

The signature in rough path theory provides a graduated summary of a path through an examination of the effects of its increments. Inspired by recent developments of signature features in the context of machine learning, we explore a transformation that is able to embed the effect of the absolute position of the data stream into signature features. This unified feature is particularly effective for its simplifying role in allowing the signature feature set to accommodate nonlinear functions of absolute and relative values

Extracting motion primitives from natural handwriting data

Abstract. For the past 10 years it has become clear that biological movement is made up of sub-routine type blocks, or motor primitives, with a central controller timing the activation of these blocks, creating synergies of muscle activation. This paper shows that it is possible to use a factorial hidden Markov model to infer primitives in handwriting data. These primitives are not predefined in terms of location of occurrence within the handwriting, and they are not limited or defined by a particular character set. Also, the variation in the data can to a large extent be explained by timing variation in the triggering of the primitives. Once an appropriate set of primitives has been inferred, the characters can be represented as a set of timings of primitive activations, along with variances, giving a very compact representation of the character. Separating the motor system into a motor primitive part, and a timing control gives us a possible insight into how we might create scribbles on paper.

Extracting motion primitives from natural handwriting data

Humans and animals can plan and execute movements much more adaptably and reliably than current computers can calculate robotic limb trajectories. Over recent decades, it has been suggested that our brains use motor primitives as blocks to build up movements. In broad terms a primitive is a segment of pre-optimised movement allowing a simplified movement planning solution. This thesis explores a generative model of handwriting based upon the concept of motor primitives. Unlike most primitive extraction studies, the primitives here are time extended blocks that are superimposed with character specific offsets to create a pen trajectory. This thesis shows how handwriting can be represented using a simple fixed function superposition model, where the variation in the handwriting arises from timing variation in the onset of the functions. Furthermore, it is shown how handwriting style variations could be due to primitive function differences between individuals, and how the timing code could provide a style invariant representation of the handwriting. The spike timing representation of the pen movements provides an extremely compact code, which could resemble internal spiking neural representations in the brain. The model proposes an novel way to infer primitives in data, and the proposed formalised probabilistic model allows informative priors to be introduced providing a more accurate inference of primitive shape and timing.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Intention Inference and Decision Making with Hierarchical Gaussian Process Dynamics Models

Anticipation is crucial for fluent human-robot interaction, which allows a robot to independently coordinate its actions with human beings in joint activities. An anticipatory robot relies on a predictive model of its human partners, and selects its own action according to the model's predictions. Intention inference and decision making are key elements towards such anticipatory robots. In this thesis, we present a machine-learning approach to intention inference and decision making, based on Hierarchical Gaussian Process Dynamics Models (H-GPDMs). We first introduce the H-GPDM, a class of generic latent-variable dynamics models. The H-GPDM represents the generative process of complex human movements that are directed by exogenous driving factors. Incorporating the exogenous variables in the dynamics model, the H-GPDM achieves improved interpretation, analysis, and prediction of human movements. While exact inference of the exogenous variables and the latent states is intractable, we introduce an approximate method using variational Bayesian inference, and demonstrate the merits of the H-GPDM in three different applications of human movement analysis. The H-GPDM lays a foundation for the following studies on intention inference and decision making. Intention inference is an essential step towards anticipatory robots. For this purpose, we consider a special case of the H-GPDM, the Intention-Driven Dynamics Model (IDDM), which considers the human partners' intention as exogenous driving factors. The IDDM is applicable to intention inference from observed movements using Bayes' theorem, where the latent state variables are marginalized out. As most robotics applications are subject to real-time constraints, we introduce an efficient online algorithm that allows for real-time intention inference. We show that the IDDM achieved state-of-the-art performance in intention inference using two human-robot interaction scenarios, i.e., target prediction for robot table tennis and action recognition for interactive robots. Decision making based on a time series of predictions allows a robot to be proactive in its action selection, which involves a trade-off between the accuracy and confidence of the prediction and the time for executing a selected action. To address the problem of action selection and optimal timing for initiating the movement, we formulate the anticipatory action selection using Partially Observable Markov Decision Process, where the H-GPDM is adopted to update belief state and to estimate transition model. We present two approaches to policy learning and decision making, and show their effectiveness using human-robot table tennis. In addition, we consider decision making solely based on the preference of the human partners, where observations are not sufficient for reliable intention inference. We formulate it as a repeated game and present a learning approach to safe strategies that exploit the humans' preferences. The learned strategy enables action selection when reliable intention inference is not available due to insufficient observation, e.g., for a robot to return served balls from a human table tennis player. In this thesis, we use human-robot table tennis as a running example, where a key bottleneck is the limited amount of time for executing a hitting movement. Movement initiation usually requires an early decision on the type of action, such as a forehand or backhand hitting movement, at least 80ms before the opponent has hit the ball. The robot, therefore, needs to be anticipatory and proactive of the opponent's intended target. Using the proposed methods, the robot can predict the intended target of the opponent and initiate an appropriate hitting movement according to the prediction. Experimental results show that the proposed intention inference and decision making methods can substantially enhance the capability of the robot table tennis player, using both a physically realistic simulation and a real Barrett WAM robot arm with seven degrees of freedom