Towards a humanoid-oriented movement writing

This paper introduces humanoid-oriented movement writing (HOM), focusing on a notation in which body postures allow easy visual interpretation by both humans and humanoid robots. HOM Writing, derived from Sutton Movement Writing and Shorthand [1], is a natural modality for encoding the movements that humans perform during various work activities. Beyond its use as a record of human movement, the intent is to use the writing as a modality of communicating to robots what movements to execute. Humanoid robots could directly map the key postures represented in the notation to their own postures, imitating the postures captured in the description, and calculating intermediate postures by interpolation. A motion generator would ensure the motor control needed to create continuous movements. This paper focuses on the generation of the activity movement scripts, and addresses two modalities of producing the scripts: 1) ‘hand-coded’ by a human, using an editor, and 2) automatic extraction from video, e.g. from video-recordings of a human performing an activity. A software tool developed to allow easy script writing, the HOM Editor, is described and illustrated in hand-coding of a sequence of movements. The automatic generation of scripts from video is done using the pose estimation system by Yang and Ramanan [2], which takes an image and produces the joint coordinates of the limb parts; this is illustrated with a task of moving and arranging chairs. The posture extraction from a video provided by a single camera may often lead to occlusions of body parts during activities in which objects are manipulated. We show the advantages of using an additional camera, which significantly increases the correct posture estimation, and discuss how to further improve the automatic generation of scripts