Gesture semantics reconstruction based on motion capturing and complex event processing

A fundamental problem in manual based gesture semantics reconstruction is the specification of preferred semantic concepts for gesture trajectories. This issue is complicated by problems human raters have annotating fast-paced three dimensional trajectories. Based on a detailed example of a gesticulated circular trajectory, we present a data-driven approach that covers parts of the semantic reconstruction by making use of motion capturing (mocap) technology. In our FA3ME framework we use a complex event processing approach to analyse and annotate multi-modal events. This framework provides grounds for a detailed description of how to get at the semantic concept of circularity observed in the data

Evaluation of human movement qualities: A methodology based on transferable-utility games on graphs.

Abstract A novel computational method for the analysis of expressive full-body movement qualities is introduced, which exploits concepts and tools from graph theory and game theory. The human skeletal structure is modeled as an undirected graph, where the joints are the vertices and the edge set contains both physical and nonphysical links. Physical links correspond to connections between adjacent physical body joints (e.g., the forearm, which connects the elbow to the wrist). Nonphysical links act as \u201cbridges\u201d between parts of the body not directly connected by the skeletal structure, but sharing very similar feature values. The edge weights depend on features obtained by using Motion Capture data. Then, a mathematical game is constructed over the graph structure, where the vertices represent the players and the edges represent communication channels between them. Hence, the body movement is modeled in terms of a game built on the graph structure. Since the vertices and the edges contribute to the overall quality of the movement, the adopted game-theoretical model is of cooperative nature. A game-theoretical concept, called Shapley value, is exploited as a centrality index to estimate the contribution of each vertex to a shared goal (e.g., to the way a particular movement quality is transferred among the vertices). The proposed method is applied to a data set of Motion Capture data of subjects performing expressive movements, recorded in the framework of the H2020-ICT-2015 EU Project WhoLoDance, Project no. 688865. Results are presented: development of novel method, contribution to the scientific community with a new data corpus, application the discussed method to 100 movement recordings and creation of database archive of stimuli for further use in research studies in the framework of the WhoLoDance Project

A human motion feature based on semi-supervised learning of GMM

Using motion capture to create naturally looking motion sequences for virtual character animation has become a standard procedure in the games and visual effects industry. With the fast growth of motion data, the task of automatically annotating new motions is gaining an importance. In this paper, we present a novel statistic feature to represent each motion according to the pre-labeled categories of key-poses. A probabilistic model is trained with semi-supervised learning of the Gaussian mixture model (GMM). Each pose in a given motion could then be described by a feature vector of a series of probabilities by GMM. A motion feature descriptor is proposed based on the statistics of all pose features. The experimental results and comparison with existing work show that our method performs more accurately and efficiently in motion retrieval and annotation

VCoach: A Customizable Visualization and Analysis System for Video-based Running Coaching

Videos are accessible media for analyzing sports postures and providing feedback to athletes. Existing video-based coaching systems often present feedback on the correctness of poses by augmenting videos with visual markers either manually by a coach or automatically by computing key parameters from poses. However, previewing and augmenting videos limit the analysis and visualization of human poses due to the fixed viewpoints, which confine the observation of captured human movements and cause ambiguity in the augmented feedback. Besides, existing sport-specific systems with embedded bespoke pose attributes can hardly generalize to new attributes; directly overlaying two poses might not clearly visualize the key differences that viewers would like to pursue. To address these issues, we analyze and visualize human pose data with customizable viewpoints and attributes in the context of common biomechanics of running poses, such as joint angles and step distances. Based on existing literature and a formative study, we have designed and implemented a system, VCoach, to provide feedback on running poses for amateurs. VCoach provides automatic low-level comparisons of the running poses between a novice and an expert, and visualizes the pose differences as part-based 3D animations on a human model. Meanwhile, it retains the users' controllability and customizability in high-level functionalities, such as navigating the viewpoint for previewing feedback and defining their own pose attributes through our interface. We conduct a user study to verify our design components and conduct expert interviews to evaluate the usefulness of the system

Human Motion Analysis Using Very Few Inertial Measurement Units

Realistic character animation and human motion analysis have become major topics of research. In this doctoral research work, three different aspects of human motion analysis and synthesis have been explored. Firstly, on the level of better management of tens of gigabytes of publicly available human motion capture data sets, a relational database approach has been proposed. We show that organizing motion capture data in a relational database provides several benefits such as centralized access to major freely available mocap data sets, fast search and retrieval of data, annotations based retrieval of contents, entertaining data from non-mocap sensor modalities etc. Moreover, the same idea is also proposed for managing quadruped motion capture data. Secondly, a new method of full body human motion reconstruction using very sparse configuration of sensors is proposed. In this setup, two sensor are attached to the upper extremities and one sensor is attached to the lower trunk. The lower trunk sensor is used to estimate ground contacts, which are later used in the reconstruction process along with the low dimensional inputs from the sensors attached to the upper extremities. The reconstruction results of the proposed method have been compared with the reconstruction results of the existing approaches and it has been observed that the proposed method generates lower average reconstruction errors. Thirdly, in the field of human motion analysis, a novel method of estimation of human soft biometrics such as gender, height, and age from the inertial data of a simple human walk is proposed. The proposed method extracts several features from the time and frequency domains for each individual step. A random forest classifier is fed with the extracted features in order to estimate the soft biometrics of a human. The results of classification have shown that it is possible with a higher accuracy to estimate the gender, height, and age of a human from the inertial data of a single step of his/her walk

The Role of Emotional and Facial Expression in Synthesised Sign Language Avatars

This thesis explores the role that underlying emotional facial expressions might have in regards to understandability in sign language avatars. Focusing specifically on Irish Sign Language (ISL), we examine the Deaf community’s requirement for a visual-gestural language as well as some linguistic attributes of ISL which we consider fundamental to this research. Unlike spoken language, visual-gestural languages such as ISL have no standard written representation. Given this, we compare current methods of written representation for signed languages as we consider: which, if any, is the most suitable transcription method for the medical receptionist dialogue corpus. A growing body of work is emerging from the field of sign language avatar synthesis. These works are now at a point where they can benefit greatly from introducing methods currently used in the field of humanoid animation and, more specifically, the application of morphs to represent facial expression. The hypothesis underpinning this research is: augmenting an existing avatar (eSIGN) with various combinations of the 7 widely accepted universal emotions identified by Ekman (1999) to deliver underlying facial expressions, will make that avatar more human-like. This research accepts as true that this is a factor in improving usability and understandability for ISL users. Using human evaluation methods (Huenerfauth, et al., 2008) the research compares an augmented set of avatar utterances against a baseline set with regards to 2 key areas: comprehension and naturalness of facial configuration. We outline our approach to the evaluation including our choice of ISL participants, interview environment, and evaluation methodology. Remarkably, the results of this manual evaluation show that there was very little difference between the comprehension scores of the baseline avatars and those augmented withEFEs. However, after comparing the comprehension results for the synthetic human avatar “Anna” against the caricature type avatar “Luna”, the synthetic human avatar Anna was the clear winner. The qualitative feedback allowed us an insight into why comprehension scores were not higher in each avatar and we feel that this feedback will be invaluable to the research community in the future development of sign language avatars. Other questions asked in the evaluation focused on sign language avatar technology in a more general manner. Significantly, participant feedback in regard to these questions indicates a rise in the level of literacy amongst Deaf adults as a result of mobile technology

Expressive movement generation with machine learning

Movement is an essential aspect of our lives. Not only do we move to interact with our physical environment, but we also express ourselves and communicate with others through our movements. In an increasingly computerized world where various technologies and devices surround us, our movements are essential parts of our interaction with and consumption of computational devices and artifacts. In this context, incorporating an understanding of our movements within the design of the technologies surrounding us can significantly improve our daily experiences. This need has given rise to the field of movement computing – developing computational models of movement that can perceive, manipulate, and generate movements. In this thesis, we contribute to the field of movement computing by building machine-learning-based solutions for automatic movement generation. In particular, we focus on using machine learning techniques and motion capture data to create controllable, generative movement models. We also contribute to the field by creating datasets, tools, and libraries that we have developed during our research. We start our research by reviewing the works on building automatic movement generation systems using machine learning techniques and motion capture data. Our review covers background topics such as high-level movement characterization, training data, features representation, machine learning models, and evaluation methods. Building on our literature review, we present WalkNet, an interactive agent walking movement controller based on neural networks. The expressivity of virtual, animated agents plays an essential role in their believability. Therefore, WalkNet integrates controlling the expressive qualities of movement with the goal-oriented behaviour of an animated virtual agent. It allows us to control the generation based on the valence and arousal levels of affect, the movement’s walking direction, and the mover’s movement signature in real-time. Following WalkNet, we look at controlling movement generation using more complex stimuli such as music represented by audio signals (i.e., non-symbolic music). Music-driven dance generation involves a highly non-linear mapping between temporally dense stimuli (i.e., the audio signal) and movements, which renders a more challenging modelling movement problem. To this end, we present GrooveNet, a real-time machine learning model for music-driven dance generation

Motion capture based motion analysis and motion synthesis for human-like character animation.

Motion capture technology is recognised as a standard tool in the computer animation pipeline. It provides detailed movement for animators; however, it also introduces problems and brings concerns for creating realistic and convincing motion for character animation. In this thesis, the post-processing techniques are investigated that result in realistic motion generation. Anumber of techniques are introduced that are able to improve the quality of generated motion from motion capture data, especially when integrating motion transitions from different motion clips. The presented motion data reconstruction technique is able to build convincing realistic transitions from existing motion database, and overcome the inconsistencies introduced by traditional motion blending techniques. It also provides a method for animators to re-use motion data more efficiently. Along with the development of motion data transition reconstruction, the motion capture data mapping technique was investigated for skeletal movement estimation. The per-frame based method provides animators with a real-time and accurate solution for a key post-processing technique. Although motion capture systems capture physically-based motion for character animation, no physical information is included in the motion capture data file. Using the knowledge of biomechanics and robotics, the relevant information for the captured performer are able to be abstracted and a mathematical-physical model are able to be constructed; such information is then applied for physics-based motion data correction whenever the motion data is edited

Motor control and strategy discovery for physically simulated characters

In physics-based character animation, motions are realized through control of simulated characters along with their interactions with the virtual environment. In this thesis, we study the problem of character control on two levels: joint-level motor control which transforms control signals to joint torques, and high-level motion control which outputs joint-level control signals given the current state of the character and the environment and the task objective. We propose a Modified Articulated-Body Algorithm (MABA) which achieves stable proportional-derivative (PD) low-level motor control with superior theoretical time complexity, practical efficiency and stability than prior implementations. We further propose a high-level motion control framework based on deep reinforcement learning (DRL) which enables the discovery of appropriate motion strategies without human demonstrations to complete a task objective. To facilitate the learning of realistic human motions, we propose a Pose Variational Autoencoder (P-VAE) to constrain the DRL actions to a subspace of natural poses. Our learning framework can be further combined with a sample-efficient Bayesian Diversity Search (BDS) algorithm and novel policy seeking to discover diverse strategies for tasks with multiple modes, such as various athletic jumping tasks

Dancing Media: The Contagious Movement of Posthuman Bodies (or Towards A Posthuman Theory of Dance)

My dissertation seeks to define a posthuman theory of dance through a historical study of the dancer as an instrument or technology for exploring emergent visual media, and by positioning screendance as an experimental technique for animating posthuman relation and thought. Commonly understood as ephemeral, dance is produced by assemblages that include bodies but are not limited to them. In this way, dance exceeds the human body. There is a central tension in the practice of dance, between the persistent presumption of the dancing body as a channel for human expression, and dance as a technicity of the body—a discipline and a practice of repeated gesture—that calls into question categories of the human. A posthuman theory of dance invites examination of such tensions and interrogates traditional notions of authenticity, ownership and commodification, as well as the bounded, individual subject who can assess the surrounding world with precise clarity, certain of where the human begins and ends. The guiding historical question for my dissertation is: if it is possible to describe both a modern form of posthuman dance (turn of the 19th-20th century), and a more recent form of posthuman dance (turn of the 20th-21st century), are they part of the same assemblage or are they constituted differently, and if so, how? Throughout my four chapters, I explore an array of case studies from early modernism to advanced capitalism, including Loie Fuller’s otherworldly stage dances; the scientific motion studies of Muybridge and Marey; Fritz Lang’s dancing maschinenmensch (or the first on-screen dancing machine) in the 1927 film Metropolis; the performances of singer-dancer hologram pop star, Hatsune Miku; and American engineering firm Boston Dynamics’ dancing military robots. The figure of the “dancing machine” (McCarren) is central to my project, especially given that dance has historically been used as a means of testing machines—from automata to robots to CGI images animated with MoCap—in their capacity to be lively or human-like. In each case, I am interested in how dance continues to be productive of some kind of subjectivity (or interiority, or “soul”), even in the absence of the human body, and how technique and gesture passes between bodies, both virtual and organic, dispersing agency often attributed to the human alone. I propose that a posthuman theory of dance is a necessary intervention to the broad and contradictory field of posthumanism because dance returns us to questions about bodies that are often suspiciously ignored in theories of posthumanism, especially regarding race (and historically racist categories of non/inhumanity), thereby exposing many of posthumanism’s biases, appropriations, dispossessions and erasures. Throughout my dissertation, I look to dance as both a concrete example and as a method of thinking through the potentials and limitations of posthumanism