Linear Bellman combination for control of character animation

Controllers are necessary for physically-based synthesis of character animation. However, creating controllers requires either manual tuning or expensive computer optimization. We introduce linear Bellman combination as a method for reusing existing controllers. Given a set of controllers for related tasks, this combination creates a controller that performs a new task. It naturally weights the contribution of each component controller by its relevance to the current state and goal of the system. We demonstrate that linear Bellman combination outperforms naive combination often succeeding where naive combination fails. Furthermore, this combination is provably optimal for a new task if the component controllers are also optimal for related tasks. We demonstrate the applicability of linear Bellman combination to interactive character control of stepping motions and acrobatic maneuvers.Singapore-MIT GAMBIT Game LabNational Science Foundation (U.S.) (Grant 2007043041)National Science Foundation (U.S.) (Grant CCF-0810888)Adobe SystemsPixar (Firm

Real Time Animation of Virtual Humans: A Trade-off Between Naturalness and Control

Virtual humans are employed in many interactive applications using 3D virtual environments, including (serious) games. The motion of such virtual humans should look realistic (or ‘natural’) and allow interaction with the surroundings and other (virtual) humans. Current animation techniques differ in the trade-off they offer between motion naturalness and the control that can be exerted over the motion. We show mechanisms to parametrize, combine (on different body parts) and concatenate motions generated by different animation techniques. We discuss several aspects of motion naturalness and show how it can be evaluated. We conclude by showing the promise of combinations of different animation paradigms to enhance both naturalness and control

Neural Categorical Priors for Physics-Based Character Control

Recent advances in learning reusable motion priors have demonstrated their effectiveness in generating naturalistic behaviors. In this paper, we propose a new learning framework in this paradigm for controlling physics-based characters with significantly improved motion quality and diversity over existing state-of-the-art methods. The proposed method uses reinforcement learning (RL) to initially track and imitate life-like movements from unstructured motion clips using the discrete information bottleneck, as adopted in the Vector Quantized Variational AutoEncoder (VQ-VAE). This structure compresses the most relevant information from the motion clips into a compact yet informative latent space, i.e., a discrete space over vector quantized codes. By sampling codes in the space from a trained categorical prior distribution, high-quality life-like behaviors can be generated, similar to the usage of VQ-VAE in computer vision. Although this prior distribution can be trained with the supervision of the encoder's output, it follows the original motion clip distribution in the dataset and could lead to imbalanced behaviors in our setting. To address the issue, we further propose a technique named prior shifting to adjust the prior distribution using curiosity-driven RL. The outcome distribution is demonstrated to offer sufficient behavioral diversity and significantly facilitates upper-level policy learning for downstream tasks. We conduct comprehensive experiments using humanoid characters on two challenging downstream tasks, sword-shield striking and two-player boxing game. Our results demonstrate that the proposed framework is capable of controlling the character to perform considerably high-quality movements in terms of behavioral strategies, diversity, and realism. Videos, codes, and data are available at https://tencent-roboticsx.github.io/NCP/

Learning Task-Agnostic Action Spaces for Movement Optimization

We propose a novel method for exploring the dynamics of physically based animated characters, and learning a task-agnostic action space that makes movement optimization easier. Like several previous papers, we parameterize actions as target states, and learn a short-horizon goal-conditioned low-level control policy that drives the agent's state towards the targets. Our novel contribution is that with our exploration data, we are able to learn the low-level policy in a generic manner and without any reference movement data. Trained once for each agent or simulation environment, the policy improves the efficiency of optimizing both trajectories and high-level policies across multiple tasks and optimization algorithms. We also contribute novel visualizations that show how using target states as actions makes optimized trajectories more robust to disturbances; this manifests as wider optima that are easy to find. Due to its simplicity and generality, our proposed approach should provide a building block that can improve a large variety of movement optimization methods and applications.Comment: Accepted as a regular paper by IEEE Transactions on Visualization and Computer Graphics (TVCG) in July 202

사람의 자연스러운 보행 동작 생성을 위한 물리 시뮬레이션 기반 휴머노이드 제어 방법

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2014. 8. 이제희.휴머노이드를 제어하여 사람의 자연스러운 이동 동작을 만들어내는 것은 컴퓨터그래픽스 및 로봇공학 분야에서 중요한 문제로 생각되어 왔다. 하지만, 이는 사람의 이동에서 구동기가 부족한 (underactuated) 특성과 사람의 몸의 복잡한 구조를 모방하고 시뮬레이션해야 한다는 점 때문에 매우 어려운 문제로 알려져왔다. 본 학위논문은 물리 시뮬레이션 기반 휴머노이드가 외부의 변화에 안정적으로 대응하고 실제 사람처럼 자연스럽고 다양한 이동 동작을 만들어내도록 하는 제어 방법을 제안한다. 우리는 실제 사람으로부터 얻을 수 있는 관찰 가능하고 측정 가능한 데이터를 최대한으로 활용하여 문제의 어려움을 극복했다. 우리의 접근 방법은 모션 캡처 시스템으로부터 획득한 사람의 모션 데이터를 활용하며, 실제 사람의 측정 가능한 물리적, 생리학적 특성을 복원하여 사용하는 것이다. 우리는 토크로 구동되는 이족 보행 모델이 다양한 스타일로 걸을 수 있도록 제어하는 데이터 기반 알고리즘을 제안한다. 우리의 알고리즘은 모션 캡처 데이터에 내재된 이동 동작 자체의 강건성을 활용하여 실제 사람과 같은 사실적인 이동 제어를 구현한다. 구체적으로는, 참조 모션 데이터를 재현하는 자연스러운 보행 시뮬레이션을 위한 관절 토크를 계산하게 된다. 알고리즘에서 가장 핵심적인 아이디어는 간단한 추종 제어기만으로도 참조 모션을 재현할 수 있도록 참조 모션을 연속적으로 조절하는 것이다. 우리의 방법은 모션 블렌딩, 모션 와핑, 모션 그래프와 같은 기존에 존재하는 데이터 기반 기법들을 이족 보행 제어에 활용할 수 있게 한다. 우리는 보다 사실적인 이동 동작을 생성하기 위해 사람의 몸을 세부적으로 모델링한, 근육에 의해 관절이 구동되는 인체 모델을 제어하는 이동 제어 시스템을 제안한다. 시뮬레이션에 사용되는 휴머노이드는 실제 사람의 몸에서 측정된 수치들에 기반하고 있으며 최대 120개의 근육을 가진다. 우리의 알고리즘은 최적의 근육 활성화 정도를 계산하여 시뮬레이션을 수행하며, 참조 모션을 충실히 재현하거나 혹은 새로운 상황에 맞게 모션을 적응시키기 위해 주어진 참조 모션을 수정하는 방식으로 동작한다. 우리의 확장가능한 알고리즘은 다양한 종류의 근골격 인체 모델을 최적의 근육 조합을 사용하며 균형을 유지하도록 제어할 수 있다. 우리는 다양한 스타일로 걷기 및 달리기, 모델의 변화 (근육의 약화, 경직, 관절의 탈구), 환경의 변화 (외력), 목적의 변화 (통증의 감소, 효율성의 최대화)에 대한 대응, 방향 전환, 회전, 인터랙티브하게 방향을 바꾸며 걷기 등과 같은 보다 난이도 높은 동작들로 이루어진 예제를 통해 우리의 접근 방법이 효율적임을 보였다.Controlling artificial humanoids to generate realistic human locomotion has been considered as an important problem in computer graphics and robotics. However, it has been known to be very difficult because of the underactuated characteristics of the locomotion dynamics and the complex human body structure to be imitated and simulated. In this thesis, we presents controllers for physically simulated humanoids that exhibit a rich set of human-like and resilient simulated locomotion. Our approach exploits observable and measurable data of a human to effectively overcome difficulties of the problem. More specifically, our approach utilizes observed human motion data collected by motion capture systems and reconstructs measured physical and physiological properties of a human body. We propose a data-driven algorithm to control torque-actuated biped models to walk in a wide range of locomotion skills. Our algorithm uses human motion capture data and realizes an human-like locomotion control facilitated by inherent robustness of the locomotion motion. Concretely, it takes reference motion and generates a set of joint torques to generate human-like walking simulation. The idea is continuously modulating the reference motion such that even a simple tracking controller can reproduce the reference motion. A number of existing data-driven techniques such as motion blending, motion warping, and motion graph can facilitate the biped control with this framework. We present a locomotion control system that controls detailed models of a human body with the musculotendon actuating process to create more human-like simulated locomotion. The simulated humanoids are based on measured properties of a human body and contain maximum 120 muscles. Our algorithm computes the optimal coordination of muscle activations and actively modulates the reference motion to fathifully reproduce the reference motion or adapt the motion to meet new conditions. Our scalable algorithm can control various types of musculoskeletal humanoids while seeking harmonious coordination of many muscles and maintaining balance. We demonstrate the strength of our approach with examples that allow simulated humanoids to walk and run in various styles, adapt to change of models (e.g., muscle weakness, tightness, joint dislocation), environments (e.g., external pushes), goals (e.g., pain reduction and efficiency maximization), and perform more challenging locomotion tasks such as turn, spin, and walking while steering its direction interactively.Contents Abstract i Contents iii List of Figures v 1 Introduction 1 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.1 Computer Graphics Perspective . . . . . . . . . . . . . . . . . 3 1.1.2 Robotics Perspective . . . . . . . . . . . . . . . . . . . . . . . 5 1.1.3 Biomechanics Perspective . . . . . . . . . . . . . . . . . . . . 7 1.2 Aim of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.4 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.5 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2 Previous Work 16 2.1 Biped Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.1.1 Controllers with Optimization . . . . . . . . . . . . . . . . . . 18 2.1.2 Controllers with Motion Capture Data . . . . . . . . . . . . . 20 2.2 Simulation of Musculoskeletal Humanoids . . . . . . . . . . . . . . . 21 2.2.1 Simulation of Specic Body Parts . . . . . . . . . . . . . . . . 21 2.2.2 Simulation of Full-Body Models . . . . . . . . . . . . . . . . . 22 2.2.3 Controllers for Musculoskeletal Humanoids . . . . . . . . . . . 23 3 Data-Driven Biped Control 24 3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.3 Data-Driven Control . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.3.1 Balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.3.2 Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4 Locomotion Control for Many-Muscle Humanoids 56 4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.2 Humanoid Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.2.1 Muscle Force Generation . . . . . . . . . . . . . . . . . . . . . 61 4.2.2 Muscle Force Transfer . . . . . . . . . . . . . . . . . . . . . . 64 4.2.3 Equation of Motion . . . . . . . . . . . . . . . . . . . . . . . . 66 4.3 Muscle Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.3.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.3.2 Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.3.3 Quadratic Programming Formulation . . . . . . . . . . . . . . 70 4.4 Trajectory Optimization . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 5 Conclusion 84 A Mathematical Definitions 88 A.1 Definitions of Transition Function . . . . . . . . . . . . . . . . . . . . 88 B Humanoid Models 89 B.1 Torque-Actuated Biped Models . . . . . . . . . . . . . . . . . . . . . 89 B.2 Many-Muscle Humanoid Models . . . . . . . . . . . . . . . . . . . . . 91 C Dynamics of Musculotendon Actuators 94 C.1 Contraction Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . 94 C.2 Initial Muscle States . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Glossary for Medical Terms 99 Bibliography 102 초록 113Docto

컴퓨터를 활용한 여러 사람의 동작 연출

학위논문 (박사)-- 서울대학교 대학원 공과대학 전기·컴퓨터공학부, 2017. 8. 이제희.Choreographing motion is the process of converting written stories or messages into the real movement of actors. In performances or movie, directors spend a consid-erable time and effort because it is the primary factor that audiences concentrate. If multiple actors exist in the scene, choreography becomes more challenging. The fundamental difficulty is that the coordination between actors should precisely be ad-justed. Spatio-temporal coordination is the first requirement that must be satisfied, and causality/mood are also another important coordinations. Directors use several assistant tools such as storyboards or roughly crafted 3D animations, which can visu-alize the flow of movements, to organize ideas or to explain them to actors. However, it is difficult to use the tools because artistry and considerable training effort are required. It also doesnt have ability to give any suggestions or feedbacks. Finally, the amount of manual labor increases exponentially as the number of actor increases. In this thesis, we propose computational approaches on choreographing multiple actor motion. The ultimate goal is to enable novice users easily to generate motions of multiple actors without substantial effort. We first show an approach to generate motions for shadow theatre, where actors should carefully collaborate to achieve the same goal. The results are comparable to ones that are made by professional ac-tors. In the next, we present an interactive animation system for pre-visualization, where users exploits an intuitive graphical interface for scene description. Given a de-scription, the system can generate motions for the characters in the scene that match the description. Finally, we propose two controller designs (combining regression with trajectory optimization, evolutionary deep reinforcement learning) for physically sim-ulated actors, which guarantee physical validity of the resultant motions.Chapter 1 Introduction 1 Chapter 2 Background 8 2.1 Motion Generation Technique 9 2.1.1 Motion Editing and Synthesis for Single-Character 9 2.1.2 Motion Editing and Synthesis for Multi-Character 9 2.1.3 Motion Planning 10 2.1.4 Motion Control by Reinforcement Learning 11 2.1.5 Pose/Motion Estimation from Incomplete Information 11 2.1.6 Diversity on Resultant Motions 12 2.2 Authoring System 12 2.2.1 System using High-level Input 12 2.2.2 User-interactive System 13 2.3 Shadow Theatre 14 2.3.1 Shadow Generation 14 2.3.2 Shadow for Artistic Purpose 14 2.3.3 Viewing Shadow Theatre as Collages/Mosaics of People 15 2.4 Physics-based Controller Design 15 2.4.1 Controllers for Various Characters 15 2.4.2 Trajectory Optimization 15 2.4.3 Sampling-based Optimization 16 2.4.4 Model-Based Controller Design 16 2.4.5 Direct Policy Learning 17 2.4.6 Deep Reinforcement Learning for Control 17 Chapter 3 Motion Generation for Shadow Theatre 19 3.1 Overview 19 3.2 Shadow Theatre Problem 21 3.2.1 Problem Definition 21 3.2.2 Approaches of Professional Actors 22 3.3 Discovery of Principal Poses 24 3.3.1 Optimization Formulation 24 3.3.2 Optimization Algorithm 27 3.4 Animating Principal Poses 29 3.4.1 Initial Configuration 29 3.4.2 Optimization for Motion Generation 30 3.5 Experimental Results 32 3.5.1 Implementation Details 33 3.5.2 Animation 34 3.5.3 3D Fabrication 34 3.6 Discussion 37 Chapter 4 Interactive Animation System for Pre-visualization 40 4.1 Overview 40 4.2 Graphical Scene Description 42 4.3 Candidate Scene Generation 45 4.3.1 Connecting Paths 47 4.3.2 Motion Cascade 47 4.3.3 Motion Selection For Each Cycle 49 4.3.4 Cycle Ordering 51 4.3.5 Generalized Paths and Cycles 52 4.3.6 Motion Editing 54 4.4 Scene Ranking 54 4.4.1 Ranking Criteria 54 4.4.2 Scene Ranking Measures 57 4.5 Scene Refinement 58 4.6 Experimental Results 62 4.7 Discussion 65 Chapter 5 Physics-based Design and Control 69 5.1 Overview 69 5.2 Combining Regression with Trajectory Optimization 70 5.2.1 Simulation and Motor Skills 71 5.2.2 Control Adaptation 75 5.2.3 Control Parameterization 79 5.2.4 Efficient Construction 81 5.2.5 Experimental Results 84 5.2.6 Discussion 89 5.3 Example-Guided Control by Deep Reinforcement Learning 91 5.3.1 System Overview 92 5.3.2 Initial Policy Construction 95 5.3.3 Evolutionary Deep Q-Learning 100 5.3.4 Experimental Results 107 5.3.5 Discussion 114 Chapter 6 Conclusion 119 6.1 Contribution 119 6.2 Future Work 120 요약 135Docto

Pre-computation for controlling character behavior in interactive physical simulations

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2010.Cataloged from PDF version of thesis.Includes bibliographical references (p. 129-136).The development of advanced computer animation tools has allowed talented artists to create digital actors, or characters, in films and commercials that move in a plausible and compelling way. In interactive applications, however, the artist does not have total control over the scenarios the character will experience. Unexpected changes in the environment of the character or unexpected interactions with dynamic elements of the virtual world can lead to implausible motions. This work investigates the use of physical simulation to automatically synthesize plausible character motions in interactive applications. We show how to simulate a realistic motion for a humanoid character by creating a feedback controller that tracks a motion capture recording. By applying the right forces at the right time, the controller is able to recover from a range of interesting changes to the environment and unexpected disturbances. Controlling physically simulated humanoid characters is non-trivial as they are governed by non-linear, non-smooth, and high-dimensional equations of motion. We simplify the problem by using a linearized and simplified dynamics model near a reference trajectory. Tracking a reference trajectory is an effective way of getting a character to perform a single task. However, simulated characters need to perform many tasks form a variety of possible configurations. This work also describes a method for combining existing controllers by adding their output forces to perform new tasks. This allows one to reuse existing controllers. A surprising fact is that combined controllers can perform optimally under certain conditions. These methods allow us to interactively simulate many interesting humanoid character behaviors in two and three dimensions. These characters have many more degrees of freedom than typical robot systems and move much more naturally. Simulation is fast enough that the controllers could soon be used to animate characters in interactive games. It is also possible that these simulations could be used to test robotic designs and biomechanical hypotheses.by Marco Jorge Tome da Silva.Ph.D

Animation basée sur la physique : extrapolation de mouvements humains plausibles et réalistes par optimisation incrémentale

L'objectif de nos travaux est de faire la synthèse de mouvements plausibles et réalistes de marche humaine dans divers environnements de synthèse. Bien que la solution proposée puisse également s'appliquer aux autres mouvements de locomotion humains ou animaux, nos travaux traitent uniquement du problème de la marche humaine. Afin de résoudre ce problème, nous avons développé une approche permettant de générer une multitude de variations d'une animation issue de capture de mouvement. Ces variations sont obtenues en adaptant le mouvement original à un environnement de synthèse dont les paramètres, tels que l'inclinaison du sol ou la courbure de la trajectoire, sont variés. Nous sommes donc en mesure de produire un mouvement de marche courbe ou de marche sur un plan incliné à partir d'un mouvement de marche en ligne droite sur un sol horizontal, ce que nous qualifions d'extrapolation de mouvement. Une animation initiale, obtenue par capture de mouvement, est essentielle à la solution proposée. Adapter ce mouvement à un nouvel environnement de synthèse consiste essentiellement à ajuster les caractéristiques globales du mouvement, telles que l'orientation du personnage et sa vitesse de déplacement. Ce faisant, nous sommes en mesure de conserver les détails plus fins du mouvement qui lui confèrent son aspect humain, tels que le mouvement des bras ou la vitesse avec laquelle un pied entre en contact avec le sol. En conservant les détails fins du mouvement d'origine, la solution proposée assure un certain réalisme dans les mouvements synthétisés. Dans la solution proposée, l'adaptation du mouvement initial est basée sur le paradigme des contraintes spatio-temporelles, où la synthèse du mouvement est posée comme un problème d'optimisation numérique. En plus d'être une formulation élégante du problème, ce paradigme est tout indiqué pour faire la synthèse de mouvements physiquement plausibles. En combinant ce paradigme avec l'utilisation d'une animation initiale issue de capture de mouvement, nous sommes en mesure de produire des animations de mouvements humains plausibles et réalistes. En pratique, le problème d'optimisation sous-tendu par l'adaptation d'un mouvement par contraintes spatio-temporelles est fortement non linéaire et opère dans un espace à très grande dimensionnalité. Cette complexité peut fortement ralentir le processus d'optimisation et aller jusqu'à en empêcher la convergence. La solution proposée fait donc appel à plusieurs mécanismes afin de réduire cette complexité. Notons qu'aucun de ces mécanismes ne vient compromettre la polyvalence de l'approche, en limitant la complexité du modèle biomécanique du personnage par exemple. Parmi ces mécanismes, deux sont des contributions originales : une technique d'estimation rapide des forces de réaction du sol et une approche d'optimisation incrémentale. Ces deux mécanismes visent à simplifier le processus d'optimisation en fournissant une solution initiale très proche de la solution optimale. La technique d'estimation des forces de réaction du sol sert à donner à ces paramètres une valeur initiale qui est relativement proche de leur valeur optimale, ce qui simplifie significativement la tâche d'optimisation subséquente. Cette technique consiste à trouver, pour les phases de support double, les forces de réaction du sol minimisant l'effort interne du personnage. Ce problème peut être exprimé comme une séquence de sous-problèmes de programmation quadratiques. Cette formulation est un aspect central de notre contribution et elle permet d'atteindre la solution très efficacement. L'approche d'optimisation incrémentale proposée s'inspire des méthodes de continuation. Le mouvement original est considéré comme une solution, un mouvement optimal, pour l'environnement de capture. L'environnement de synthèse est ensuite modifié graduellement, en augmentant l'inclinaison du sol par petits incréments par exemple. À chaque incrément, un nouveau mouvement optimal est trouvé en utilisant la solution de l'incrément précédent comme point de départ. On procède de la sorte jusqu'à l'obtention du mouvement désiré pour l'environnement de synthèse considéré. Si les incréments sont suffisamment petits, la différence entre deux problèmes d'optimisation consécutifs sera petite et il en sera de même pour leur optimum respectif