A Sampling Approach to Generating Closely Interacting 3D Pose-pairs from 2D Annotations

We introduce a data-driven method to generate a large number of plausible, closely interacting 3D human pose-pairs, for a given motion category, e.g., wrestling or salsa dance. With much difficulty in acquiring close interactions using 3D sensors, our approach utilizes abundant existing video data which cover many human activities. Instead of treating the data generation problem as one of reconstruction, either through 3D acquisition or direct 2D-to-3D data lifting from video annotations, we present a solution based on Markov Chain Monte Carlo (MCMC) sampling. With a focus on efficient sampling over the space of close interactions, rather than pose spaces, we develop a novel representation called interaction coordinates (IC) to encode both poses and their interactions in an integrated manner. Plausibility of a 3D pose-pair is then defined based on the ICs and with respect to the annotated 2D pose-pairs from video. We show that our sampling-based approach is able to efficiently synthesize a large volume of plausible, closely interacting 3D pose-pairs which provide a good coverage of the input 2D pose-pairs

컴퓨터를 활용한 여러 사람의 동작 연출

학위논문 (박사)-- 서울대학교 대학원 공과대학 전기·컴퓨터공학부, 2017. 8. 이제희.Choreographing motion is the process of converting written stories or messages into the real movement of actors. In performances or movie, directors spend a consid-erable time and effort because it is the primary factor that audiences concentrate. If multiple actors exist in the scene, choreography becomes more challenging. The fundamental difficulty is that the coordination between actors should precisely be ad-justed. Spatio-temporal coordination is the first requirement that must be satisfied, and causality/mood are also another important coordinations. Directors use several assistant tools such as storyboards or roughly crafted 3D animations, which can visu-alize the flow of movements, to organize ideas or to explain them to actors. However, it is difficult to use the tools because artistry and considerable training effort are required. It also doesnt have ability to give any suggestions or feedbacks. Finally, the amount of manual labor increases exponentially as the number of actor increases. In this thesis, we propose computational approaches on choreographing multiple actor motion. The ultimate goal is to enable novice users easily to generate motions of multiple actors without substantial effort. We first show an approach to generate motions for shadow theatre, where actors should carefully collaborate to achieve the same goal. The results are comparable to ones that are made by professional ac-tors. In the next, we present an interactive animation system for pre-visualization, where users exploits an intuitive graphical interface for scene description. Given a de-scription, the system can generate motions for the characters in the scene that match the description. Finally, we propose two controller designs (combining regression with trajectory optimization, evolutionary deep reinforcement learning) for physically sim-ulated actors, which guarantee physical validity of the resultant motions.Chapter 1 Introduction 1 Chapter 2 Background 8 2.1 Motion Generation Technique 9 2.1.1 Motion Editing and Synthesis for Single-Character 9 2.1.2 Motion Editing and Synthesis for Multi-Character 9 2.1.3 Motion Planning 10 2.1.4 Motion Control by Reinforcement Learning 11 2.1.5 Pose/Motion Estimation from Incomplete Information 11 2.1.6 Diversity on Resultant Motions 12 2.2 Authoring System 12 2.2.1 System using High-level Input 12 2.2.2 User-interactive System 13 2.3 Shadow Theatre 14 2.3.1 Shadow Generation 14 2.3.2 Shadow for Artistic Purpose 14 2.3.3 Viewing Shadow Theatre as Collages/Mosaics of People 15 2.4 Physics-based Controller Design 15 2.4.1 Controllers for Various Characters 15 2.4.2 Trajectory Optimization 15 2.4.3 Sampling-based Optimization 16 2.4.4 Model-Based Controller Design 16 2.4.5 Direct Policy Learning 17 2.4.6 Deep Reinforcement Learning for Control 17 Chapter 3 Motion Generation for Shadow Theatre 19 3.1 Overview 19 3.2 Shadow Theatre Problem 21 3.2.1 Problem Definition 21 3.2.2 Approaches of Professional Actors 22 3.3 Discovery of Principal Poses 24 3.3.1 Optimization Formulation 24 3.3.2 Optimization Algorithm 27 3.4 Animating Principal Poses 29 3.4.1 Initial Configuration 29 3.4.2 Optimization for Motion Generation 30 3.5 Experimental Results 32 3.5.1 Implementation Details 33 3.5.2 Animation 34 3.5.3 3D Fabrication 34 3.6 Discussion 37 Chapter 4 Interactive Animation System for Pre-visualization 40 4.1 Overview 40 4.2 Graphical Scene Description 42 4.3 Candidate Scene Generation 45 4.3.1 Connecting Paths 47 4.3.2 Motion Cascade 47 4.3.3 Motion Selection For Each Cycle 49 4.3.4 Cycle Ordering 51 4.3.5 Generalized Paths and Cycles 52 4.3.6 Motion Editing 54 4.4 Scene Ranking 54 4.4.1 Ranking Criteria 54 4.4.2 Scene Ranking Measures 57 4.5 Scene Refinement 58 4.6 Experimental Results 62 4.7 Discussion 65 Chapter 5 Physics-based Design and Control 69 5.1 Overview 69 5.2 Combining Regression with Trajectory Optimization 70 5.2.1 Simulation and Motor Skills 71 5.2.2 Control Adaptation 75 5.2.3 Control Parameterization 79 5.2.4 Efficient Construction 81 5.2.5 Experimental Results 84 5.2.6 Discussion 89 5.3 Example-Guided Control by Deep Reinforcement Learning 91 5.3.1 System Overview 92 5.3.2 Initial Policy Construction 95 5.3.3 Evolutionary Deep Q-Learning 100 5.3.4 Experimental Results 107 5.3.5 Discussion 114 Chapter 6 Conclusion 119 6.1 Contribution 119 6.2 Future Work 120 요약 135Docto

사람 동작 생성을 위한 의미 분석

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2017. 2. 이제희.One of main goals of computer-generated character animation is to reduce cost to create animated scenes. Using human motion in makes it easier to animate characters, so motion capture technology is used as a standard technique. However, it is difficult to get the desired motion because it requires a large space, high-performance cameras, actors, and a significant amount of work for post-processing. Data-driven character animation includes a set of techniques that make effective use of captured motion data. In this thesis, I introduce methods that analyze the semantics of motion data to enhance the utilization of the data. To accomplish this, various techniques in other fields are integrated so that we can understand the semantics of a unit motion clip, the implicit structure of a motion sequence, and a natural description of movements. Based upon that understanding, we can generate new animation systems. The first animation system in this thesis allows the user to generate an animation of basketball play from the tactics board. In order to handle complex basketball rule that players must follow, we use context-free grammars for motion representation. Our motion grammar enables the user to define implicit/explicit rules of human behavior and generates valid movement of basketball players. Interactions between players or between players and the environment are represented with semantic rules, which results in plausible animation. When we compose motion sequences, we rely on motion corpus storing the prepared motion clips and the transition between them. It is important to construct good motion corpus to create natural and rich animations, but it requires the efforts of experts. We introduce a semi-supervised learning technique for automatic generation of motion corpus. Stacked autoencoders are used to find latent features for large amounts of motion capture data and the features are used to effectively discover worthwhile motion clips. The other animation system uses natural language processing technology to understand the meaning of the animated scene that the user wants to make. Specifically, the script of an animated scene is used to synthesize the movements of characters. Like the sketch interface, scripts are very sparse input sources. Understanding motion allows the system to interpret abstract user input and generate scenes that meet user needs.1 Introduction 1 2 Background 8 2.1 RepresentationofHumanMovements 8 2.2 MotionAnnotation 11 2.3 MotionGrammars 12 2.4 NaturalLanguageProcessing 15 3 Motion Grammar 17 3.1 Overview 18 3.2 MotionGrammar 20 3.2.1 Instantiation, Semantics, and Plausibility 22 3.2.2 ASimpleExample 25 3.3 BasketballTacticsBoard 27 3.4 MotionSynthesis 29 3.5 Results 35 3.6 Discussion 39 4 Motion Embedding 49 4.1 Overview 50 4.2 MotionData 51 4.3 Autoencoders 52 4.3.1 Stackedautoencoders 53 4.4 MotionCorpus 53 4.4.1 Training 53 4.4.2 FindingMotionClips 55 4.5 Results 55 4.6 Discussion 57 5 Text to Animation 62 5.1 Overview 63 5.2 UnderstandingSemantics 64 5.3 ActionChains 65 5.3.1 WordEmbedding 66 5.3.2 MotionPlausibility 67 5.4 SceneGeneration 69 5.5 Results 70 5.6 Discussion 70 6 Conclusion 74 Bibliography 76 초록 100Docto

Menge: A Modular Framework for Simulating Crowd Movement

We present Menge, a cross-platform, extensible, modular framework for simulating pedestrian movement in a crowd.  Menge's architecture is inspired by an implicit decomposition of the problem of simulating crowds into component subproblems.  These subproblems can typically be solved in many ways; different combinations of subproblem solutions yield crowd simulators with likewise varying properties.  Menge creates abstractions for those subproblems and provides a plug-in architecture so that a novel simulator can be dynamically configured by connecting built-in and bespoke implementations of solutions to the various subproblems.  Use of this type of framework could facilitate crowd simulation research, evaluation, and applications by reducing the cost of entering the domain, facilitating collaboration, and making comparisons between algorithms simpler.  We show how the Menge framework is compatible with many prior models and algorithms used in crowd simulation and illustrate its flexibility via a varied set of scenarios and applications

적은 수의 사용자 입력으로부터 인간 동작의 합성 및 편집

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2014. 8. 이제희.An ideal 3D character animation system can easily synthesize and edit human motion and also will provide an efficient user interface for an animator. However, despite advancements of animation systems, building effective systems for synthesizing and editing realistic human motion still remains a difficult problem. In the case of a single character, the human body is a significantly complex structure because it consists of as many as hundreds of degrees of freedom. An animator should manually adjust many joints of the human body from user inputs. In a crowd scene, many individuals in a human crowd have to respond to user inputs when an animator wants a given crowd to fit a new environment. The main goal of this thesis is to improve interactions between a user and an animation system. As 3D character animation systems are usually driven by low-dimensional inputs, there is no method for a user to directly generate a high-dimensional character animation. To address this problem, we propose a data-driven mapping model that is built by motion data obtained from a full-body motion capture system, crowd simulation, and data-driven motion synthesis algorithm. With the data-driven mapping model in hand, we can transform low-dimensional user inputs into character animation because motion data help to infer missing parts of system inputs. As motion capture data have many details and provide realism of the movement of a human, it is easier to generate a realistic character animation than without motion capture data. To demonstrate the generality and strengths of our approach, we developed two animation systems that allow the user to synthesize a single character animation in realtime and edit crowd animation via low-dimensional user inputs interactively. The first system entails controlling a virtual avatar using a small set of three-dimensional (3D) motion sensors. The second system manipulates large-scale crowd animation that consists of hundreds of characters with a small number of user constraints. Examples show that our system is much less laborious and time-consuming than previous animation systems, and thus is much more suitable for a user to generate desired character animation.Contents Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . II Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IV List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V 1 Introduction 1 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2 Background 10 2.1 Performance Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1.1 Performance-based Interfaces for Character Animation . . . . . . . 11 2.1.2 Statistical Models for Motion Synthesis . . . . . . . . . . . . . . . 12 2.1.3 Retrieval of Motion Capture Data . . . . . . . . . . . . . . . . . . 13 2.2 Crowd Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2.1 Crowd Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2.2 Motion Editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2.3 Geometry Deformation . . . . . . . . . . . . . . . . . . . . . . . . 15 3 Realtime Performance Animation Using Sparse 3D Motion Sensors 17 3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.2 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.3 Sensor Data and Calibration . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.4 Motion Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.4.1 Online Local Model . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.4.2 Kernel CCA-based Regression . . . . . . . . . . . . . . . . . . . . 25 3.4.3 Motion Post-processing . . . . . . . . . . . . . . . . . . . . . . . 27 3.5 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4 Interactive Manipulation of Large-Scale Crowd Animation 40 4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 4.2 Crowd Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.3 Cage-based Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.3.1 Cage Construction . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.3.2 Cage Representation . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.4 Editing Crowd Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.4.1 Spatial Manipulation . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.4.2 Temporal Manipulation . . . . . . . . . . . . . . . . . . . . . . . . 57 4.5 Collision Avoidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 4.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5 Conclusion 69 Bibliography I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XIIIDocto

Conception et animation interactive de foules pour de vastes environnements

Crowds are increasingly present in audio-visual media, such as movies or video games. They help to strengthen the immersion of the subject in the virtual environment. However, creating crowds is most of the time based on models hard to master and which do not offer a direct control on the motion that you want to create. In this thesis we propose contributions for designing crowd motions through interactive and intuitive tools. Firstly, we present an interactive method for designing the crowds by distorting it like clay. The user can stretch, compress and twist the overall shape of the crowd to give it the shape he or she wishes. The inner characters of the crowd automatically adapt to the new shape imposed by the user. Secondly, we present a method to paint the motion and the density of the crowd to create it. We offer the opportunity to the user to create crowds by painting a grayscale density map and a motion map by gradients. Its colored maps are transformed by our system to crowds, thanks to our iterative algorithm seeking to optimize the different values of colored maps. Crowds obtained by these methods can occupy a very large space and are animated indefinitely. Unlike conventional methods of creating crowds, that are based on the adjustment of model parameters, our methods allow to design crowd motions based on higher level features of the crowd, as its overall shape, its internal movement or density. This offers the possibility to simply, quickly and intuitively create animated crowd contents.Les foules sont de plus en plus présentes dans les médias grands publics, comme le cinéma ou les jeux vidéo. Elles permettent de renforcer l'immersion du sujet dans l'environnement qui lui est présenté. Or, la création de mouvement de foule est la plus part du temps basé sur des modèles dures à prendre en main et qui n'offrent pas un contrôle direct sur le mouvement de foule que l'on souhaite créer. Dans cette thèse nous proposons des contributions sous forme de méthodes pour concevoir des mouvements de foules par le biais d'outils interactifs et intuitifs. Dans un premier temps, nous présentons une méthode interactive permettant de concevoir des foules en les déformant comme de l'argile. L'utilisateur peut tirer, compresser et torde la forme global de des foules pour leurs donner la forme qu'il ou elle souhaite. Les personnages qui composent la foule s'adaptent automatiquement à la nouvelle forme imposée par l'utilisateur. Dans un second temps, nous présentons une méthode permettant de peindre les mouvements et la densité de la foule pour la créer. Nous offrons la possibilité à l'utilisateur de créer des foules en peignant une carte de densité en niveau de gris, et une carte de mouvement via des dégradés. Ses cartes de couleurs sont utilisées par notre système pour le transformer en un mouvement de foule, via un algorithme itératif cherchant à optimiser les différentes valeurs des cartes de couleurs. Les foules obtenues par ces méthodes peuvent occupées un espace très large, et sont animées indéfiniment. Contrairement aux méthodes classiques de création de foules qui se basent sur l'ajustement de paramètres de modèles, nos méthodes permettent de concevoir les mouvements de foules en se basant sur des caractéristiques plus hauts niveaux de la foule, comme sa forme globale, ses mouvements internes ou sa densité. Ce qui offre la possibilité de créer du contenu de foule animée de manière simple, rapide et intuitif

Planning Plausible Human Motions for Navigation and Collision Avoidance

This thesis investigates the plausibility of computer-generated human motions for navigation and collision avoidance. To navigate a human character through obstacles in an virtual environment, the problem is often tackled by finding the shortest possible path to the destination with smoothest motions available. This is because such solution is regarded as cost-effective and free-flowing in that it implicitly minimises the biomechanical efforts and potentially precludes anomalies such as frequent and sudden change of behaviours, and hence more plausible to human eyes. Previous research addresses this problem in two stages: finding the shortest collision-free path (motion planning) and then fitting motions onto this path accordingly (motion synthesis). This conventional approach is not optimal because the decoupling of these two stages introduces two problems. First, it forces the motion-planning stage to deliberately simplify the collision model to avoid obstacles. Secondly, it over-constrains the motion-synthesis stage to approximate motions to a sub-optimal trajectory. This often results in implausible animations that travel along erratic long paths while making frequent and sudden behaviour changes. In this research, I argue that to provide more plausible navigation and collision avoidance animation, close-proximity interaction with obstacles is crucial. To address this, I propose to combine motion planning and motion synthesis to search for shorter and smoother solutions. The intuition is that by incorporating precise collision detection and avoidance with motion capture database queries, we will be able to plan fine-scale interactions between obstacles and moving crowds. The results demonstrate that my approach can discover shorter paths with steadier behaviour transitions in scene navigation and crowd avoidance. In addition, this thesis attempts to propose a set of metrics that can be used to evaluate the plausibility of computer-generated navigation animations

Interactive control of multi-agent motion in virtual environments

With the increased use of crowd simulation in animation, specification of crowd motion can be very time consuming, requiring a lot of user input. To alleviate this cost, we wish to allow a user to interactively manipulate the many degrees of freedom in a crowd, whilst accounting for the limitation of low-dimensional signals from standard input devices. In this thesis we present two approaches for achieving this: 1) Combining shape deformation methods with a multitouch input device, allowing a user to control the motion of the crowd in dynamic environments, and 2) applying a data-driven approach to learn the mapping between a crowd’s motion and the corresponding user input to enable intuitive control of a crowd. In our first approach, we represent the crowd as a deformable mesh, allowing a user to manipulate it using a multitouch device. The user controls the shape and motion of the crowd by altering the mesh, and the mesh in turn deforms according to the environment. We handle congestion and perturbation by having agents dynamically reassign their goals in the formation using a mass transport solver. Our method allows control of a crowd in a single pass, improving on the time taken by previous, multistage, approaches. We validate our method with a user study, comparing our control algorithm against a common mouse-based controller. We develop a simplified version of motion data patches to model character-environment interactions that are largely ignored in previous crowd research. We design an environment-aware cost metric for the mass transport solver that considers how these interactions affect a character’s ability to track the user’s commands. Experimental results show that our system can produce realistic crowd scenes with minimal, high-level, input signals from the user. In our second approach, we propose that crowd simulation control algorithms inherently impose restrictions on how user input affects the motion of the crowd. To bypass this, we investigate a data-driven approach for creating a direct mapping between low-dimensional user input and the resulting high-dimensional crowd motion. Results show that the crowd motion can be inferred directly from variations in a user’s input signals, providing a user with greater freedom to define the animation