HIGH QUALITY HUMAN 3D BODY MODELING, TRACKING AND APPLICATION

Geometric reconstruction of dynamic objects is a fundamental task of computer vision and graphics, and modeling human body of high fidelity is considered to be a core of this problem. Traditional human shape and motion capture techniques require an array of surrounding cameras or subjects wear reflective markers, resulting in a limitation of working space and portability. In this dissertation, a complete process is designed from geometric modeling detailed 3D human full body and capturing shape dynamics over time using a flexible setup to guiding clothes/person re-targeting with such data-driven models. As the mechanical movement of human body can be considered as an articulate motion, which is easy to guide the skin animation but has difficulties in the reverse process to find parameters from images without manual intervention, we present a novel parametric model, GMM-BlendSCAPE, jointly taking both linear skinning model and the prior art of BlendSCAPE (Blend Shape Completion and Animation for PEople) into consideration and develop a Gaussian Mixture Model (GMM) to infer both body shape and pose from incomplete observations. We show the increased accuracy of joints and skin surface estimation using our model compared to the skeleton based motion tracking. To model the detailed body, we start with capturing high-quality partial 3D scans by using a single-view commercial depth camera. Based on GMM-BlendSCAPE, we can then reconstruct multiple complete static models of large pose difference via our novel non-rigid registration algorithm. With vertex correspondences established, these models can be further converted into a personalized drivable template and used for robust pose tracking in a similar GMM framework. Moreover, we design a general purpose real-time non-rigid deformation algorithm to accelerate this registration. Last but not least, we demonstrate a novel virtual clothes try-on application based on our personalized model utilizing both image and depth cues to synthesize and re-target clothes for single-view videos of different people

Contribution à la planification de mouvement pour robots humanoïdes

cette thèse porte sur des algorithmes de contrôle et de planification de mouvements pour les robots humanoïdes. Le grand nombre de paramètres caractérisant ces systèmes a conduit au développement de méthodes numériques, d'abord appliquées aux bras manipulateurs et récemment adaptées pour les structures plus complexes. On relève particulièrement les formalismes de commande cinématique et dynamique par priorité qui permettent de produire un mouvement selon une hiérarchie préétablie des tâches. Au cours de ce travail, nous avons identifié le besoin d'étendre ce formalisme afin de tenir compte de contraintes unilatérales. Nous nous sommes par ailleurs intéressés à la planification de la locomotion en fonction des tâches. Nous proposons une modélisation jointe du robot et de sa trajectoire de marche comme une structure articulée unique saisissant à la fois les degrés de liberté actionnés (articulations motorisées du robot) et non actionnés (positionnement absolu dans l'espace). L'ensemble de ces algorithmes, qui seront longuement illustrés, ont été implémentés au sein du projet HPP (Humanoid Path Planner) et validés sur le robot humanoïde HRP-2.this thesis is related to motion control and planning algorithms for humanoid robots. For such highly-parameterized systems, numerical methods are well adapted and have thus been the enter of increasing attention in the recent years. Among the prominent numerical schemes, we recognized the prioritized inverse kinematics and dynamics frameworks to hold key features to plan motion for humanoid robots, such as the possibility to control the motion while enforcing a strict priority order among tasks. We have, however, identified a lack of support of strict priority enforcement when inequality constraints are to be accounted for in the numerical schemes and we were successful in proposing a solution to this shortcoming. We also considered the problem of planning bipedal locomotion according to any given tasks. We proposed to model this problem as an inverse kinematics problem, by considering the kinematic structure of the robot and its walk path as a single unified structure that captures both the degrees of freedom of the robot which are actuated (motorized joints) and those which are not (position and orientation in space). The presented algorithms, which will be abundantly illustrated, have been implemented within the HPP (Humanoid Path Planner) project and validated on the humanoid robot HRP-2

The Rocketbox Library and the Utility of Freely Available Rigged Avatars

As part of the open sourcing of the Microsoft Rocketbox avatar library for research and academic purposes, here we discuss the importance of rigged avatars for the Virtual and Augmented Reality (VR, AR) research community. Avatars, virtual representations of humans, are widely used in VR applications. Furthermore many research areas ranging from crowd simulation to neuroscience, psychology, or sociology have used avatars to investigate new theories or to demonstrate how they influence human performance and interactions. We divide this paper in two main parts: the first one gives an overview of the different methods available to create and animate avatars. We cover the current main alternatives for face and body animation as well introduce upcoming capture methods. The second part presents the scientific evidence of the utility of using rigged avatars for embodiment but also for applications such as crowd simulation and entertainment. All in all this paper attempts to convey why rigged avatars will be key to the future of VR and its wide adoption

Learning, Moving, And Predicting With Global Motion Representations

In order to effectively respond to and influence the world they inhabit, animals and other intelligent agents must understand and predict the state of the world and its dynamics. An agent that can characterize how the world moves is better equipped to engage it. Current methods of motion computation rely on local representations of motion (such as optical flow) or simple, rigid global representations (such as camera motion). These methods are useful, but they are difficult to estimate reliably and limited in their applicability to real-world settings, where agents frequently must reason about complex, highly nonrigid motion over long time horizons. In this dissertation, I present methods developed with the goal of building more flexible and powerful notions of motion needed by agents facing the challenges of a dynamic, nonrigid world. This work is organized around a view of motion as a global phenomenon that is not adequately addressed by local or low-level descriptions, but that is best understood when analyzed at the level of whole images and scenes. I develop methods to: (i) robustly estimate camera motion from noisy optical flow estimates by exploiting the global, statistical relationship between the optical flow field and camera motion under projective geometry; (ii) learn representations of visual motion directly from unlabeled image sequences using learning rules derived from a formulation of image transformation in terms of its group properties; (iii) predict future frames of a video by learning a joint representation of the instantaneous state of the visual world and its motion, using a view of motion as transformations of world state. I situate this work in the broader context of ongoing computational and biological investigations into the problem of estimating motion for intelligent perception and action

Form, function, mind: what doesn't compute (and what might)

The applicability of computational and dynamical systems models to organisms is scrutinized, using examples from developmental biology and cognition. Developmental morphogenesis is dependent on the inherent material properties of developing tissues, a non-computational modality, but cell differentiation, which utilizes chromatin-based revisable memory banks and program-like function-calling, via the developmental gene co-expression system unique to metazoans, has a quasi-computational basis. Multi-attractor dynamical models are argued to be misapplied to global properties of development, and it is suggested that along with computationalism, dynamicism is similarly unsuitable to accounting for cognitive phenomena. Proposals are made for treating brains and other nervous tissues as novel forms of excitable matter with inherent properties which enable the intensification of cell-based basal cognition capabilities present throughout the tree of life

Et Cetera

Et Cetera is woven together with five works that are essentially five bodies of writings as digital poetry -- a poetic practice that is made possible by digital media and technology in which aesthetic possibilities are extended through the semantic impact of data, alphabets, visuals, sound, etc. Interlaced by multimedial meaning-making, Et Cetera re(produces) installations that are engineered with algorithmic materials utilizing real-time data feeds, animated letterforms, performative instructions and sensory synthesis. Exploring different scenarios of human-machine coupling that consequently lead to multifarious illegibilities, Et Cetera amplifies the noise of information overflow in the concurrent mediascape with its rhizomatic networks largely beyond human conscious apprehension. On the B-side, Et Cetera is also involved with writing about the alphabetic writing apparatus, the role of artist as author as human-machine-centaur and networked subjectivity