Deep Manifold Traversal: Changing Labels with Convolutional Features

Many tasks in computer vision can be cast as a "label changing" problem, where the goal is to make a semantic change to the appearance of an image or some subject in an image in order to alter the class membership. Although successful task-specific methods have been developed for some label changing applications, to date no general purpose method exists. Motivated by this we propose deep manifold traversal, a method that addresses the problem in its most general form: it first approximates the manifold of natural images then morphs a test image along a traversal path away from a source class and towards a target class while staying near the manifold throughout. The resulting algorithm is surprisingly effective and versatile. It is completely data driven, requiring only an example set of images from the desired source and target domains. We demonstrate deep manifold traversal on highly diverse label changing tasks: changing an individual's appearance (age and hair color), changing the season of an outdoor image, and transforming a city skyline towards nighttime

Semi-Supervised Facial Animation Retargeting

This paper presents a system for facial animation retargeting that al- lows learning a high-quality mapping between motion capture data and arbitrary target characters. We address one of the main chal- lenges of existing example-based retargeting methods, the need for a large number of accurate training examples to define the corre- spondence between source and target expression spaces. We show that this number can be significantly reduced by leveraging the in- formation contained in unlabeled data, i.e. facial expressions in the source or target space without corresponding poses. In contrast to labeled samples that require time-consuming and error-prone manual character posing, unlabeled samples are easily obtained as frames of motion capture recordings or existing animations of the target character. Our system exploits this information by learning a shared latent space between motion capture and character param- eters in a semi-supervised manner. We show that this approach is resilient to noisy input and missing data and significantly improves retargeting accuracy. To demonstrate its applicability, we integrate our algorithm in a performance-driven facial animation system

Amélioration de la capture des visages pour l'industrie du jeu vidéo

Réaliser une animation réaliste d’un visage dans l’industrie du jeu vidéo est un défi technique. Il est extrêmement complexe, y compris pour les meilleurs animateurs 3D, de produire une animation faciale réaliste. Une approche à ce problème est de capturer la performance d’un acteur et de la reproduire automatiquement sur un modèle de visage paramétrable, communément appelé un «rig». Le processus reste toutefois toujours peu automatisé et repose énormément sur les artistes qui l’effectuent. La question est alors de savoir comment arriver à trouver les paramètres d’animations sur un rig de visage en 3D à partir d’une source vidéo 2D mono-caméra ? Nous proposons de ramener le problème dans un contexte physique : déterminer ces paramètres d’animation sans intervention humaine à partir de la session de capture de mouvement. L’objectif est alors de proposer aux artistes animateurs un nouvel outil qui leur permettra d’obtenir des résultats semblables à ceux du processus précédent, d’une façon plus rapide sans pour autant changer leurs habitudes de travail. Notre nouveau pipeline consiste en un transfert de déformation entre la performance de l’acteur retranscrite dans un suivi de points en 3D vers les mêmes points sur le Rig visé, afin de réaliser une cible atteignable. L’artiste n’a pas besoin de fournir une pose neutre au système, elle est déterminée sans intervention humaine en projetant la pose neutre du Rig dans l’espace du suivi. Les paramètres d’animation optimaux sont ensuite déterminés itérativement en minimisant une énergie des moindres carrés entre la cible et le rig à l’aide d’une descente de gradient dans l’espace des paramètres d’animation. Afin de ne pas tomber dans un minimum local, un réalignement au cours du processus itératif est réalisé par zones du visage entre le rig et la cible. Si l’artiste n’est pas satisfait du résultat alors produit, il peut le corriger à l’aide des mêmes outils qu’il a l’habitude d’utiliser, que nous avons adaptés à la correction d’erreurs. En utilisant notre nouveau pipeline, un artiste peut désormais réaliser une animation faciale réaliste en une demi-heure contre presque dix heures pour le même résultat avec le pipeline précédent. L’artiste n’a en effet plus qu’à se concentrer sur la correction des résultats insatisfaisants du nouveau pipeline plutôt que de réaliser l’animation entière du début.----------ABSTRACT: Performing a realistic 3D facial animation in the video game industry is a real technical challenge. It is indeed extremely complex to produce a realistic facial animation, even for the most skilled artists. One way to solve this issue is to look at it differently: why simulate something we can capture? One approach is thus to capture and reproduce the actor’s performance automatically on a parametrable model. This process, however, still relies heavily on artists and is poorly automatized. How to automatically determine facial animation parameters on a 3D face rig from a Monocular 2D video source? We propose to bring the problem back to the physical world: to automatically determine these animation parameters without human intervention from the motion capture session. Our goal is to give the artists a new tool which would be yielding similar results to the previous process in a more efficient way, without changing the way they work. Our new pipeline consists in a deformation transfer between the actor’s performance and the 3D model by tracking a set of 3D points on the actor’s face and reproducing their movements on the same points on the rig to create a reachable target. The artist doesn’t have to input a neutral pose to the system since it is automatically determined by projecting the neutral pose of the rig into the tracking space. We then iteratively determine the optimal animation parameters by minimizing a least square energy between the rig and the target in animation parameter space. To prevent local minimas, we realign the rig and the target by zones through the iterative process. If the artist is not satisfied with the output, he can correct it using the same tools he is used to use that we adapted to allow error corrections. An artist can now produce a realistic facial animation in half an hour down ten hours using the previous pipeline for the same results. The artist indeed only has to focus on correcting the frames he judges to be not satisfying rather than starting animating from scratch

Realtime Face Tracking and Animation

Capturing and processing human geometry, appearance, and motion is at the core of computer graphics, computer vision, and human-computer interaction. The high complexity of human geometry and motion dynamics, and the high sensitivity of the human visual system to variations and subtleties in faces and bodies make the 3D acquisition and reconstruction of humans in motion a challenging task. Digital humans are often created through a combination of 3D scanning, appearance acquisition, and motion capture, leading to stunning results in recent feature films. However, these methods typically require complex acquisition systems and substantial manual post-processing. As a result, creating and animating high-quality digital avatars entails long turn-around times and substantial production costs. Recent technological advances in RGB-D devices, such as Microsoft Kinect, brought new hopes for realtime, portable, and affordable systems allowing to capture facial expressions as well as hand and body motions. RGB-D devices typically capture an image and a depth map. This permits to formulate the motion tracking problem as a 2D/3D non-rigid registration of a deformable model to the input data. We introduce a novel face tracking algorithm that combines geometry and texture registration with pre-recorded animation priors in a single optimization. This led to unprecedented face tracking quality on a low cost consumer level device. The main drawback of this approach in the context of consumer applications is the need for an offline user-specific training. Robust and efficient tracking is achieved by building an accurate 3D expression model of the user's face who is scanned in a predefined set of facial expressions. We extended this approach removing the need of a user-specific training or calibration, or any other form of manual assistance, by modeling online a 3D user-specific dynamic face model. In complement of a realtime face tracking and modeling algorithm, we developed a novel system for animation retargeting that allows learning a high-quality mapping between motion capture data and arbitrary target characters. We addressed one of the main challenges of existing example-based retargeting methods, the need for a large number of accurate training examples to define the correspondence between source and target expression spaces. We showed that this number can be significantly reduced by leveraging the information contained in unlabeled data, i.e. facial expressions in the source or target space without corresponding poses. Finally, we present a novel realtime physics-based animation technique allowing to simulate a large range of deformable materials such as fat, flesh, hair, or muscles. This approach could be used to produce more lifelike animations by enhancing the animated avatars with secondary effects. We believe that the realtime face tracking and animation pipeline presented in this thesis has the potential to inspire numerous future research in the area of computer-generated animation. Already, several ideas presented in thesis have been successfully used in industry and this work gave birth to the startup company faceshift AG