Fast and deep deformation approximations

Character rigs are procedural systems that compute the shape of an animated character for a given pose. They can be highly complex and must account for bulges, wrinkles, and other aspects of a character's appearance. When comparing film-quality character rigs with those designed for real-time applications, there is typically a substantial and readily apparent difference in the quality of the mesh deformations. Real-time rigs are limited by a computational budget and often trade realism for performance. Rigs for film do not have this same limitation, and character riggers can make the rig as complicated as necessary to achieve realistic deformations. However, increasing the rig complexity slows rig evaluation, and the animators working with it can become less efficient and may experience frustration. In this paper, we present a method to reduce the time required to compute mesh deformations for film-quality rigs, allowing better interactivity during animation authoring and use in real-time games and applications. Our approach learns the deformations from an existing rig by splitting the mesh deformation into linear and nonlinear portions. The linear deformations are computed directly from the transformations of the rig's underlying skeleton. We use deep learning methods to approximate the remaining nonlinear portion. In the examples we show from production rigs used to animate lead characters, our approach reduces the computational time spent on evaluating deformations by a factor of 5×-10×. This significant savings allows us to run the complex, film-quality rigs in real-time even when using a CPU-only implementation on a mobile device

LEARNING TO RIG CHARACTERS

With the emergence of 3D virtual worlds, 3D social media, and massive online games, the need for diverse, high-quality, animation-ready characters and avatars is greater than ever. To animate characters, artists hand-craft articulation structures, such as animation skeletons and part deformers, which require significant amount of manual and laborious interaction with 2D/3D modeling interfaces. This thesis presents deep learning methods that are able to significantly automate the process of character rigging. First, the thesis introduces RigNet, a method capable of predicting an animation skeleton for an input static 3D shape in the form of a polygon mesh. The predicted skeletons match the animator expectations in joint placement and topology. RigNet also estimates surface skin weights which determine how the mesh is animated given the different skeletal poses. In contrast to prior work that fits pre-defined skeletal templates with hand-tuned objectives, RigNet is able to automatically rig diverse characters, such as humanoids, quadrupeds, toys, birds, with varying articulation structure and geometry. RigNet is based on a deep neural architecture that directly operates on the mesh representation. The architecture is trained on a diverse dataset of rigged models that we mined online and curated. The dataset includes 2.7K polygon meshes, along with their associated skeletons and corresponding skin weights. Second, the thesis introduces Morig, a method that automatically rigs character meshes driven by single-view point cloud streams capturing the motion of performing characters. Compared to RigNet, MoRig\u27s rigging is \emph{motion-aware}: its neural network encodes motion cues from the point clouds into compact feature representations that are informative about the articulated parts of the performing character. These motion-aware features guide the inference of an appropriate skeletal rig for the input mesh. Furthermore, Morig is able to animate the rig according to the captured point cloud motion. Morig can handle diverse characters with different morphologies (e.g., humanoids, quadrupeds, toy characters). It also accounts for occluded regions in the point clouds and mismatches in the part proportions between the input mesh and captured character. Third, the thesis introduces APES, a method that takes as input 2D raster images depicting a small set of poses of a character shown in a sprite sheet, and identifies articulated parts useful for rigging the character. APES uses a combination of neural network inference and integer linear programming to identify a compact set of articulated body parts, e.g. head, torso and limbs, that best reconstruct the input poses. Compared to Morig and RigNet that require a large collection of training models with associated skeletons and skinning weights, APES\u27 neural architecture relies on less effortful supervision from (i) pixel correspondences readily available in existing large cartoon image datasets (e.g., Creative Flow), (ii) a relatively small dataset of 57 cartoon characters segmented into moving parts. Finally, the thesis discusses future research directions related to combining neural rigging with 3D and 4D reconstruction of characters from point cloud data and 2D video as well as automating the process of motion synthesis for 3D characters

Velum: A 3D Puzzle Game and Facial Analysis Study

Velum is a first-person 3D puzzle/exploration game set in a timeless version of the Boston Public Garden. The project’s narrative framework and aesthetics are based on one of the Garden’s most prominent features, the Ether Monument, which commemorates the 1846 discovery of diethyl ether’s effectiveness as a medical anesthetic. A sequence of nine abstract challenges is rewarded by a progressive revelation of the player’s mysterious identity and purpose. The puzzle design was informed by the use of crowdsourced playtesting involving 300+ volunteers, combining standard data telemetry with AI-based facial image analysis capable of mapping player emotions to gameplay events

Velum: A 3D Puzzle Game and Facial Analysis Study

Velum is a first-person 3D puzzle/exploration game set in a timeless version of the Boston Public Garden. The project’s narrative framework and aesthetics are based on one of the Garden’s most prominent features, the Ether Monument, which commemorates the 1846 discovery of diethyl ether’s effectiveness as a medical anesthetic. A sequence of nine abstract challenges is rewarded by a progressive revelation of the player’s mysterious identity and purpose. The puzzle design was informed by the use of crowdsourced playtesting involving 300+ volunteers, combining standard data telemetry with AI-based facial image analysis capable of mapping player emotions to gameplay events

VELUM: A 3D Puzzle/Exploration Game Designed Using Crowdsourced AI Facial Analysis

Velum is a first-person 3D puzzle/exploration game set in a timeless version of the Boston Public Garden. The project’s narrative framework and aesthetics are based on one of the Garden’s most prominent features, the Ether Monument, which commemorates the 1846 discovery of diethyl ether’s effectiveness as a medical anesthetic. A sequence of nine abstract challenges is rewarded by a progressive revelation of the player’s mysterious identity and purpose. The puzzle design was informed by the use of crowdsourced playtesting involving 300+ volunteers, combining standard data telemetry with AI-based facial image analysis capable of mapping player emotions to gameplay events

Epi-genomic determinants of HIV-1 integration in primary CD4+ T cells and macrophages

The infection with HIV-1 nowadays does not represent a condition with a deadly outcome. Due to current therapeutic approaches, the infection with HIV-1 represents a chronic condition in which viral load is kept at undetectable levels, but patients depend on a lifelong therapy without a chance of cure. The eradication of integrated viral DNA still remains the biggest challenge in curing HIV-1. The aim of this work was to contribute to a better understanding and definition of genomic regions and epi-genomic features that HIV-1 targets for integration, and give a detailed description on the importance of chromatin accessibility, as well as the importance of certain genomic features in the process of HIV-1 integration. The first part of this project deals with the importance of HMT G9a activity and H3K9me2 histone mark distribution and deposition in the context of HIV-1 integration in primary CD4+ T cells, which was studied by the application of G9a inhibitor BIX0129, also known as a very potent latency reversing agent. The significance of G9a activity and facultative heterochromatin mark H3K9me2 deposition has previously been shown to affect T cell development and impact shaping of the nuclear architecture. In this work it was demonstrated that the chemical inhibition of G9a and depletion of H3K9me2 by BIX01294 has an increasing effect on HIV-1 integration. The increase in integration was also followed by increased viral transcriptional activity, as well as spatial repositioning of the provirus from the preferred nuclear periphery towards the nuclear center. Similar spatial repositioning has been demonstrated for genes highly and recurrently targeted by HIV-1 for integration (RIGs). However, genic nuclear repositioning upon BIX01294 treatment did not affect transcriptional profiles of HIV-1 RIGs, as demonstrated by RNA microarray analysis, but other groups of genes mainly involved in iron metabolism and inflammatory response were upregulated upon BIX01294 treatment. In addition, HIV-1 integration patterns were shown not to be affected by H3K9me2 depletion, and the virus was still targeting similar genic regions for integration. The analysis of chromatin mark distribution and chromatin binding elements upon BIX01294 treatment on RIGs revealed increased binding profiles of open chromatin mark H3K36me3 which is followed by increased LEDGF/p75 binding upon H3K9me2 depletion. The observed phenomenon might provide an explanation for the observed increased viral integration upon BIX01294 treatment, considering that LEDGF/p75 is a prominent host cell factor involved in the viral integration process. Overall, the first part of this study clearly demonstrated that chromatin accessibility significantly affects HIV-1 integration levels which are directly proportional to viral expression levels and viral activity. The second part of this study deals with the relevance of R-loops, as specific genomic structures, as sites selected for HIV-1 integration in primary CD4+ T cells and macrophages. It was demonstrated that the GFP tagged IN enzyme of HIV-1, in a high occurrence, colocalizes with R-loops in cells, and that for the occurrence of this process a functionally active IN is required. This finding implicated that the observed colocalization is not randomly taking place and that HIV-1 is actively docked to R-loop forming genomic sites. In addition, biochemical as well as computational meta data analysis revealed that HIV-1 RIGs are enriched in R-loops and that R-loop forming sites can accommodate integrated viral DNA. Further on, it was demonstrated that HIV-1 IN has R-loop binding capacity and is also capable of performing the strand transfer reaction on R-loop containing DNA templates. It was also demonstrated that R-loop depletion by RNase H1 overexpression in several cell lines, as well as in primary cells, significantly impairs HIV-1 integration, indicating that R-loop presence is crucial for efficient HIV-1 integration. In line with this result was the finding that RIGs expression was not affected by R-loop removal, indicating that only the presence of R-loops, as structural genomic elements, is more affecting HIV-1 integration compared to gene expression levels. The final finding is also in line with previous work from our lab. In summary, the second part of this study provides strong evidence that R-loops represent structural genomic elements targeted by HIV-1 for integration and also gives new insight into HIV-1 IN functional III features which have not been addressed before

Vocaodoru - Rhythm Gaming and Artificial Cinematography in Virtual Reality

Vocaodoru is a virtual reality rhythm game centered around two novel components. The gameplay of Vocaodoru is a never before-seen pose-based gameplay system that uses a player’s measurements to adapt gameplay to their needs. Tied to the gameplay is a human-in-the-loop utility AI that controls a cinematographic camera to allow streamers to broadcast a more interesting, dynamic view of the player. We discuss our efforts to develop and connect these components and how we plan to continue development after the conclusion of the MQP