Capturing Hands in Action using Discriminative Salient Points and Physics Simulation

Hand motion capture is a popular research field, recently gaining more attention due to the ubiquity of RGB-D sensors. However, even most recent approaches focus on the case of a single isolated hand. In this work, we focus on hands that interact with other hands or objects and present a framework that successfully captures motion in such interaction scenarios for both rigid and articulated objects. Our framework combines a generative model with discriminatively trained salient points to achieve a low tracking error and with collision detection and physics simulation to achieve physically plausible estimates even in case of occlusions and missing visual data. Since all components are unified in a single objective function which is almost everywhere differentiable, it can be optimized with standard optimization techniques. Our approach works for monocular RGB-D sequences as well as setups with multiple synchronized RGB cameras. For a qualitative and quantitative evaluation, we captured 29 sequences with a large variety of interactions and up to 150 degrees of freedom.Comment: Accepted for publication by the International Journal of Computer Vision (IJCV) on 16.02.2016 (submitted on 17.10.14). A combination into a single framework of an ECCV'12 multicamera-RGB and a monocular-RGBD GCPR'14 hand tracking paper with several extensions, additional experiments and detail

Doctor of Philosophy in Computing

dissertationPhysics-based animation has proven to be a powerful tool for creating compelling animations for film and games. Most techniques in graphics are based on methods developed for predictive simulation for engineering applications; however, the goals for graphics applications are dramatically different than the goals of engineering applications. As a result, most physics-based animation tools are difficult for artists to work with, providing little direct control over simulation results. In this thesis, we describe tools for physics-based animation designed with artist needs and expertise in mind. Most materials can be modeled as elastoplastic: they recover from small deformations, but large deformations permanently alter their rest shape. Unfortunately, large plastic deformations, common in graphical applications, cause simulation instabilities if not addressed. Most elastoplastic simulation techniques in graphics rely on a finite-element approach where objects are discretized into a tetrahedral mesh. Using these approaches, maintaining simulation stability during large plastic flows requires remeshing, a complex and computationally expensive process. We introduce a new point-based approach that does not rely on an explicit mesh and avoids the expense of remeshing. Our approach produces comparable results with much lower implementation complexity. Points are a ubiquitous primitive for many effects, so our approach also integrates well with existing artist pipelines. Next, we introduce a new technique for animating stylized images which we call Dynamic Sprites. Artists can use our tool to create digital assets that interact in a natural, but stylized, way in virtual environments. In order to support the types of nonphysical, exaggerated motions often desired by artists, our approach relies on a heavily modified deformable body simulator, equipped with a set of new intuitive controls and an example-based deformation model. Our approach allows artists to specify how the shape of the object should change as it moves and collides in interactive virtual environments. Finally, we introduce a new technique for animating destructive scenes. Our approach is built on the insight that the most important visual aspects of destruction are plastic deformation and fracture. Like with Dynamic Sprites, we use an example-based model of deformation for intuitive artist control. Our simulator treats objects as rigid when computing dynamics but allows them to deform plastically and fracture in between timesteps based on interactions with the other objects. We demonstrate that our approach can efficiently animate the types of destructive scenes common in film and games. These animation techniques are designed to exploit artist expertise to ease creation of complex animations. By using artist-friendly primitives and allowing artists to provide characteristic deformations as input, our techniques enable artists to create more compelling animations, more easily

MULTISCALE DYNAMIC MONTE CARLO / CONTINUUM MODELING OF DRYING AND CURING IN SOL-GEL SILICA FILMS

Modeling the competition between drying and curing processes in polymerizing films is of great importance to many existing and developing materials synthesis processes. These processes involve multiple length and time scales ranging from molecular to macroscopic, and are challenging to fully model in situations where the polymerization is non-ideal, such as sol-gel silica thin film formation. A comprehensive model of sol-gel silica film formation should link macroscopic flow and drying (controlled by process parameters) to film microstructure (which dictates the properties of the films). This dissertation describes a multiscale model in which dynamic Monte Carlo (DMC) polymerization simulations are coupled to a continuum model of drying. Unlike statistical methods, DMC simulations track the entire molecular structure distribution to allow the calculation not only of molecular weight but also of cycle ranks and topological indices related to molecular size and shape. The entire DMC simulation (containing 106 monomers) is treated as a particle of sol whose position and composition are tracked in the continuum mass transport model of drying. The validity of the multiscale model is verified by the good agreement of the conversion evolution of DMC and continuum simulations for ideal polycondensation and first shell substitution effect (FSSE) cases. Because our model allows cyclic and cage-like siloxanes to form, it is better able to predict the silica gelation conversion than other reported kinetic models. By studying the competition between molecular growth and cyclization, and the competition between mass transfer (drying) and reaction (gelation) on the drying process of the sol-gel silica film, we observe that cyclization delays gelation, shrinks the molecular size, increases the likelihood of skin formation, and leads to a molecular structure gradient inside the film. We also find that compared with a model with only 3-membered rings, the molecular structure is more complicated and the structure gradients in the films are larger with 4- membered rings. We expect that our simulation will allow better prediction of the formation of structure gradients in sol-gel derived ceramics and other nonideal multifunctional polycondensation products, and that this will help in developing procedures to reduce coating defects

Split and Join: An Efficient Approach for Simulating Stapled Intestinal Anastomosis in Virtual Reality

Colorectal cancer is a life-threatening disease. It is the second leading cause of cancer-related deaths in the United States. Stapled anastomosis is a rapid treatment for colorectal cancer and other intestinal diseases and has become an integral part of routine surgical practice. However, to the best of our knowledge, there is no existing work simulating intestinal anastomosis that often involves sophisticated soft tissue manipulations such as cutting and stitching. In this paper, for the first time, we propose a novel split and join approach to simulate a side-to-side stapled intestinal anastomosis in virtual reality. We mimic the intestine model using a new hybrid representation—a grid-linked particles model for physics simulation and a surface mesh for rendering. The proposed split and join operations handle the updates of both the grid-linked particles model and the surface mesh during the anastomosis procedure. The simulation results demonstrate the feasibility of the proposed approach in simulating intestine models and the side-to-side anastomosis operation

Deep Detail Enhancement for Any Garment

Creating fine garment details requires significant efforts and huge computational resources. In contrast, a coarse shape may be easy to acquire in many scenarios (e.g., via low-resolution physically-based simulation, linear blend skinning driven by skeletal motion, portable scanners). In this paper, we show how to enhance, in a data-driven manner, rich yet plausible details starting from a coarse garment geometry. Once the parameterization of the garment is given, we formulate the task as a style transfer problem over the space of associated normal maps. In order to facilitate generalization across garment types and character motions, we introduce a patch-based formulation, that produces high-resolution details by matching a Gram matrix based style loss, to hallucinate geometric details (i.e., wrinkle density and shape). We extensively evaluate our method on a variety of production scenarios and show that our method is simple, light-weight, efficient, and generalizes across underlying garment types, sewing patterns, and body motion.Comment: 12 page

Embedded Implicit Stand-ins for Animated Meshes: a Case of Hybrid Modelling

In this paper we address shape modelling problems, encountered in computer animation and computer games development that are difficult to solve just using polygonal meshes. Our approach is based on a hybrid modelling concept that combines polygonal meshes with implicit surfaces. A hybrid model consists of an animated polygonal mesh and an approximation of this mesh by a convolution surface stand-in that is embedded within it or is attached to it. The motions of both objects are synchronised using a rigging skeleton. This approach is used to model the interaction between an animated mesh object and a viscoelastic substance, normally modelled in implicit form. The adhesive behaviour of the viscous object is modelled using geometric blending operations on the corresponding implicit surfaces. Another application of this approach is the creation of metamorphosing implicit surface parts that are attached to an animated mesh. A prototype implementation of the proposed approach and several examples of modelling and animation with near real-time preview times are presented