New editing techniques for video post-processing

This thesis contributes to capturing 3D cloth shape, editing cloth texture and altering object shape and motion in multi-camera and monocular video recordings. We propose a technique to capture cloth shape from a 3D scene flow by determining optical flow in several camera views. Together with a silhouette matching constraint we can track and reconstruct cloth surfaces in long video sequences. In the area of garment motion capture, we present a system to reconstruct time-coherent triangle meshes from multi-view video recordings. Texture mapping of the acquired triangle meshes is used to replace the recorded texture with new cloth patterns. We extend this work to the more challenging single camera view case. Extracting texture deformation and shading effects simultaneously enables us to achieve texture replacement effects for garments in monocular video recordings. Finally, we propose a system for the keyframe editing of video objects. A color-based segmentation algorithm together with automatic video inpainting for filling in missing background texture allows us to edit the shape and motion of 2D video objects. We present examples for altering object trajectories, applying non-rigid deformation and simulating camera motion.In dieser Dissertation stellen wir Beiträge zur 3D-Rekonstruktion von Stoffoberfächen, zum Editieren von Stofftexturen und zum Editieren von Form und Bewegung von Videoobjekten in Multikamera- und Einkamera-Aufnahmen vor. Wir beschreiben eine Methode für die 3D-Rekonstruktion von Stoffoberflächen, die auf der Bestimmung des optischen Fluß in mehreren Kameraansichten basiert. In Kombination mit einem Abgleich der Objektsilhouetten im Video und in der Rekonstruktion erhalten wir Rekonstruktionsergebnisse für längere Videosequenzen. Für die Rekonstruktion von Kleidungsstücken beschreiben wir ein System, das zeitlich kohärente Dreiecksnetze aus Multikamera-Aufnahmen rekonstruiert. Mittels Texturemapping der erhaltenen Dreiecksnetze wird die Stofftextur in der Aufnahme mit neuen Texturen ersetzt. Wir setzen diese Arbeit fort, indem wir den anspruchsvolleren Fall mit nur einer einzelnen Videokamera betrachten. Um realistische Resultate beim Ersetzen der Textur zu erzielen, werden sowohl Texturdeformationen durch zugrundeliegende Deformation der Oberfläche als auch Beleuchtungseffekte berücksichtigt. Im letzten Teil der Dissertation stellen wir ein System zum Editieren von Videoobjekten mittels Keyframes vor. Dies wird durch eine Kombination eines farbbasierten Segmentierungsalgorithmus mit automatischem Auffüllen des Hintergrunds erreicht, wodurch Form und Bewegung von 2D-Videoobjekten editiert werden können. Wir zeigen Beispiele für editierte Objekttrajektorien, beliebige Deformationen und simulierte Kamerabewegung

Model-free Consensus Maximization for Non-Rigid Shapes

Many computer vision methods use consensus maximization to relate measurements containing outliers with the correct transformation model. In the context of rigid shapes, this is typically done using Random Sampling and Consensus (RANSAC) by estimating an analytical model that agrees with the largest number of measurements (inliers). However, small parameter models may not be always available. In this paper, we formulate the model-free consensus maximization as an Integer Program in a graph using `rules' on measurements. We then provide a method to solve it optimally using the Branch and Bound (BnB) paradigm. We focus its application on non-rigid shapes, where we apply the method to remove outlier 3D correspondences and achieve performance superior to the state of the art. Our method works with outlier ratio as high as 80\%. We further derive a similar formulation for 3D template to image matching, achieving similar or better performance compared to the state of the art.Comment: ECCV1

Multi-Garment Net: {L}earning to Dress {3D} People from Images

We present Multi-Garment Network (MGN), a method to predict body shape and clothing, layered on top of the SMPL model from a few frames (1-8) of a video. Several experiments demonstrate that this representation allows higher level of control when compared to single mesh or voxel representations of shape. Our model allows to predict garment geometry, relate it to the body shape, and transfer it to new body shapes and poses. To train MGN, we leverage a digital wardrobe containing 712 digital garments in correspondence, obtained with a novel method to register a set of clothing templates to a dataset of real 3D scans of people in different clothing and poses. Garments from the digital wardrobe, or predicted by MGN, can be used to dress any body shape in arbitrary poses. We will make publicly available the digital wardrobe, the MGN model, and code to dress SMPL with the garments

DIFFERENTIABLE POSITION-BASED SIMULATIONS

The present disclosure generally relates to enabling an efficient differentiable unified simulation model with contact, and more specifically, by computing derivatives of high degrees of freedom (DoF) and high geometric resolutions

Diffusion Shape Prior for Wrinkle-Accurate Cloth Registration

Registering clothes from 4D scans with vertex-accurate correspondence is challenging, yet important for dynamic appearance modeling and physics parameter estimation from real-world data. However, previous methods either rely on texture information, which is not always reliable, or achieve only coarse-level alignment. In this work, we present a novel approach to enabling accurate surface registration of texture-less clothes with large deformation. Our key idea is to effectively leverage a shape prior learned from pre-captured clothing using diffusion models. We also propose a multi-stage guidance scheme based on learned functional maps, which stabilizes registration for large-scale deformation even when they vary significantly from training data. Using high-fidelity real captured clothes, our experiments show that the proposed approach based on diffusion models generalizes better than surface registration with VAE or PCA-based priors, outperforming both optimization-based and learning-based non-rigid registration methods for both interpolation and extrapolation tests.Comment: Project page: https://www-users.cse.umn.edu/~guo00109/projects/3dv2024

Telemetry tails: a practical method for attaching animal-borne devices to small vertebrates in the field

Context. Continued miniaturisation of tracking technology increases its utility in animal applications. However, species morphology often dictates the type of animal-borne device (ABD) that can be used, and how it is attached. The morphology of species within Peramelemorphia preclude them from the standard collar attachment of ABDs for terrestrial mammals. Aims. This paper describes a method for the tail-mount attachment of ABDs, and deployment results for Peramelemorphia across arid, semi-arid and temperate Australia to (a) test the performance of attachments and ABDs in the field and (b) discuss the animal welfare considerations for this attachment method. Methods. Tail-mount attachment of ABDs were field-tested on a total of 80 greater bilbies (Macrotis lagotis), and 14 long-nosed bandicoots (Perameles nasuta). Key results. Time to natural detachment (TTND) was between 2 and 52 days, with 65.74% (142 of 216) remaining on until manual removal. For ABDs that were manually removed, attachments were retained for up to 94 days. The method used for tail-mount attachment of ABDs to long-nosed bandicoots resulted in significantly shorter TTND compared with the method used for bilbies, and environmental factors (high temperatures and rainfall) had a negative effect on TTND. Tail-mount attached global positioning system (GPS) sensors collected large quantities of accurate data, with a maximum fix success rate of 83.38%. Damage to GPS (antenna breakage and water ingress) during deployment, however, impacted performance. In environments with frequent rainfall and waterlogged soils, the tape on a small proportion (6.25%) of (n = 192)attachments to bilbies caused tail injury. All injuries were resolvable, with most requiringminimal to no veterinary intervention. Key conclusions. Attachment longevity can be affectedby how the ABD is mounted to the tail, the species and the deployment environment. Theenvironment can also affect which adhesive tapes are suitable for ABD attachment. However, thismethod is highly modifiable, practical for field application and can have long retention timesrelative to other temporary methods. Implications. This ABD tail-mount attachment methodadds another tool to the telemetry tool-kit, with all the benefits of a low-tech, low-cost, passivedrop-off type attachment. This method has demonstrated practicality for Peramelemorphia, withpotential application to other suitable small vertebrates

Corticosterone secretion in tuatara (Sphenodon punctatus): Influential factors and conservation applications

Animals are regularly exposed to environmental, social and physiological challenges. In reaction to these challenges, individuals adjust their physiology and behaviour to maintain essential processes and optimise fitness. The most widely used indicators of physiological stress in vertebrates are glucocorticoid hormones (corticosterone (CORT) or cortisol), which are commonly referred to as ‘stress hormones’. The use of CORT as a tool to understand how individuals respond to natural or human-caused challenges is central to stress physiology research. Here, I investigated intrinsic and extrinsic factors associated with CORT secretion, CORT secretion as an indicator of physiological response to challenges/stressors, and the value of CORT secretion as conservation tool in an iconic protected reptile (the tuatara, Sphenodon punctatus). A capture-restraint time series revealed a significant CORT response over a 24 h period in male and female (non-gravid and gravid) tuatara. Baseline CORT and the CORT response to capture and restraint (i.e. a standardised capture-stress protocol) were similar between sexes; however, female reproductive condition was correlated with CORT secretion in that higher baseline CORT and a lower CORT response were observed in gravid females. An observational study incorporating data across a wide range of ambient temperatures (from four island sites) confirmed that body temperature (Tb) is positively correlated with baseline CORT in gravid females only, and revealed a positive correlation between the CORT response and higher Tb in all adults. A supporting experimental study showed that acute ambient temperature increase (in which mean Tb reached 21.4±0.4°C) elicits a significant CORT response to capture-restraint in gravid females. These results confirmed that gravid females are not secreting CORT maximally during nesting, but actively modulate secretion. An inter-island comparison of CORT secretion (for four populations) revealed that baseline CORT secretion was similar among populations during the non-breeding and breeding seasons; however, the CORT response to capture-restraint varied significantly among populations. Inter-population variation in testosterone (T) was observed in males (but not females) and was positively linked with increased baseline CORT from the non-breeding season to the breeding season, suggesting male reproductive activity may drive seasonal change of baseline CORT. Significant correlations were observed between the CORT response to capture-restraint (but not baseline CORT) and habitat elements of latitude, tuatara density and seabird abundance and 2) demogenetic factors of sex ratio and genetic diversity. The measurement of CORT as a physiological monitoring tool indicated that short- and long- term dynamics of CORT secretion in tuatara are not altered through translocation to a new island, as the acute CORT response remained stable throughout exposure to cumulative stressors and long-term dynamics of CORT secretion in translocated populations simultaneously mirrored those in source populations. These findings deliver the most detailed study of CORT secretion patterns in tuatara to date. Moreover, as the first study to apply CORT secretion data as an conservation physiology monitoring tool in tuatara, these results serve as a baseline reference for future research and monitoring of conservation efforts