The Virtualization Gate Project

Ercim News 80International audienceThe Vgate project introduces a new type of immersive environment that allows full-body immersion and interaction with virtual worlds. The project is a joint initiative between computer scientists from research teams in computer vision, parallel computing and computer graphics at the INRIA Grenoble Rhone-Alpes, and the 4D View Solutions company

ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions

To enable machines to learn how humans interact with the physical world in our daily activities, it is crucial to provide rich data that encompasses the 3D motion of humans as well as the motion of objects in a learnable 3D representation. Ideally, this data should be collected in a natural setup, capturing the authentic dynamic 3D signals during human-object interactions. To address this challenge, we introduce the ParaHome system, designed to capture and parameterize dynamic 3D movements of humans and objects within a common home environment. Our system consists of a multi-view setup with 70 synchronized RGB cameras, as well as wearable motion capture devices equipped with an IMU-based body suit and hand motion capture gloves. By leveraging the ParaHome system, we collect a novel large-scale dataset of human-object interaction. Notably, our dataset offers key advancement over existing datasets in three main aspects: (1) capturing 3D body and dexterous hand manipulation motion alongside 3D object movement within a contextual home environment during natural activities; (2) encompassing human interaction with multiple objects in various episodic scenarios with corresponding descriptions in texts; (3) including articulated objects with multiple parts expressed with parameterized articulations. Building upon our dataset, we introduce new research tasks aimed at building a generative model for learning and synthesizing human-object interactions in a real-world room setting

Modélisation 4D à partir de plusieurs caméras

Les systèmes multi-caméras permettent de nos jours d'obtenir à la fois des flux d'images couleur mais aussi des flux de modèles 3D. Ils permettent ainsi l'étude de scènes complexes à la fois de par les éléments qui la composent mais aussi de par les mouvements et les déformations que subissent ces éléments au fil du temps. Une des principales limitations de ces données est le manque de cohérence temporelle entre les observations obtenues à deux instants de temps successifs. Les travaux présentés dans cette thèse proposent des pistes pour retrouver cette cohérence temporelle. Dans un premier temps nous nous sommes penchés sur le problème de l'estimation de champs de déplacement denses à la surface des objets de la scène. L'approche que nous proposons permet de combiner efficacement des informations photométriques provenant des caméras avec des informations géométriques. Cette méthode a été étendue, par la suite, au cas de systèmes multi-caméras hybrides composés de capteurs couleurs et de profondeur (tel que le capteur kinect). Dans un second temps nous proposons une méthode nouvelle permettant l'apprentissage de la vraie topologie d'une scène dynamique au fil d'une séquence de données 4D (3D + temps). Ces travaux permettent de construire au fur et à mesure des observations un modèle de référence de plus en plus complet de la scène observée.Nowadays mutli-camera setups allow the acquisition of both color image streams and 3D models streams. Thus permitting the study of complex scenes. These scenes can be composed of any number of non-rigid objects moving freely. One of the main limitations of such data is its lack of temporal coherence between two consecutive observations. The work presented in this thesis consider this issue and propose novel methods to recover this temporal coherence. First we present a new approach that computes at each frame a dense motion field over the surface of the scene (i.e. Scene Flow), gathering both photometric and geometric information. We then extend this approach to hybrid multi-camera setups composed of color and depth sensor (such as the kinect sensor). Second, we introduce "Progressive Shape Models", a new method that allows to gather topology information over a complete sequence of 3D models and incrementally build a complete and coherent surface template.SAVOIE-SCD - Bib.électronique (730659901) / SudocGRENOBLE1/INP-Bib.électronique (384210012) / SudocGRENOBLE2/3-Bib.électronique (384219901) / SudocSudocFranceF

Virtualization Gate

International audienceThis project associates multi-camera 3D modeling, physical simulation, and tracked head-mounted displays for a strong full-body immersion and presence in virtual worlds. Three-dimensional modeling is based on the EPHV algorithm, which provides an exact geometrical surface according to input data. The geometry enables computation of full-body collisions with virtual objects animated by a physical simulation. Since the algorithm is exact, it allows for consistent texture mapping and yields qualitative models. This full-body representation can thus be rendered on a distant site for telepresence. It can also be rendered into a HMD. Users see their 3D models superposed with their real bodies occluded by the HMD. Since the displays are hold in front of the eyes, the image projection is not impaired by elements of the real world. With a fixed screen, even in an immersive Cave-like configuration, users would not be able to see a virtual object in their palms, because their hands would occlude the light emitted by the displays. With our approach, users see the 3D model of their hands and the virtual object correctly positioned in their palms. It enables a first-person viewing and occlusion-free, co-located interactions. The 3D modeling system makes no assumption about the scene observed. One or several persons can stand in the acquisition space. The number of persons only affects the model quality and the computation time. We presented a similar concept, the Grimage platform, in SIGGRAPH 2007 Emerging Technologies. But the acquisition space was limited to a small volume for modeling the user's hands, and images were rendered on a fixed screen positioned behind the acquisition space, providing little immersion and only third-person interactions

The Virtualization Gate Project

Ercim News 80International audienceThe Vgate project introduces a new type of immersive environment that allows full-body immersion and interaction with virtual worlds. The project is a joint initiative between computer scientists from research teams in computer vision, parallel computing and computer graphics at the INRIA Grenoble Rhone-Alpes, and the 4D View Solutions company