1,879 research outputs found
GINA-3D: Learning to Generate Implicit Neural Assets in the Wild
Modeling the 3D world from sensor data for simulation is a scalable way of
developing testing and validation environments for robotic learning problems
such as autonomous driving. However, manually creating or re-creating
real-world-like environments is difficult, expensive, and not scalable. Recent
generative model techniques have shown promising progress to address such
challenges by learning 3D assets using only plentiful 2D images -- but still
suffer limitations as they leverage either human-curated image datasets or
renderings from manually-created synthetic 3D environments. In this paper, we
introduce GINA-3D, a generative model that uses real-world driving data from
camera and LiDAR sensors to create realistic 3D implicit neural assets of
diverse vehicles and pedestrians. Compared to the existing image datasets, the
real-world driving setting poses new challenges due to occlusions,
lighting-variations and long-tail distributions. GINA-3D tackles these
challenges by decoupling representation learning and generative modeling into
two stages with a learned tri-plane latent structure, inspired by recent
advances in generative modeling of images. To evaluate our approach, we
construct a large-scale object-centric dataset containing over 520K images of
vehicles and pedestrians from the Waymo Open Dataset, and a new set of 80K
images of long-tail instances such as construction equipment, garbage trucks,
and cable cars. We compare our model with existing approaches and demonstrate
that it achieves state-of-the-art performance in quality and diversity for both
generated images and geometries.Comment: Accepted by CVPR 202
Deep geometric probabilistic models
La géométrie moléculaire, également connue sous le nom de conformation, est la représentation la plus intrinsèque et la plus informative des molécules. Cependant, prédire des conformations stables à partir de graphes moléculaires reste un problème difficile et fondamental en chimie et en biologie computationnelles. Les méthodes expérimentales et computationelles traditionnelles sont généralement coûteuses et chronophages. Récemment, nous avons assisté à des progrès considérables dans l'utilisation de l'apprentissage automatique, en particulier des modèles génératifs, pour accélérer cette procédure. Cependant, les approches actuelles basées sur les données n'ont généralement pas la capacité de modéliser des distributions complexes et ne tiennent pas compte de caractéristiques géométriques importantes. Dans cette thèse, nous cherchons à construire des modèles génératifs basés sur des principes pour la génération de conformation moléculaire qui peuvent surmonter les problèmes ci-dessus. Plus précisément, nous avons proposé des modèles de diffusion basés sur les flux, sur l'énergie et de débruitage pour la génération de structures moléculaires. Cependant, il n'est pas trivial d'appliquer ces modèles à cette tâche où la vraisemblance des géométries devrait avoir la propriété importante d'invariance par rotation par de translation. Inspirés par les progrès récents de l'apprentissage des représentations géométriques, nous fournissons à la fois une justification théorique et une mise en œuvre pratique sur la manière d'imposer cette propriété aux modèles. Des expériences approfondies sur des jeux de données de référence démontrent l'efficacité de nos approches proposées par rapport aux méthodes de référence existantes.Molecular geometry, also known as conformation, is the most intrinsic and informative representation of molecules. However, predicting stable conformations from molecular graphs remains a challenging and fundamental problem in computational chemistry and biology. Traditional experimental and computational methods are usually expensive and time-consuming. Recently, we have witnessed considerable progress in using machine learning, especially generative models, to accelerate this procedure. However, current data-driven approaches usually lack the capacity for modeling complex distributions and fail to take important geometric features into account. In this thesis, we seek to build principled generative models for molecular conformation generation that can overcome the above problems. Specifically, we proposed flow-based, energy-based, and denoising diffusion models for molecular structure generation. However, it's nontrivial to apply these models to this task where the likelihood of the geometries should have the important property of rotational and translation invariance. Inspired by the recent progress of geometric representation learning, we provide both theoretical justification and practical implementation about how to impose this property into the models. Extensive experiments on common benchmark datasets demonstrate the effectiveness of our proposed approaches over existing baseline methods
Vulnerability Clustering and other Machine Learning Applications of Semantic Vulnerability Embeddings
Cyber-security vulnerabilities are usually published in form of short natural
language descriptions (e.g., in form of MITRE's CVE list) that over time are
further manually enriched with labels such as those defined by the Common
Vulnerability Scoring System (CVSS). In the Vulnerability AI (Analytics and
Intelligence) project, we investigated different types of semantic
vulnerability embeddings based on natural language processing (NLP) techniques
to obtain a concise representation of the vulnerability space. We also
evaluated their use as a foundation for machine learning applications that can
support cyber-security researchers and analysts in risk assessment and other
related activities. The particular applications we explored and briefly
summarize in this report are clustering, classification, and visualization, as
well as a new logic-based approach to evaluate theories about the vulnerability
space.Comment: 27 pages, 13 figure
Learning from Invalid Data: On Constraint Satisfaction in Generative Models
Generative models have demonstrated impressive results in vision, language,
and speech. However, even with massive datasets, they struggle with precision,
generating physically invalid or factually incorrect data. This is particularly
problematic when the generated data must satisfy constraints, for example, to
meet product specifications in engineering design or to adhere to the laws of
physics in a natural scene. To improve precision while preserving diversity and
fidelity, we propose a novel training mechanism that leverages datasets of
constraint-violating data points, which we consider invalid. Our approach
minimizes the divergence between the generative distribution and the valid
prior while maximizing the divergence with the invalid distribution. We
demonstrate how generative models like GANs and DDPMs that we augment to train
with invalid data vastly outperform their standard counterparts which solely
train on valid data points. For example, our training procedure generates up to
98 % fewer invalid samples on 2D densities, improves connectivity and stability
four-fold on a stacking block problem, and improves constraint satisfaction by
15 % on a structural topology optimization benchmark in engineering design. We
also analyze how the quality of the invalid data affects the learning procedure
and the generalization properties of models. Finally, we demonstrate
significant improvements in sample efficiency, showing that a tenfold increase
in valid samples leads to a negligible difference in constraint satisfaction,
while less than 10 % invalid samples lead to a tenfold improvement. Our
proposed mechanism offers a promising solution for improving precision in
generative models while preserving diversity and fidelity, particularly in
domains where constraint satisfaction is critical and data is limited, such as
engineering design, robotics, and medicine
- …