8 research outputs found
Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design
Deep generative models such as Variational Autoencoders (VAEs), Generative
Adversarial Networks (GANs), Diffusion Models, and Transformers, have shown
great promise in a variety of applications, including image and speech
synthesis, natural language processing, and drug discovery. However, when
applied to engineering design problems, evaluating the performance of these
models can be challenging, as traditional statistical metrics based on
likelihood may not fully capture the requirements of engineering applications.
This paper doubles as a review and practical guide to evaluation metrics for
deep generative models (DGMs) in engineering design. We first summarize the
well-accepted `classic' evaluation metrics for deep generative models grounded
in machine learning theory. Using case studies, we then highlight why these
metrics seldom translate well to design problems but see frequent use due to
the lack of established alternatives. Next, we curate a set of design-specific
metrics which have been proposed across different research communities and can
be used for evaluating deep generative models. These metrics focus on unique
requirements in design and engineering, such as constraint satisfaction,
functional performance, novelty, and conditioning. Throughout our discussion,
we apply the metrics to models trained on simple-to-visualize 2-dimensional
example problems. Finally, we evaluate four deep generative models on a bicycle
frame design problem and structural topology generation problem. In particular,
we showcase the use of proposed metrics to quantify performance target
achievement, design novelty, and geometric constraints. We publicly release the
code for the datasets, models, and metrics used throughout the paper at
https://decode.mit.edu/projects/metrics/
Learning from Invalid Data: On Constraint Satisfaction in Generative Models
Generative models have demonstrated impressive results in vision, language,
and speech. However, even with massive datasets, they struggle with precision,
generating physically invalid or factually incorrect data. This is particularly
problematic when the generated data must satisfy constraints, for example, to
meet product specifications in engineering design or to adhere to the laws of
physics in a natural scene. To improve precision while preserving diversity and
fidelity, we propose a novel training mechanism that leverages datasets of
constraint-violating data points, which we consider invalid. Our approach
minimizes the divergence between the generative distribution and the valid
prior while maximizing the divergence with the invalid distribution. We
demonstrate how generative models like GANs and DDPMs that we augment to train
with invalid data vastly outperform their standard counterparts which solely
train on valid data points. For example, our training procedure generates up to
98 % fewer invalid samples on 2D densities, improves connectivity and stability
four-fold on a stacking block problem, and improves constraint satisfaction by
15 % on a structural topology optimization benchmark in engineering design. We
also analyze how the quality of the invalid data affects the learning procedure
and the generalization properties of models. Finally, we demonstrate
significant improvements in sample efficiency, showing that a tenfold increase
in valid samples leads to a negligible difference in constraint satisfaction,
while less than 10 % invalid samples lead to a tenfold improvement. Our
proposed mechanism offers a promising solution for improving precision in
generative models while preserving diversity and fidelity, particularly in
domains where constraint satisfaction is critical and data is limited, such as
engineering design, robotics, and medicine
Data-Driven Bicycle Design using Performance-Aware Deep Generative Models
This treatise explores the application of Deep Generative Machine Learning Models to bicycle design and optimization. Deep Generative Models have been growing in popularity across the design community thanks to their ability to learn and mimic complex data distributions. This work addresses several key bottlenecks in the developing field, such as performance-aware generation, inverse design, and design validity. To support development of deep generative models, this treatise develops a foundation for data-driven design of bicycles, introducing three datasets: BIKED, BIKER, and FRAMED, considering holistic bicycle design, aerodynamic optimization, and structural optimization of bicycles respectively. It further proposes a set of tractable bicycle design tools, such as surrogate models to rapidly estimate performance of design candidates, analysis tools to guide the design process, and targeted design refinement tools using counterfactual explanations. This treatise finally proposes the first Deep Generative Model that actively optimizes for realism, performance, diversity, feasibility, and target satisfaction simultaneously. The proposed model achieves sweeping improvements over numerous evaluation criteria when compared to existing methods and establishes state-of-the-art performance on the bicycle design problem.S.M
Design Target Achievement Index: A Differentiable Metric to Enhance Deep Generative Models in Multi-Objective Inverse Design
Deep Generative Machine Learning Models have been growing in popularity
across the design community thanks to their ability to learn and mimic complex
data distributions. While early works are promising, further advancement will
depend on addressing several critical considerations such as design quality,
feasibility, novelty, and targeted inverse design. We propose the Design Target
Achievement Index (DTAI), a differentiable, tunable metric that scores a
design's ability to achieve designer-specified minimum performance targets. We
demonstrate that DTAI can drastically improve the performance of generated
designs when directly used as a training loss in Deep Generative Models. We
apply the DTAI loss to a Performance-Augmented Diverse GAN (PaDGAN) and
demonstrate superior generative performance compared to a set of baseline Deep
Generative Models including a Multi-Objective PaDGAN and specialized tabular
generation algorithms like the Conditional Tabular GAN (CTGAN). We further
enhance PaDGAN with an auxiliary feasibility classifier to encourage feasible
designs. To evaluate methods, we propose a comprehensive set of evaluation
metrics for generative methods that focus on feasibility, diversity, and
satisfaction of design performance targets. Methods are tested on a challenging
benchmarking problem: the FRAMED bicycle frame design dataset featuring
mixed-datatype parametric data, heavily skewed and multimodal distributions,
and ten competing performance objectives
Towards Goal, Feasibility, and Diversity-Oriented Deep Generative Models in Design
Deep Generative Machine Learning Models (DGMs) have been growing in
popularity across the design community thanks to their ability to learn and
mimic complex data distributions. DGMs are conventionally trained to minimize
statistical divergence between the distribution over generated data and
distribution over the dataset on which they are trained. While sufficient for
the task of generating "realistic" fake data, this objective is typically
insufficient for design synthesis tasks. Instead, design problems typically
call for adherence to design requirements, such as performance targets and
constraints. Advancing DGMs in engineering design requires new training
objectives which promote engineering design objectives. In this paper, we
present the first Deep Generative Model that simultaneously optimizes for
performance, feasibility, diversity, and target achievement. We benchmark
performance of the proposed method against several Deep Generative Models over
eight evaluation metrics that focus on feasibility, diversity, and satisfaction
of design performance targets. Methods are tested on a challenging
multi-objective bicycle frame design problem with skewed, multimodal data of
different datatypes. The proposed framework was found to outperform all Deep
Generative Models in six of eight metrics.Comment: arXiv admin note: substantial text overlap with arXiv:2205.0300
BIKED: A Dataset for Computational Bicycle Design with Machine Learning Benchmarks
Abstract
In this paper, we present “BIKED,” a dataset comprised of 4500 individually designed bicycle models sourced from hundreds of designers. We expect BIKED to enable a variety of data-driven design applications for bicycles and support the development of data-driven design methods. The dataset is comprised of a variety of design information including assembly images, component images, numerical design parameters, and class labels. In this paper, we first discuss the processing of the dataset, then highlight some prominent research questions that BIKED can help address. Of these questions, we further explore the following in detail: 1) How can we explore, understand, and visualize the current design space of bicycles and utilize this information? We apply unsupervised embedding methods to study the design space and identify key takeaways from this analysis. 2) When designing bikes using algorithms, under what conditions can machines understand the design of a given bike? We train a multitude of classifiers to understand designs, then examine the behavior of these classifiers through confusion matrices and permutation-based interpretability analysis. 3) Can machines learn to synthesize new bicycle designs by studying existing ones? We test Variational Autoencoders on random generation, interpolation, and extrapolation tasks after training on BIKED data. The dataset and code are available at http://decode.mit.edu/projects/biked/</jats:p