Concept generation is a creative step in the conceptual design phase, where
designers often turn to brainstorming, mindmapping, or crowdsourcing design
ideas to complement their own knowledge of the domain. Recent advances in
natural language processing (NLP) and machine learning (ML) have led to the
rise of Large Language Models (LLMs) capable of generating seemingly creative
outputs from textual prompts. The success of these models has led to their
integration and application across a variety of domains, including art,
entertainment, and other creative work. In this paper, we leverage LLMs to
generate solutions for a set of 12 design problems and compare them to a
baseline of crowdsourced solutions. We evaluate the differences between
generated and crowdsourced design solutions through multiple perspectives,
including human expert evaluations and computational metrics. Expert
evaluations indicate that the LLM-generated solutions have higher average
feasibility and usefulness while the crowdsourced solutions have more novelty.
We experiment with prompt engineering and find that leveraging few-shot
learning can lead to the generation of solutions that are more similar to the
crowdsourced solutions. These findings provide insight into the quality of
design solutions generated with LLMs and begins to evaluate prompt engineering
techniques that could be leveraged by practitioners to generate higher-quality
design solutions synergistically with LLMs.Comment: Proceedings of the ASME 2023 International Design Engineering
Technical Conferences and Computers and Information in Engineering
Conference