Although large language models (LLMs) have advanced the state-of-the-art in
NLP significantly, deploying them for downstream applications is still
challenging due to cost, responsiveness, control, or concerns around privacy
and security. As such, trainable models are still the preferred option in some
cases. However, these models still require human-labeled data for optimal
performance, which is expensive and time-consuming to obtain. In order to
address this issue, several techniques to reduce human effort involve labeling
or generating data using LLMs. Although these methods are effective for certain
applications, in practice they encounter difficulties in real-world scenarios.
Labeling data requires careful data selection, while generating data
necessitates task-specific prompt engineering. In this paper, we propose a
unified data creation pipeline that requires only a single formatting example,
and which is applicable to a broad range of tasks, including traditionally
problematic ones with semantically devoid label spaces. In our experiments we
demonstrate that instruction-following LLMs are highly cost-effective data
creators, and that models trained with these data exhibit performance better
than those trained with human-labeled data (by up to 17.5%) on
out-of-distribution evaluation, while maintaining comparable performance on
in-distribution tasks. These results have important implications for the
robustness of NLP systems deployed in the real-world.Comment: Accepted to EMNLP 2023 main conference. 12 pages, 5 figures, 6
tables. Code is available at https://github.com/microsoft/llm-data-creatio