One of the most impressive results of recent NLP history is the ability of
pre-trained language models to solve new tasks in a zero-shot setting. To
achieve this, NLP tasks are framed as natural language prompts, generating a
response indicating the predicted output. Nonetheless, the performance in such
settings often lags far behind its supervised counterpart, suggesting a large
space for potential improvement. In this paper, we explore methods to utilize
unlabeled data to improve zero-shot performance. Specifically, we take
advantage of the fact that multiple prompts can be used to specify a single
task, and propose to regularize prompt consistency, encouraging consistent
predictions over this diverse set of prompts. Our method makes it possible to
fine-tune the model either with extra unlabeled training data, or directly on
test input at inference time in an unsupervised manner. In experiments, our
approach outperforms the state-of-the-art zero-shot learner, T0 (Sanh et al.,
2022), on 9 out of 11 datasets across 4 NLP tasks by up to 10.6 absolute points
in terms of accuracy. The gains are often attained with a small number of
unlabeled examples.Comment: Preprint. Code is available at
https://github.com/violet-zct/swarm-distillation-zero-sho