2 research outputs found
Zero-shot topic generation
We present an approach to generating topics using a model trained only for
document title generation, with zero examples of topics given during training.
We leverage features that capture the relevance of a candidate span in a
document for the generation of a title for that document. The output is a
weighted collection of the phrases that are most relevant for describing the
document and distinguishing it within a corpus, without requiring access to the
rest of the corpus. We conducted a double-blind trial in which human annotators
scored the quality of our machine-generated topics along with original
human-written topics associated with news articles from The Guardian and The
Huffington Post. The results show that our zero-shot model generates topic
labels for news documents that are on average equal to or higher quality than
those written by humans, as judged by humans.Comment: 12 pages, 9 figures, 3 table
GeDi: Generative Discriminator Guided Sequence Generation
Class-conditional language models (CC-LMs) can be used to generate natural
language with specific attributes, such as style or sentiment, by conditioning
on an attribute label, or control code. However, we find that these models
struggle to control generation when applied to out-of-domain prompts or unseen
control codes. To overcome these limitations, we propose generative
discriminator (GeDi) guided contrastive generation, which uses CC-LMs as
generative discriminators (GeDis) to efficiently guide generation from a
(potentially much larger) LM towards a desired attribute. In our human
evaluation experiments, we show that GeDis trained for sentiment control on
movie reviews are able to control the tone of book text. We also demonstrate
that GeDis are able to detoxify generation and control topic while maintaining
the same level of linguistic acceptability as direct generation from GPT-2
(1.5B parameters). Lastly, we show that a GeDi trained on only 4 topics can
generalize to new control codes from word embeddings, allowing it to guide
generation towards wide array of topics