Creating an essay based on a few given topics is a challenging NLP task.
Although several effective methods for this problem, topic-to-essay generation,
have appeared recently, there is still much room for improvement, especially in
terms of the coverage of the given topics and the coherence of the generated
text. In this paper, we propose a novel approach called TegFormer which
utilizes the Transformer architecture where the encoder is enriched with
domain-specific contexts while the decoder is enhanced by a large-scale
pre-trained language model. Specifically, a \emph{Topic-Extension} layer
capturing the interaction between the given topics and their domain-specific
contexts is plugged into the encoder. Since the given topics are usually
concise and sparse, such an additional layer can bring more topic-related
semantics in to facilitate the subsequent natural language generation.
Moreover, an \emph{Embedding-Fusion} module that combines the domain-specific
word embeddings learnt from the given corpus and the general-purpose word
embeddings provided by a GPT-2 model pre-trained on massive text data is
integrated into the decoder. Since GPT-2 is at a much larger scale, it contains
a lot more implicit linguistic knowledge which would help the decoder to
produce more grammatical and readable text. Extensive experiments have shown
that the pieces of text generated by TegFormer have better topic coverage and
higher text coherence than those from SOTA topic-to-essay techniques, according
to automatic and human evaluations. As revealed by ablation studies, both the
Topic-Extension layer and the Embedding-Fusion module contribute substantially
to TegFormer's performance advantage