43 research outputs found
Topically Driven Neural Language Model
Language models are typically applied at the sentence level, without access
to the broader document context. We present a neural language model that
incorporates document context in the form of a topic model-like architecture,
thus providing a succinct representation of the broader document context
outside of the current sentence. Experiments over a range of datasets
demonstrate that our model outperforms a pure sentence-based model in terms of
language model perplexity, and leads to topics that are potentially more
coherent than those produced by a standard LDA topic model. Our model also has
the ability to generate related sentences for a topic, providing another way to
interpret topics.Comment: 11 pages, Proceedings of the 55th Annual Meeting of the Association
for Computational Linguistics (ACL 2017) (to appear
Towards Summarizing Multiple Documents with Hierarchical Relationships
Most existing multi-document summarization (MDS) datasets lack
human-generated and genuine (i.e., not synthetic) summaries or source documents
with explicit inter-document relationships that a summary must capture. To
enhance the capabilities of MDS systems we present PeerSum, a novel dataset for
generating meta-reviews of scientific papers, where the meta-reviews are highly
abstractive and genuine summaries of reviews and corresponding discussions.
These source documents have rich inter-document relationships of an explicit
hierarchical structure with cross-references and often feature conflicts. As
there is a scarcity of research that incorporates hierarchical relationships
into MDS systems through attention manipulation on pre-trained language models,
we additionally present Rammer (Relationship-aware Multi-task Meta-review
Generator), a meta-review generation model that uses sparse attention based on
the hierarchical relationships and a multi-task objective that predicts several
metadata features in addition to the standard text generation objective. Our
experimental results show that PeerSum is a challenging dataset, and Rammer
outperforms other strong baseline MDS models under various evaluation metrics.Comment: 10 page
The Next Chapter: A Study of Large Language Models in Storytelling
To enhance the quality of generated stories, recent story generation models
have been investigating the utilization of higher-level attributes like plots
or commonsense knowledge. The application of prompt-based learning with large
language models (LLMs), exemplified by GPT-3, has exhibited remarkable
performance in diverse natural language processing (NLP) tasks. This paper
conducts a comprehensive investigation, utilizing both automatic and human
evaluation, to compare the story generation capacity of LLMs with recent models
across three datasets with variations in style, register, and length of
stories. The results demonstrate that LLMs generate stories of significantly
higher quality compared to other story generation models. Moreover, they
exhibit a level of performance that competes with human authors, albeit with
the preliminary observation that they tend to replicate real stories in
situations involving world knowledge, resembling a form of plagiarism.Comment: Accepted to INLG202