3 research outputs found
Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Information
Sequence-to-sequence (seq2seq) neural models have been actively investigated
for abstractive summarization. Nevertheless, existing neural abstractive
systems frequently generate factually incorrect summaries and are vulnerable to
adversarial information, suggesting a crucial lack of semantic understanding.
In this paper, we propose a novel semantic-aware neural abstractive
summarization model that learns to generate high quality summaries through
semantic interpretation over salient content. A novel evaluation scheme with
adversarial samples is introduced to measure how well a model identifies
off-topic information, where our model yields significantly better performance
than the popular pointer-generator summarizer. Human evaluation also confirms
that our system summaries are uniformly more informative and faithful as well
as less redundant than the seq2seq model
Fill in the BLANC: Human-free quality estimation of document summaries
We present BLANC, a new approach to the automatic estimation of document
summary quality. Our goal is to measure the functional performance of a summary
with an objective, reproducible, and fully automated method. Our approach
achieves this by measuring the performance boost gained by a pre-trained
language model with access to a document summary while carrying out its
language understanding task on the document's text. We present evidence that
BLANC scores have as good correlation with human evaluations as do the ROUGE
family of summary quality measurements. And unlike ROUGE, the BLANC method does
not require human-written reference summaries, allowing for fully human-free
summary quality estimation.Comment: 10 pages, 9 figures, 3 tables. In: Proceedings of the First Workshop
on Evaluation and Comparison of NLP Systems (Eval4NLP, Nov. 2020) p.11-20,
AC
BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization
Most existing text summarization datasets are compiled from the news domain,
where summaries have a flattened discourse structure. In such datasets,
summary-worthy content often appears in the beginning of input articles.
Moreover, large segments from input articles are present verbatim in their
respective summaries. These issues impede the learning and evaluation of
systems that can understand an article's global content structure as well as
produce abstractive summaries with high compression ratio. In this work, we
present a novel dataset, BIGPATENT, consisting of 1.3 million records of U.S.
patent documents along with human written abstractive summaries. Compared to
existing summarization datasets, BIGPATENT has the following properties: i)
summaries contain a richer discourse structure with more recurring entities,
ii) salient content is evenly distributed in the input, and iii) lesser and
shorter extractive fragments are present in the summaries. Finally, we train
and evaluate baselines and popular learning models on BIGPATENT to shed light
on new challenges and motivate future directions for summarization research.Comment: Proceedings of the 57th Annual Meeting of the Association for
Computational Linguistics. ACL 2019 (10 pages