1,858 research outputs found
Self-Supervised and Controlled Multi-Document Opinion Summarization
We address the problem of unsupervised abstractive summarization of
collections of user generated reviews with self-supervision and control. We
propose a self-supervised setup that considers an individual document as a
target summary for a set of similar documents. This setting makes training
simpler than previous approaches by relying only on standard log-likelihood
loss. We address the problem of hallucinations through the use of control
codes, to steer the generation towards more coherent and relevant
summaries.Finally, we extend the Transformer architecture to allow for multiple
reviews as input. Our benchmarks on two datasets against graph-based and recent
neural abstractive unsupervised models show that our proposed method generates
summaries with a superior quality and relevance.This is confirmed in our human
evaluation which focuses explicitly on the faithfulness of generated summaries
We also provide an ablation study, which shows the importance of the control
setup in controlling hallucinations and achieve high sentiment and topic
alignment of the summaries with the input reviews.Comment: 18 pages including 5 pages appendi
Neural Natural Language Processing for Long Texts: A Survey of the State-of-the-Art
The adoption of Deep Neural Networks (DNNs) has greatly benefited Natural
Language Processing (NLP) during the past decade. However, the demands of long
document analysis are quite different from those of shorter texts, while the
ever increasing size of documents uploaded on-line renders automated
understanding of long texts a critical area of research. This article has two
goals: a) it overviews the relevant neural building blocks, thus serving as a
short tutorial, and b) it surveys the state-of-the-art in long document NLP,
mainly focusing on two central tasks: document classification and document
summarization. Sentiment analysis for long texts is also covered, since it is
typically treated as a particular case of document classification.
Additionally, this article discusses the main challenges, issues and current
solutions related to long document NLP. Finally, the relevant, publicly
available, annotated datasets are presented, in order to facilitate further
research.Comment: 53 pages, 2 figures, 171 citation
Discourse-Aware Soft Prompting for Text Generation
Current efficient fine-tuning methods (e.g., adapters, prefix-tuning, etc.)
have optimized conditional text generation via training a small set of extra
parameters of the neural language model, while freezing the rest for
efficiency. While showing strong performance on some generation tasks, they
don't generalize across all generation tasks. We show that soft-prompt based
conditional text generation can be improved with simple and efficient methods
that simulate modeling the discourse structure of human written text. We
investigate two design choices: First, we apply \textit{hierarchical blocking}
on the prefix parameters to simulate a higher-level discourse structure of
human written text. Second, we apply \textit{attention sparsity} on the prefix
parameters at different layers of the network and learn sparse transformations
on the softmax-function. We show that structured design of prefix parameters
yields more coherent, faithful and relevant generations than the baseline
prefix-tuning on all generation tasks
- …