37 research outputs found
Efficient Keyphrase Generation with GANs
Keyphrase Generation is the task of predicting keyphrases: short text sequences that convey the main semantic meaning of a document. In this paper, we introduce a keyphrase generation approach that makes use of a Generative Adversarial Networks (GANs) architecture. In our system, the Generator produces a sequence of keyphrases for an input document. The Discriminator, in turn, tries to distinguish between machine generated and human curated keyphrases. We propose a novel Discriminator architecture based on a BERT pretrained model fine-tuned for Sequence Classification. We train our proposed architecture using only a small subset of the standard available training dataset, amounting to less than 1% of the total, achieving a great level of data efficiency. The resulting model is evaluated on five public datasets, obtaining competitive and promising results with respect to four state-of-the-art generative models
Keyphrase Generation with GANs in Low-Resources Scenarios
Keyphrase Generation is the task of predicting Keyphrases (KPs), short phrases that summarize the semantic meaning of a given document.
Several past studies provided diverse approaches to generate Keyphrases for an input document. However, all of these approaches still need to be trained on very large datasets. In this paper, we introduce BeGan-KP, a new conditional GAN model to address the problem of Keyphrase Generation in a low-resource scenario. Our main contribution relies in the Discriminator\u2019s architecture: a new BERT-based module which is able to distinguish between the generated and humancurated KPs reliably. Its characteristics allow us to use it in a low-resource scenario, where only a small amount of training data are available, obtaining an efficient Generator. The resulting architecture achieves, on five public datasets, competitive results with respect to the state-of-the-art approaches, using less than 1% of the training data
Data Augmentation for Low-Resource Keyphrase Generation
Keyphrase generation is the task of summarizing the contents of any given
article into a few salient phrases (or keyphrases). Existing works for the task
mostly rely on large-scale annotated datasets, which are not easy to acquire.
Very few works address the problem of keyphrase generation in low-resource
settings, but they still rely on a lot of additional unlabeled data for
pretraining and on automatic methods for pseudo-annotations. In this paper, we
present data augmentation strategies specifically to address keyphrase
generation in purely resource-constrained domains. We design techniques that
use the full text of the articles to improve both present and absent keyphrase
generation. We test our approach comprehensively on three datasets and show
that the data augmentation strategies consistently improve the state-of-the-art
performance. We release our source code at
https://github.com/kgarg8/kpgen-lowres-data-aug.Comment: 9 pages, 8 tables, To appear at the Findings of the Proceedings of
the 61st Annual Meeting of the Association for Computational Linguistics,
Toronto, Canad
ChatGPT vs State-of-the-Art Models: A Benchmarking Study in Keyphrase Generation Task
Transformer-based language models, including ChatGPT, have demonstrated
exceptional performance in various natural language generation tasks. However,
there has been limited research evaluating ChatGPT's keyphrase generation
ability, which involves identifying informative phrases that accurately reflect
a document's content. This study seeks to address this gap by comparing
ChatGPT's keyphrase generation performance with state-of-the-art models, while
also testing its potential as a solution for two significant challenges in the
field: domain adaptation and keyphrase generation from long documents. We
conducted experiments on six publicly available datasets from scientific
articles and news domains, analyzing performance on both short and long
documents. Our results show that ChatGPT outperforms current state-of-the-art
models in all tested datasets and environments, generating high-quality
keyphrases that adapt well to diverse domains and document lengths
Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models
Keyphrase Generation (KPG) is a longstanding task in NLP with widespread
applications. The advent of sequence-to-sequence (seq2seq) pre-trained language
models (PLMs) has ushered in a transformative era for KPG, yielding promising
performance improvements. However, many design decisions remain unexplored and
are often made arbitrarily. This paper undertakes a systematic analysis of the
influence of model selection and decoding strategies on PLM-based KPG. We begin
by elucidating why seq2seq PLMs are apt for KPG, anchored by an
attention-driven hypothesis. We then establish that conventional wisdom for
selecting seq2seq PLMs lacks depth: (1) merely increasing model size or
performing task-specific adaptation is not parameter-efficient; (2) although
combining in-domain pre-training with task adaptation benefits KPG, it does
partially hinder generalization. Regarding decoding, we demonstrate that while
greedy search achieves strong F1 scores, it lags in recall compared with
sampling-based methods. Based on these insights, we propose DeSel, a
likelihood-based decode-select algorithm for seq2seq PLMs. DeSel improves
greedy search by an average of 4.7% semantic F1 across five datasets. Our
collective findings pave the way for deeper future investigations into
PLM-based KPG.Comment: EMNLP 2023 camera read
KPEval: Towards Fine-grained Semantic-based Evaluation of Keyphrase Extraction and Generation Systems
Despite the significant advancements in keyphrase extraction and keyphrase
generation methods, the predominant approach for evaluation only relies on
exact matching with human references and disregards reference-free attributes.
This scheme fails to recognize systems that generate keyphrases that are
semantically equivalent to the references or keyphrases that have practical
utility. To better understand the strengths and weaknesses of different
keyphrase systems, we propose a comprehensive evaluation framework consisting
of six critical dimensions: naturalness, faithfulness, saliency, coverage,
diversity, and utility. For each dimension, we discuss the desiderata and
design semantic-based metrics that align with the evaluation objectives.
Rigorous meta-evaluation studies demonstrate that our evaluation strategy
correlates better with human preferences compared to a range of previously used
metrics. Using this framework, we re-evaluate 18 keyphrase systems and further
discover that (1) the best model differs in different dimensions, with
pre-trained language models achieving the best in most dimensions; (2) the
utility in downstream tasks does not always correlate well with reference-based
metrics; and (3) large language models exhibit a strong performance in
reference-free evaluation