111 research outputs found
Detecting and Mitigating Hallucinations in Multilingual Summarisation
Hallucinations pose a significant challenge to the reliability of neural
models for abstractive summarisation. While automatically generated summaries
may be fluent, they often lack faithfulness to the original document. This
issue becomes even more pronounced in low-resource settings, such as
cross-lingual transfer. With the existing faithful metrics focusing on English,
even measuring the extent of this phenomenon in cross-lingual settings is hard.
To address this, we first develop a novel metric, mFACT, evaluating the
faithfulness of non-English summaries, leveraging translation-based transfer
from multiple English faithfulness metrics. We then propose a simple but
effective method to reduce hallucinations with a cross-lingual transfer, which
weighs the loss of each training example by its faithfulness score. Through
extensive experiments in multiple languages, we demonstrate that mFACT is the
metric that is most suited to detect hallucinations. Moreover, we find that our
proposed loss weighting method drastically increases both performance and
faithfulness according to both automatic and human evaluation when compared to
strong baselines for cross-lingual transfer such as MAD-X. Our code and dataset
are available at https://github.com/yfqiu-nlp/mfact-summ
Abstractive Multi-Document Summarization based on Semantic Link Network
The key to realize advanced document summarization is semantic representation of documents. This paper investigates the role of Semantic Link Network in representing and understanding documents for multi-document summarization. It proposes a novel abstractive multi-document summarization framework by first transforming documents into a Semantic Link Network of concepts and events and then transforming the Semantic Link Network into the summary of the documents based on the selection of important concepts and events while keeping semantics coherence. Experiments on benchmark datasets show that the proposed summarization approach significantly outperforms relevant state-of-the-art baselines and the Semantic Link Network plays an important role in representing and understanding documents
Recommended from our members
Effective and Efficient Transfer Learning in the Era of Large Language Models
Substantial progress has been made in the field of natural language processing (NLP) due to the advent of large language models (LLMs)—deep neural networks with millions or billions of parameters pre-trained on large amounts of unlabeled data. However, these models have common weaknesses, including degenerate performance in data-scarce scenarios, and substantial computational resource requirements. This thesis aims to develop methods to address these limitations for improved applicability and performance of LLMs in resource-constrained settings with limited data and/or computational resources.
To address the need for labeled data in data-scarce scenarios, I present two methods, in Chapter 2 and Chapter 3, respectively. The first method leverages beneficial relationships between NLP tasks for transfer learning, while the second method combines data augmentation and self-training to boost few-shot learning performance—the ability to perform novel tasks from only a few labeled examples. Additionally, in Chapter 4, I introduce a novel parameter-efficient transfer learning approach that reuses a single frozen model for all tasks while only learning minimal task-specific parameters (soft/continuous prompts) to represent tasks and transfer knowledge. Our method can match or outperform fine-tuning task-specific models (training the whole model on each task). In Chapter 5, I demonstrate the benefits of parameter-efficient transfer learning in a cross-lingual transfer setting. Finally, I conclude the thesis in Chapter 6 by outlining potential avenues for future research that aim to advance NLP through large-scale multi-task learning using multilingual and multimodal data
XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages
Contemporary works on abstractive text summarization have focused primarily
on high-resource languages like English, mostly due to the limited availability
of datasets for low/mid-resource ones. In this work, we present XL-Sum, a
comprehensive and diverse dataset comprising 1 million professionally annotated
article-summary pairs from BBC, extracted using a set of carefully designed
heuristics. The dataset covers 44 languages ranging from low to high-resource,
for many of which no public dataset is currently available. XL-Sum is highly
abstractive, concise, and of high quality, as indicated by human and intrinsic
evaluation. We fine-tune mT5, a state-of-the-art pretrained multilingual model,
with XL-Sum and experiment on multilingual and low-resource summarization
tasks. XL-Sum induces competitive results compared to the ones obtained using
similar monolingual datasets: we show higher than 11 ROUGE-2 scores on 10
languages we benchmark on, with some of them exceeding 15, as obtained by
multilingual training. Additionally, training on low-resource languages
individually also provides competitive performance. To the best of our
knowledge, XL-Sum is the largest abstractive summarization dataset in terms of
the number of samples collected from a single source and the number of
languages covered. We are releasing our dataset and models to encourage future
research on multilingual abstractive summarization. The resources can be found
at \url{https://github.com/csebuetnlp/xl-sum}.Comment: Findings of the Association for Computational Linguistics, ACL 2021
(camera-ready
Creating language resources for under-resourced languages: methodologies, and experiments with Arabic
Language resources are important for those working on computational methods to analyse and study languages. These resources are needed to help advancing the research in fields such as natural language processing, machine learning, information retrieval and text analysis in general. We describe the creation of useful resources for languages that currently lack them, taking resources for Arabic summarisation as a case study. We illustrate three different paradigms for creating language resources, namely: (1) using crowdsourcing to produce a small resource rapidly and relatively cheaply; (2) translating an existing gold-standard dataset, which is relatively easy but potentially of lower quality; and (3) using manual effort with appropriately skilled human participants to create a resource that is more expensive but of high quality. The last of these was used as a test collection for TAC-2011. An evaluation of the resources is also presented
ViCLEVR: A Visual Reasoning Dataset and Hybrid Multimodal Fusion Model for Visual Question Answering in Vietnamese
In recent years, Visual Question Answering (VQA) has gained significant
attention for its diverse applications, including intelligent car assistance,
aiding visually impaired individuals, and document image information retrieval
using natural language queries. VQA requires effective integration of
information from questions and images to generate accurate answers. Neural
models for VQA have made remarkable progress on large-scale datasets, with a
primary focus on resource-rich languages like English. To address this, we
introduce the ViCLEVR dataset, a pioneering collection for evaluating various
visual reasoning capabilities in Vietnamese while mitigating biases. The
dataset comprises over 26,000 images and 30,000 question-answer pairs (QAs),
each question annotated to specify the type of reasoning involved. Leveraging
this dataset, we conduct a comprehensive analysis of contemporary visual
reasoning systems, offering valuable insights into their strengths and
limitations. Furthermore, we present PhoVIT, a comprehensive multimodal fusion
that identifies objects in images based on questions. The architecture
effectively employs transformers to enable simultaneous reasoning over textual
and visual data, merging both modalities at an early model stage. The
experimental findings demonstrate that our proposed model achieves
state-of-the-art performance across four evaluation metrics. The accompanying
code and dataset have been made publicly accessible at
\url{https://github.com/kvt0012/ViCLEVR}. This provision seeks to stimulate
advancements within the research community, fostering the development of more
multimodal fusion algorithms, specifically tailored to address the nuances of
low-resource languages, exemplified by Vietnamese.Comment: A pre-print version and submitted to journa
- …