54 research outputs found
Detecting and Mitigating Hallucinations in Multilingual Summarisation
Hallucinations pose a significant challenge to the reliability of neural
models for abstractive summarisation. While automatically generated summaries
may be fluent, they often lack faithfulness to the original document. This
issue becomes even more pronounced in low-resource settings, such as
cross-lingual transfer. With the existing faithful metrics focusing on English,
even measuring the extent of this phenomenon in cross-lingual settings is hard.
To address this, we first develop a novel metric, mFACT, evaluating the
faithfulness of non-English summaries, leveraging translation-based transfer
from multiple English faithfulness metrics. We then propose a simple but
effective method to reduce hallucinations with a cross-lingual transfer, which
weighs the loss of each training example by its faithfulness score. Through
extensive experiments in multiple languages, we demonstrate that mFACT is the
metric that is most suited to detect hallucinations. Moreover, we find that our
proposed loss weighting method drastically increases both performance and
faithfulness according to both automatic and human evaluation when compared to
strong baselines for cross-lingual transfer such as MAD-X. Our code and dataset
are available at https://github.com/yfqiu-nlp/mfact-summ
University of Essex at the TAC 2011 Multilingual Summarisation Pilot
We present the results of our Arabic and English runs at the TAC 2011 Multilingual summarisation (MultiLing) task. We partic- ipated with centroid-based clustering for multi- document summarisation. The automatically generated Arabic and English summaries were evaluated by human participants and by two automatic evaluation metrics, ROUGE and Au- toSummENG. The results are compared with the other systems that participated in the same track on both Arabic and English languages. Our Arabic summariser performed particularly well in the human evaluation
MILDSum: A Novel Benchmark Dataset for Multilingual Summarization of Indian Legal Case Judgments
Automatic summarization of legal case judgments is a practically important
problem that has attracted substantial research efforts in many countries. In
the context of the Indian judiciary, there is an additional complexity --
Indian legal case judgments are mostly written in complex English, but a
significant portion of India's population lacks command of the English
language. Hence, it is crucial to summarize the legal documents in Indian
languages to ensure equitable access to justice. While prior research primarily
focuses on summarizing legal case judgments in their source languages, this
study presents a pioneering effort toward cross-lingual summarization of
English legal documents into Hindi, the most frequently spoken Indian language.
We construct the first high-quality legal corpus comprising of 3,122 case
judgments from prominent Indian courts in English, along with their summaries
in both English and Hindi, drafted by legal practitioners. We benchmark the
performance of several diverse summarization approaches on our corpus and
demonstrate the need for further research in cross-lingual summarization in the
legal domain.Comment: Accepted at EMNLP 2023 (Main Conference
Creating language resources for under-resourced languages: methodologies, and experiments with Arabic
Language resources are important for those working on computational methods to analyse and study languages. These resources are needed to help advancing the research in fields such as natural language processing, machine learning, information retrieval and text analysis in general. We describe the creation of useful resources for languages that currently lack them, taking resources for Arabic summarisation as a case study. We illustrate three different paradigms for creating language resources, namely: (1) using crowdsourcing to produce a small resource rapidly and relatively cheaply; (2) translating an existing gold-standard dataset, which is relatively easy but potentially of lower quality; and (3) using manual effort with appropriately skilled human participants to create a resource that is more expensive but of high quality. The last of these was used as a test collection for TAC-2011. An evaluation of the resources is also presented
Welsh automatic text summarisation
Text summarisation is a digital approach to summarising ‘key’ information contained within texts, and the creation of shortened versions of texts based on this content. Text summarisation function is to provide succinct and coherent summaries to users, something that is often time-consuming and difficult to conduct manually. This is useful in the modern digital world where the creation and sharing of text is ever-increasing, as it enables users to navigate, and make sense of, the dearth of digital information that is available, with ease. This paper reports on work on a project which aims to develop an online Automatic Text Summarisation tool for the Welsh language, ACC (Adnodd Creu Crynodebau). This paper contextualises the need for this text summarisation tool, underlines how a dataset for training and testing the methods was created, and outlines plans for the development of the summariser
PersoNER: Persian named-entity recognition
© 1963-2018 ACL. Named-Entity Recognition (NER) is still a challenging task for languages with low digital resources. The main difficulties arise from the scarcity of annotated corpora and the consequent problematic training of an effective NER pipeline. To abridge this gap, in this paper we target the Persian language that is spoken by a population of over a hundred million people world-wide. We first present and provide ArmanPerosNERCorpus, the first manually-annotated Persian NER corpus. Then, we introduce PersoNER, an NER pipeline for Persian that leverages a word embedding and a sequential max-margin classifier. The experimental results show that the proposed approach is capable of achieving interesting MUC7 and CoNNL scores while outperforming two alternatives based on a CRF and a recurrent neural network
- …