2,358 research outputs found
Text content and task performance in the evaluation of a natural language generation system
An important question in the evaluation of Natural Language Generation systems concerns the relationship between textual characteristics and task performance. If the results of task-based evaluation can be correlated to properties of the text, there are better prospects for improving the system. The present paper investigates this relationship by focusing on the outcomes of a task-based evaluation of a system that generates summaries of patient data, attempting to correlate these with the results of an analysis of the system’s texts, compared to a set of gold standard human-authored summaries.peer-reviewe
Evaluating algorithms for the generation of referring expressions : going beyond toy domains
We describe a corpus-based evaluation methodology, applied to a number of classic algorithms in the generation of referring expressions. Following up on earlier work involving very simple domains, this paper deals with the issues associated with domains that contain ‘real-life’ objects of some complexity. Results indicate that state of the art algorithms perform very differently when applied to a complex domain. Moreover, if a version of the Incremental Algorithm is used then it becomes of huge importance to select a good preference order. These results should contribute to a growing debate on the evaluation of nlg systems, arguing in favour of carefully constructed balanced and semantically transparent corpora.peer-reviewe
Automatic Identification of AltLexes using Monolingual Parallel Corpora
The automatic identification of discourse relations is still a challenging
task in natural language processing. Discourse connectives, such as "since" or
"but", are the most informative cues to identify explicit relations; however
discourse parsers typically use a closed inventory of such connectives. As a
result, discourse relations signaled by markers outside these inventories (i.e.
AltLexes) are not detected as effectively. In this paper, we propose a novel
method to leverage parallel corpora in text simplification and lexical
resources to automatically identify alternative lexicalizations that signal
discourse relation. When applied to the Simple Wikipedia and Newsela corpora
along with WordNet and the PPDB, the method allowed the automatic discovery of
91 AltLexes.Comment: 6 pages, Proceedings of Recent Advances in Natural Language
Processing (RANLP 2017
Exploring the Landscape of Natural Language Processing Research
As an efficient approach to understand, generate, and process natural
language texts, research in natural language processing (NLP) has exhibited a
rapid spread and wide adoption in recent years. Given the increasing amount of
research work in this area, several NLP-related approaches have been surveyed
in the research community. However, a comprehensive study that categorizes
established topics, identifies trends, and outlines areas for future research
remains absent to this day. Contributing to closing this gap, we have
systematically classified and analyzed research papers included in the ACL
Anthology. As a result, we present a structured overview of the research
landscape, provide a taxonomy of fields-of-study in NLP, analyze recent
developments in NLP, summarize our findings, and highlight directions for
future work.Comment: Accepted to the 14th International Conference on Recent Advances in
Natural Language Processing (RANLP 2023
Detecting Hate Speech in Social Media
In this paper we examine methods to detect hate speech in social media, while
distinguishing this from general profanity. We aim to establish lexical
baselines for this task by applying supervised classification methods using a
recently released dataset annotated for this purpose. As features, our system
uses character n-grams, word n-grams and word skip-grams. We obtain results of
78% accuracy in identifying posts across three classes. Results demonstrate
that the main challenge lies in discriminating profanity and hate speech from
each other. A number of directions for future work are discussed.Comment: Proceedings of Recent Advances in Natural Language Processing
(RANLP). pp. 467-472. Varna, Bulgari
Шестая международная конференция RANLP-2007 (Recent Advances in Natural Language Processing)
Текст аннотации отсутствует
Online Deception Detection Refueled by Real World Data Collection
The lack of large realistic datasets presents a bottleneck in online
deception detection studies. In this paper, we apply a data collection method
based on social network analysis to quickly identify high-quality deceptive and
truthful online reviews from Amazon. The dataset contains more than 10,000
deceptive reviews and is diverse in product domains and reviewers. Using this
dataset, we explore effective general features for online deception detection
that perform well across domains. We demonstrate that with generalized features
- advertising speak and writing complexity scores - deception detection
performance can be further improved by adding additional deceptive reviews from
assorted domains in training. Finally, reviewer level evaluation gives an
interesting insight into different deceptive reviewers' writing styles.Comment: 10 pages, Accepted to Recent Advances in Natural Language Processing
(RANLP) 201
- …