63 research outputs found
On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis
Text preprocessing is often the first step in the pipeline of a Natural
Language Processing (NLP) system, with potential impact in its final
performance. Despite its importance, text preprocessing has not received much
attention in the deep learning literature. In this paper we investigate the
impact of simple text preprocessing decisions (particularly tokenizing,
lemmatizing, lowercasing and multiword grouping) on the performance of a
standard neural text classifier. We perform an extensive evaluation on standard
benchmarks from text categorization and sentiment analysis. While our
experiments show that a simple tokenization of input text is generally
adequate, they also highlight significant degrees of variability across
preprocessing techniques. This reveals the importance of paying attention to
this usually-overlooked step in the pipeline, particularly when comparing
different models. Finally, our evaluation provides insights into the best
preprocessing practices for training word embeddings.Comment: Blackbox EMNLP 2018. 7 page
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning
Over the past years, distributed semantic representations have proved to be
effective and flexible keepers of prior knowledge to be integrated into
downstream applications. This survey focuses on the representation of meaning.
We start from the theoretical background behind word vector space models and
highlight one of their major limitations: the meaning conflation deficiency,
which arises from representing a word with all its possible meanings as a
single vector. Then, we explain how this deficiency can be addressed through a
transition from the word level to the more fine-grained level of word senses
(in its broader acceptation) as a method for modelling unambiguous lexical
meaning. We present a comprehensive overview of the wide range of techniques in
the two main branches of sense representation, i.e., unsupervised and
knowledge-based. Finally, this survey covers the main evaluation procedures and
applications for this type of representation, and provides an analysis of four
of its important aspects: interpretability, sense granularity, adaptability to
different domains and compositionality.Comment: 46 pages, 8 figures. Published in Journal of Artificial Intelligence
Researc
Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference
It has been shown that NLI models are usually biased with respect to the
word-overlap between premise and hypothesis; they take this feature as a
primary cue for predicting the entailment label. In this paper, we focus on an
overlooked aspect of the overlap bias in NLI models: the reverse word-overlap
bias. Our experimental results demonstrate that current NLI models are highly
biased towards the non-entailment label on instances with low overlap, and the
existing debiasing methods, which are reportedly successful on existing
challenge datasets, are generally ineffective in addressing this category of
bias. We investigate the reasons for the emergence of the overlap bias and the
role of minority examples in its mitigation. For the former, we find that the
word-overlap bias does not stem from pre-training, and for the latter, we
observe that in contrast to the accepted assumption, eliminating minority
examples does not affect the generalizability of debiasing methods with respect
to the overlap bias.Comment: Accepted at EMNLP 202
A pragmatic guide to geoparsing evaluation
Abstract: Empirical methods in geoparsing have thus far lacked a standard evaluation framework describing the task, metrics and data used to compare state-of-the-art systems. Evaluation is further made inconsistent, even unrepresentative of real world usage by the lack of distinction between the different types of toponyms, which necessitates new guidelines, a consolidation of metrics and a detailed toponym taxonomy with implications for Named Entity Recognition (NER) and beyond. To address these deficiencies, our manuscript introduces a new framework in three parts. (Part 1) Task Definition: clarified via corpus linguistic analysis proposing a fine-grained Pragmatic Taxonomy of Toponyms. (Part 2) Metrics: discussed and reviewed for a rigorous evaluation including recommendations for NER/Geoparsing practitioners. (Part 3) Evaluation data: shared via a new dataset called GeoWebNews to provide test/train examples and enable immediate use of our contributions. In addition to fine-grained Geotagging and Toponym Resolution (Geocoding), this dataset is also suitable for prototyping and evaluating machine learning NLP models
Recommended from our members
CARD-660: Cambridge rare word dataset - A reliable benchmark for infrequent word representation models
Rare word representation has recently enjoyed a surge of interest, owing to the crucial role that effective handling of infrequent words can play in accurate semantic understanding. However, there is a paucity of reliable benchmarks for evaluation and comparison of these techniques. We show in this paper that the only existing benchmark (the Stanford Rare Word dataset) suffers from low-confidence annotations and limited vocabulary; hence, it does not constitute a solid comparison framework. In order to fill this evaluation gap, we propose CAmbridge Rare word Dataset (CARD-660), an expert-annotated word similarity dataset which provides a highly reliable, yet challenging, benchmark for rare word representation techniques. Through a set of experiments we show that even the best mainstream word embeddings, with millions of words in their vocabularies, are unable to achieve performances higher than 0.43 (Pearson correlation) on the dataset, compared to a human-level upperbound of 0.90. We release the dataset and the annotation materials at https://pilehvar.github.io/card-660/
DiFair: A Benchmark for Disentangled Assessment of Gender Knowledge and Bias
Numerous debiasing techniques have been proposed to mitigate the gender bias
that is prevalent in pretrained language models. These are often evaluated on
datasets that check the extent to which the model is gender-neutral in its
predictions. Importantly, this evaluation protocol overlooks the possible
adverse impact of bias mitigation on useful gender knowledge. To fill this gap,
we propose DiFair, a manually curated dataset based on masked language modeling
objectives. DiFair allows us to introduce a unified metric, gender invariance
score, that not only quantifies a model's biased behavior, but also checks if
useful gender knowledge is preserved. We use DiFair as a benchmark for a number
of widely-used pretained language models and debiasing techniques. Experimental
results corroborate previous findings on the existing gender biases, while also
demonstrating that although debiasing techniques ameliorate the issue of gender
bias, this improvement usually comes at the price of lowering useful gender
knowledge of the model
- …