11,121 research outputs found
Language (Technology) is Power: A Critical Survey of "Bias" in NLP
We survey 146 papers analyzing "bias" in NLP systems, finding that their
motivations are often vague, inconsistent, and lacking in normative reasoning,
despite the fact that analyzing "bias" is an inherently normative process. We
further find that these papers' proposed quantitative techniques for measuring
or mitigating "bias" are poorly matched to their motivations and do not engage
with the relevant literature outside of NLP. Based on these findings, we
describe the beginnings of a path forward by proposing three recommendations
that should guide work analyzing "bias" in NLP systems. These recommendations
rest on a greater recognition of the relationships between language and social
hierarchies, encouraging researchers and practitioners to articulate their
conceptualizations of "bias"---i.e., what kinds of system behaviors are
harmful, in what ways, to whom, and why, as well as the normative reasoning
underlying these statements---and to center work around the lived experiences
of members of communities affected by NLP systems, while interrogating and
reimagining the power relations between technologists and such communities
Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Tasks
State-of-the-art models can perform well in controlled environments, but they
often struggle when presented with out-of-distribution (OOD) examples, making
OOD detection a critical component of NLP systems. In this paper, we focus on
highlighting the limitations of existing approaches to OOD detection in NLP.
Specifically, we evaluated eight OOD detection methods that are easily
integrable into existing NLP systems and require no additional OOD data or
model modifications. One of our contributions is providing a well-structured
research environment that allows for full reproducibility of the results.
Additionally, our analysis shows that existing OOD detection methods for NLP
tasks are not yet sufficiently sensitive to capture all samples characterized
by various types of distributional shifts. Particularly challenging testing
scenarios arise in cases of background shift and randomly shuffled word order
within in domain texts. This highlights the need for future work to develop
more effective OOD detection approaches for the NLP problems, and our work
provides a well-defined foundation for further research in this area.Comment: 11 pages, 3 figures, Association for Computational Linguistic
Hybrid robust deep and shallow semantic processing for creativity support in document production
The research performed in the DeepThought project (http://www.project-deepthought.net) aims at demonstrating the potential of deep linguistic processing if added to existing shallow methods that ensure robustness. Classical information retrieval is extended by high precision concept indexing and relation detection. We use this approach to demonstrate the feasibility of three ambitious applications, one of which is a tool for creativity support in document production and collective brainstorming. This application is described in detail in this paper. Common to all three applications, and the basis for their development is a platform for integrated linguistic processing. This platform is based on a generic software architecture that combines multiple NLP components and on robust minimal recursive semantics (RMRS) as a uniform representation language
Not the Whole Story of the National Literacy Strategy: A Response to Dominic Wyse
There is evidence that the National Literacy Strategy has led to a sustained increase in literacy attainment, especially in reading, although recent international comparisons also suggest some additional issues regarding pupil performance in England. The relative success of the NLS may at least partly lie in the policy application of several complementary areas of educational research, a suggestion disputed by Dominic Wyse (2003). However, his critical commentary is marred by important omissions, particularly of reference to debates about the teaching of reading and to the statutory status of the National Curriculum for English. His alternative suggestions on the use of ‘child development’ evidence lack methodological detail and are only partly formulated
An Overview on Language Models: Recent Developments and Outlook
Language modeling studies the probability distributions over strings of
texts. It is one of the most fundamental tasks in natural language processing
(NLP). It has been widely used in text generation, speech recognition, machine
translation, etc. Conventional language models (CLMs) aim to predict the
probability of linguistic sequences in a causal manner. In contrast,
pre-trained language models (PLMs) cover broader concepts and can be used in
both causal sequential modeling and fine-tuning for downstream applications.
PLMs have their own training paradigms (usually self-supervised) and serve as
foundation models in modern NLP systems. This overview paper provides an
introduction to both CLMs and PLMs from five aspects, i.e., linguistic units,
structures, training methods, evaluation methods, and applications.
Furthermore, we discuss the relationship between CLMs and PLMs and shed light
on the future directions of language modeling in the pre-trained era
- …