5 research outputs found
Towards Computing Inferences from English News Headlines
Newspapers are a popular form of written discourse, read by many people,
thanks to the novelty of the information provided by the news content in it. A
headline is the most widely read part of any newspaper due to its appearance in
a bigger font and sometimes in colour print. In this paper, we suggest and
implement a method for computing inferences from English news headlines,
excluding the information from the context in which the headlines appear. This
method attempts to generate the possible assumptions a reader formulates in
mind upon reading a fresh headline. The generated inferences could be useful
for assessing the impact of the news headline on readers including children.
The understandability of the current state of social affairs depends greatly on
the assimilation of the headlines. As the inferences that are independent of
the context depend mainly on the syntax of the headline, dependency trees of
headlines are used in this approach, to find the syntactical structure of the
headlines and to compute inferences out of them.Comment: PACLING 2019 Long paper, 15 page
Beyond Mahalanobis-Based Scores for Textual OOD Detection
Deep learning methods have boosted the adoption of NLP systems in real-life
applications. However, they turn out to be vulnerable to distribution shifts
over time which may cause severe dysfunctions in production systems, urging
practitioners to develop tools to detect out-of-distribution (OOD) samples
through the lens of the neural network. In this paper, we introduce TRUSTED, a
new OOD detector for classifiers based on Transformer architectures that meets
operational requirements: it is unsupervised and fast to compute. The
efficiency of TRUSTED relies on the fruitful idea that all hidden layers carry
relevant information to detect OOD examples. Based on this, for a given input,
TRUSTED consists in (i) aggregating this information and (ii) computing a
similarity score by exploiting the training distribution, leveraging the
powerful concept of data depth. Our extensive numerical experiments involve 51k
model configurations, including various checkpoints, seeds, and datasets, and
demonstrate that TRUSTED achieves state-of-the-art performances. In particular,
it improves previous AUROC over 3 points
Unsupervised Layer-wise Score Aggregation for Textual OOD Detection
Out-of-distribution (OOD) detection is a rapidly growing field due to new
robustness and security requirements driven by an increased number of AI-based
systems. Existing OOD textual detectors often rely on an anomaly score (e.g.,
Mahalanobis distance) computed on the embedding output of the last layer of the
encoder. In this work, we observe that OOD detection performance varies greatly
depending on the task and layer output. More importantly, we show that the
usual choice (the last layer) is rarely the best one for OOD detection and that
far better results could be achieved if the best layer were picked. To leverage
this observation, we propose a data-driven, unsupervised method to combine
layer-wise anomaly scores. In addition, we extend classical textual OOD
benchmarks by including classification tasks with a greater number of classes
(up to 77), which reflects more realistic settings. On this augmented
benchmark, we show that the proposed post-aggregation methods achieve robust
and consistent results while removing manual feature selection altogether.
Their performance achieves near oracle's best layer performance