412 research outputs found
On Task-Adaptive Pretraining for Dialogue Response Selection
Recent advancements in dialogue response selection (DRS) are based on the
\textit{task-adaptive pre-training (TAP)} approach, by first initializing their
model with BERT~\cite{devlin-etal-2019-bert}, and adapt to dialogue data with
dialogue-specific or fine-grained pre-training tasks. However, it is uncertain
whether BERT is the best initialization choice, or whether the proposed
dialogue-specific fine-grained learning tasks are actually better than MLM+NSP.
This paper aims to verify assumptions made in previous works and understand the
source of improvements for DRS. We show that initializing with RoBERTa achieve
similar performance as BERT, and MLM+NSP can outperform all previously proposed
TAP tasks, during which we also contribute a new state-of-the-art on the Ubuntu
corpus. Additional analyses shows that the main source of improvements comes
from the TAP step, and that the NSP task is crucial to DRS, different from
common NLU tasks.Comment: 6 pages, 4 figure
Advancing Regular Language Reasoning in Linear Recurrent Neural Networks
In recent studies, linear recurrent neural networks (LRNNs) have achieved
Transformer-level performance in natural language modeling and long-range
modeling while offering rapid parallel training and constant inference costs.
With the resurged interest in LRNNs, we study whether they can learn the hidden
rules in training sequences, such as the grammatical structures of regular
language. We theoretically analyze some existing LRNNs and discover their
limitations on regular language. Motivated by the analysis, we propose a new
LRNN equipped with a block-diagonal and input-dependent transition matrix.
Experiments suggest that the proposed model is the only LRNN that can perform
length extrapolation on regular language tasks such as Sum, Even Pair, and
Modular Arithmetic.Comment: The first two authors contributed equally to this wor
Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation
An ideal length-extrapolatable Transformer language model can handle
sequences longer than the training length without any fine-tuning. Such
long-context utilization capability relies heavily on a flexible positional
embedding design. Upon investigating the flexibility of existing large
pre-trained Transformer language models, we find that the T5 family deserves a
closer look, as its positional embeddings capture rich and flexible attention
patterns. However, T5 suffers from the dispersed attention issue: the longer
the input sequence, the flatter the attention distribution. To alleviate the
issue, we propose two attention alignment strategies via temperature scaling.
Our findings show improvement on the long-context utilization capability of T5
on language modeling, retrieval, multi-document question answering, and code
completion tasks without any fine-tuning. This suggests that a flexible
positional embedding design and attention alignment can go a long way toward
Transformer length extrapolation
PESCO: Prompt-enhanced Self Contrastive Learning for Zero-shot Text Classification
We present PESCO, a novel contrastive learning framework that substantially
improves the performance of zero-shot text classification. We formulate text
classification as a neural text matching problem where each document is treated
as a query, and the system learns the mapping from each query to the relevant
class labels by (1) adding prompts to enhance label matching, and (2) using
retrieved labels to enrich the training set in a self-training loop of
contrastive learning. PESCO achieves state-of-the-art performance on four
benchmark text classification datasets. On DBpedia, we achieve 98.5\% accuracy
without any labeled data, which is close to the fully-supervised result.
Extensive experiments and analyses show all the components of PESCO are
necessary for improving the performance of zero-shot text classification.Comment: accepted by ACL 202
Just Ask:An Interactive Learning Framework for Vision and Language Navigation
In the vision and language navigation task, the agent may encounter ambiguous
situations that are hard to interpret by just relying on visual information and
natural language instructions. We propose an interactive learning framework to
endow the agent with the ability to ask for users' help in such situations. As
part of this framework, we investigate multiple learning approaches for the
agent with different levels of complexity. The simplest model-confusion-based
method lets the agent ask questions based on its confusion, relying on the
predefined confidence threshold of a next action prediction model. To build on
this confusion-based method, the agent is expected to demonstrate more
sophisticated reasoning such that it discovers the timing and locations to
interact with a human. We achieve this goal using reinforcement learning (RL)
with a proposed reward shaping term, which enables the agent to ask questions
only when necessary. The success rate can be boosted by at least 15% with only
one question asked on average during the navigation. Furthermore, we show that
the RL agent is capable of adjusting dynamically to noisy human responses.
Finally, we design a continual learning strategy, which can be viewed as a data
augmentation method, for the agent to improve further utilizing its interaction
history with a human. We demonstrate the proposed strategy is substantially
more realistic and data-efficient compared to previously proposed
pre-exploration techniques.Comment: 8 pages, accepted to AAAI 202
- …