10,813 research outputs found
Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection
Linguistically diverse datasets are critical for training and evaluating
robust machine learning systems, but data collection is a costly process that
often requires experts. Crowdsourcing the process of paraphrase generation is
an effective means of expanding natural language datasets, but there has been
limited analysis of the trade-offs that arise when designing tasks. In this
paper, we present the first systematic study of the key factors in
crowdsourcing paraphrase collection. We consider variations in instructions,
incentives, data domains, and workflows. We manually analyzed paraphrases for
correctness, grammaticality, and linguistic diversity. Our observations provide
new insight into the trade-offs between accuracy and diversity in crowd
responses that arise as a result of task design, providing guidance for future
paraphrase generation procedures.Comment: Published at ACL 201
Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training
We consider the task of few-shot intent detection, which involves training a
deep learning model to classify utterances based on their underlying intents
using only a small amount of labeled data. The current approach to address this
problem is through continual pre-training, i.e., fine-tuning pre-trained
language models (PLMs) on external resources (e.g., conversational corpora,
public intent detection datasets, or natural language understanding datasets)
before using them as utterance encoders for training an intent classifier. In
this paper, we show that continual pre-training may not be essential, since the
overfitting problem of PLMs on this task may not be as serious as expected.
Specifically, we find that directly fine-tuning PLMs on only a handful of
labeled examples already yields decent results compared to methods that employ
continual pre-training, and the performance gap diminishes rapidly as the
number of labeled data increases. To maximize the utilization of the limited
available data, we propose a context augmentation method and leverage
sequential self-distillation to boost performance. Comprehensive experiments on
real-world benchmarks show that given only two or more labeled samples per
class, direct fine-tuning outperforms many strong baselines that utilize
external data sources for continual pre-training. The code can be found at
https://github.com/hdzhang-code/DFTPlus.Comment: ACL 2023, Finding
Weakly Supervised Reasoning by Neuro-Symbolic Approaches
Deep learning has largely improved the performance of various natural
language processing (NLP) tasks. However, most deep learning models are
black-box machinery, and lack explicit interpretation. In this chapter, we will
introduce our recent progress on neuro-symbolic approaches to NLP, which
combines different schools of AI, namely, symbolism and connectionism.
Generally, we will design a neural system with symbolic latent structures for
an NLP task, and apply reinforcement learning or its relaxation to perform
weakly supervised reasoning in the downstream task. Our framework has been
successfully applied to various tasks, including table query reasoning,
syntactic structure reasoning, information extraction reasoning, and rule
reasoning. For each application, we will introduce the background, our
approach, and experimental results.Comment: Compendium of Neurosymbolic Artificial Intelligence, 665--692, 2023,
IOS Pres
Answering Complex Questions by Joining Multi-Document Evidence with Quasi Knowledge Graphs
Direct answering of questions that involve multiple entities and relations is a challenge for text-based QA. This problem is most pronounced when answers can be found only by joining evidence from multiple documents. Curated knowledge graphs (KGs) may yield good answers, but are limited by their inherent incompleteness and potential staleness. This paper presents QUEST, a method that can answer complex questions directly from textual sources on-the-fly, by computing similarity joins over partial results from different documents. Our method is completely unsupervised, avoiding training-data bottlenecks and being able to cope with rapidly evolving ad hoc topics and formulation style in user questions. QUEST builds a noisy quasi KG with node and edge weights, consisting of dynamically retrieved entity names and relational phrases. It augments this graph with types and semantic alignments, and computes the best answers by an algorithm for Group Steiner Trees. We evaluate QUEST on benchmarks of complex questions, and show that it substantially outperforms state-of-the-art baselines
- …