889 research outputs found
Recommended from our members
Mitigating Data Scarcity for Neural Language Models
In recent years, pretrained neural language models (PNLMs) have taken the field of natural language processing by storm, achieving new benchmarks and state-of-theart performances. These models often rely heavily on annotated data, which may not always be available. Data scarcity are commonly found in specialized domains, such as medical, or in low-resource languages that are underexplored by AI research. In this dissertation, we focus on mitigating data scarcity using data augmentation and neural ensemble learning techniques for neural language models. In both research directions, we implement neural network algorithms and evaluate their impact on assisting neural language models in downstream NLP tasks. Specifically, for data augmentation, we explore two techniques: 1) creating positive training data by moving an answer span around its original context and 2) using text simplification techniques to introduce a variety of writing styles to the original training data. Our results indicate that these simple and effective solutions improve the performance of neural language models considerably in low-resource NLP domains and tasks. For neural ensemble learning, we use a multi-label neural classifier to select the best prediction outcome from a variety of individual pretrained neural language models trained for a low-resource medical text simplification task
Textual Entailment Recognition with Semantic Features from Empirical Text Representation
Textual entailment recognition is one of the basic natural language
understanding(NLU) tasks. Understanding the meaning of sentences is a
prerequisite before applying any natural language processing(NLP) techniques to
automatically recognize the textual entailment. A text entails a hypothesis if
and only if the true value of the hypothesis follows the text. Classical
approaches generally utilize the feature value of each word from word embedding
to represent the sentences. In this paper, we propose a novel approach to
identifying the textual entailment relationship between text and hypothesis,
thereby introducing a new semantic feature focusing on empirical
threshold-based semantic text representation. We employ an element-wise
Manhattan distance vector-based feature that can identify the semantic
entailment relationship between the text-hypothesis pair. We carried out
several experiments on a benchmark entailment classification(SICK-RTE) dataset.
We train several machine learning(ML) algorithms applying both semantic and
lexical features to classify the text-hypothesis pair as entailment, neutral,
or contradiction. Our empirical sentence representation technique enriches the
semantic information of the texts and hypotheses found to be more efficient
than the classical ones. In the end, our approach significantly outperforms
known methods in understanding the meaning of the sentences for the textual
entailment classification task.Comment: Pre-print for our paper at International Conference on Speech &
Language Technology for Low-resource Languages (SPELLL'2022
LEAP: Efficient and Automated Test Method for NLP Software
The widespread adoption of DNNs in NLP software has highlighted the need for
robustness. Researchers proposed various automatic testing techniques for
adversarial test cases. However, existing methods suffer from two limitations:
weak error-discovering capabilities, with success rates ranging from 0% to
24.6% for BERT-based NLP software, and time inefficiency, taking 177.8s to
205.28s per test case, making them challenging for time-constrained scenarios.
To address these issues, this paper proposes LEAP, an automated test method
that uses LEvy flight-based Adaptive Particle swarm optimization integrated
with textual features to generate adversarial test cases. Specifically, we
adopt Levy flight for population initialization to increase the diversity of
generated test cases. We also design an inertial weight adaptive update
operator to improve the efficiency of LEAP's global optimization of
high-dimensional text examples and a mutation operator based on the greedy
strategy to reduce the search time. We conducted a series of experiments to
validate LEAP's ability to test NLP software and found that the average success
rate of LEAP in generating adversarial test cases is 79.1%, which is 6.1%
higher than the next best approach (PSOattack). While ensuring high success
rates, LEAP significantly reduces time overhead by up to 147.6s compared to
other heuristic-based methods. Additionally, the experimental results
demonstrate that LEAP can generate more transferable test cases and
significantly enhance the robustness of DNN-based systems.Comment: Accepted at ASE 202
Comparing the production of a formula with the development of L2 competence
This pilot study investigates the production of a formula with the development of L2 competence over proficiency levels of a spoken learner corpus. The results show that the formula
in beginner production data is likely being recalled holistically from learners’ phonological
memory rather than generated online, identifiable by virtue of its fluent production in absence
of any other surface structure evidence of the formula’s syntactic properties. As learners’ L2
competence increases, the formula becomes sensitive to modifications which show structural
conformity at each proficiency level. The transparency between the formula’s modification
and learners’ corresponding L2 surface structure realisations suggest that it is the independent
development of L2 competence which integrates the formula into compositional language,
and ultimately drives the SLA process forward
Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization
Federated learning (FL) is a promising paradigm to enable collaborative model
training with decentralized data. However, the training process of Large
Language Models (LLMs) generally incurs the update of significant parameters,
which limits the applicability of FL techniques to tackle the LLMs in real
scenarios. Prompt tuning can significantly reduce the number of parameters to
update, but it either incurs performance degradation or low training
efficiency. The straightforward utilization of prompt tuning in the FL often
raises non-trivial communication costs and dramatically degrades performance.
In addition, the decentralized data is generally non-Independent and
Identically Distributed (non-IID), which brings client drift problems and thus
poor performance. This paper proposes a Parameter-efficient prompt Tuning
approach with Adaptive Optimization, i.e., FedPepTAO, to enable efficient and
effective FL of LLMs. First, an efficient partial prompt tuning approach is
proposed to improve performance and efficiency simultaneously. Second, a novel
adaptive optimization method is developed to address the client drift problems
on both the device and server sides to enhance performance further. Extensive
experiments based on 10 datasets demonstrate the superb performance (up to
60.8\% in terms of accuracy) and efficiency (up to 97.59\% in terms of training
time) of FedPepTAO compared with 9 baseline approaches. Our code is available
at https://github.com/llm-eff/FedPepTAO.Comment: 18 pages, accepted by EMNLP 202
Paraphrase Types for Generation and Detection
Current approaches in paraphrase generation and detection heavily rely on a
single general similarity score, ignoring the intricate linguistic properties
of language. This paper introduces two new tasks to address this shortcoming by
considering paraphrase types - specific linguistic perturbations at particular
text positions. We name these tasks Paraphrase Type Generation and Paraphrase
Type Detection. Our results suggest that while current techniques perform well
in a binary classification scenario, i.e., paraphrased or not, the inclusion of
fine-grained paraphrase types poses a significant challenge. While most
approaches are good at generating and detecting general semantic similar
content, they fail to understand the intrinsic linguistic variables they
manipulate. Models trained in generating and identifying paraphrase types also
show improvements in tasks without them. In addition, scaling these models
further improves their ability to understand paraphrase types. We believe
paraphrase types can unlock a new paradigm for developing paraphrase models and
solving tasks in the future.Comment: Published at EMNLP 202
Text Fact Transfer
Text style transfer is a prominent task that aims to control the style of
text without inherently changing its factual content. To cover more text
modification applications, such as adapting past news for current events and
repurposing educational materials, we propose the task of text fact transfer,
which seeks to transfer the factual content of a source text between topics
without modifying its style. We find that existing language models struggle
with text fact transfer, due to their inability to preserve the specificity and
phrasing of the source text, and tendency to hallucinate errors. To address
these issues, we design ModQGA, a framework that minimally modifies a source
text with a novel combination of end-to-end question generation and
specificity-aware question answering. Through experiments on four existing
datasets adapted for text fact transfer, we show that ModQGA can accurately
transfer factual content without sacrificing the style of the source text.Comment: Accepted to EMNLP 2023 Main Conferenc
QA-NatVer: Question Answering for Natural Logic-based Fact Verification
Fact verification systems assess a claim's veracity based on evidence. An
important consideration in designing them is faithfulness, i.e. generating
explanations that accurately reflect the reasoning of the model. Recent works
have focused on natural logic, which operates directly on natural language by
capturing the semantic relation of spans between an aligned claim with its
evidence via set-theoretic operators. However, these approaches rely on
substantial resources for training, which are only available for high-resource
languages. To this end, we propose to use question answering to predict natural
logic operators, taking advantage of the generalization capabilities of
instruction-tuned language models. Thus, we obviate the need for annotated
training data while still relying on a deterministic inference system. In a
few-shot setting on FEVER, our approach outperforms the best baseline by
accuracy points, including a state-of-the-art pre-trained seq2seq natural logic
system, as well as a state-of-the-art prompt-based classifier. Our system
demonstrates its robustness and portability, achieving competitive performance
on a counterfactual dataset and surpassing all approaches without further
annotation on a Danish verification dataset. A human evaluation indicates that
our approach produces more plausible proofs with fewer erroneous natural logic
operators than previous natural logic-based systems.Comment: EMNLP 202
- …