937 research outputs found
Zero-shot Text-to-SQL Learning with Auxiliary Task
Recent years have seen great success in the use of neural seq2seq models on
the text-to-SQL task. However, little work has paid attention to how these
models generalize to realistic unseen data, which naturally raises a question:
does this impressive performance signify a perfect generalization model, or are
there still some limitations?
In this paper, we first diagnose the bottleneck of text-to-SQL task by
providing a new testbed, in which we observe that existing models present poor
generalization ability on rarely-seen data. The above analysis encourages us to
design a simple but effective auxiliary task, which serves as a supportive
model as well as a regularization term to the generation task to increase the
models generalization. Experimentally, We evaluate our models on a large
text-to-SQL dataset WikiSQL. Compared to a strong baseline coarse-to-fine
model, our models improve over the baseline by more than 3% absolute in
accuracy on the whole dataset. More interestingly, on a zero-shot subset test
of WikiSQL, our models achieve 5% absolute accuracy gain over the baseline,
clearly demonstrating its superior generalizability
Guiding the PLMs with Semantic Anchors as Intermediate Supervision: Towards Interpretable Semantic Parsing
The recent prevalence of pretrained language models (PLMs) has dramatically
shifted the paradigm of semantic parsing, where the mapping from natural
language utterances to structured logical forms is now formulated as a Seq2Seq
task. Despite the promising performance, previous PLM-based approaches often
suffer from hallucination problems due to their negligence of the structural
information contained in the sentence, which essentially constitutes the key
semantics of the logical forms. Furthermore, most works treat PLM as a black
box in which the generation process of the target logical form is hidden
beneath the decoder modules, which greatly hinders the model's intrinsic
interpretability. To address these two issues, we propose to incorporate the
current PLMs with a hierarchical decoder network. By taking the first-principle
structures as the semantic anchors, we propose two novel intermediate
supervision tasks, namely Semantic Anchor Extraction and Semantic Anchor
Alignment, for training the hierarchical decoders and probing the model
intermediate representations in a self-adaptive manner alongside the
fine-tuning process. We conduct intensive experiments on several semantic
parsing benchmarks and demonstrate that our approach can consistently
outperform the baselines. More importantly, by analyzing the intermediate
representations of the hierarchical decoders, our approach also makes a huge
step toward the intrinsic interpretability of PLMs in the domain of semantic
parsing
Natural Language Interfaces for Tabular Data Querying and Visualization: A Survey
The emergence of natural language processing has revolutionized the way users
interact with tabular data, enabling a shift from traditional query languages
and manual plotting to more intuitive, language-based interfaces. The rise of
large language models (LLMs) such as ChatGPT and its successors has further
advanced this field, opening new avenues for natural language processing
techniques. This survey presents a comprehensive overview of natural language
interfaces for tabular data querying and visualization, which allow users to
interact with data using natural language queries. We introduce the fundamental
concepts and techniques underlying these interfaces with a particular emphasis
on semantic parsing, the key technology facilitating the translation from
natural language to SQL queries or data visualization commands. We then delve
into the recent advancements in Text-to-SQL and Text-to-Vis problems from the
perspectives of datasets, methodologies, metrics, and system designs. This
includes a deep dive into the influence of LLMs, highlighting their strengths,
limitations, and potential for future improvements. Through this survey, we aim
to provide a roadmap for researchers and practitioners interested in developing
and applying natural language interfaces for data interaction in the era of
large language models.Comment: 20 pages, 4 figures, 5 tables. Submitted to IEEE TKD
Programmable Agents
We build deep RL agents that execute declarative programs expressed in formal
language. The agents learn to ground the terms in this language in their
environment, and can generalize their behavior at test time to execute new
programs that refer to objects that were not referenced during training. The
agents develop disentangled interpretable representations that allow them to
generalize to a wide variety of zero-shot semantic tasks
Semantic Parsing in Limited Resource Conditions
This thesis explores challenges in semantic parsing, specifically focusing on
scenarios with limited data and computational resources. It offers solutions
using techniques like automatic data curation, knowledge transfer, active
learning, and continual learning.
For tasks with no parallel training data, the thesis proposes generating
synthetic training examples from structured database schemas. When there is
abundant data in a source domain but limited parallel data in a target domain,
knowledge from the source is leveraged to improve parsing in the target domain.
For multilingual situations with limited data in the target languages, the
thesis introduces a method to adapt parsers using a limited human translation
budget. Active learning is applied to select source-language samples for manual
translation, maximizing parser performance in the target language. In addition,
an alternative method is also proposed to utilize machine translation services,
supplemented by human-translated data, to train a more effective parser.
When computational resources are limited, a continual learning approach is
introduced to minimize training time and computational memory. This maintains
the parser's efficiency in previously learned tasks while adapting it to new
tasks, mitigating the problem of catastrophic forgetting.
Overall, the thesis provides a comprehensive set of methods to improve
semantic parsing in resource-constrained conditions.Comment: PhD thesis, year of award 2023, 172 page
Reboost Large Language Model-based Text-to-SQL, Text-to-Python, and Text-to-Function -- with Real Applications in Traffic Domain
The previous state-of-the-art (SOTA) method achieved a remarkable execution
accuracy on the Spider dataset, which is one of the largest and most diverse
datasets in the Text-to-SQL domain. However, during our reproduction of the
business dataset, we observed a significant drop in performance. We examined
the differences in dataset complexity, as well as the clarity of questions'
intentions, and assessed how those differences could impact the performance of
prompting methods. Subsequently, We develop a more adaptable and more general
prompting method, involving mainly query rewriting and SQL boosting, which
respectively transform vague information into exact and precise information and
enhance the SQL itself by incorporating execution feedback and the query
results from the database content. In order to prevent information gaps, we
include the comments, value types, and value samples for columns as part of the
database description in the prompt. Our experiments with Large Language Models
(LLMs) illustrate the significant performance improvement on the business
dataset and prove the substantial potential of our method. In terms of
execution accuracy on the business dataset, the SOTA method scored 21.05, while
our approach scored 65.79. As a result, our approach achieved a notable
performance improvement even when using a less capable pre-trained language
model. Last but not least, we also explore the Text-to-Python and
Text-to-Function options, and we deeply analyze the pros and cons among them,
offering valuable insights to the community
Exploring the Compositional Generalization in Context Dependent Text-to-SQL Parsing
In the context-dependent Text-to-SQL task, the generated SQL statements are
refined iteratively based on the user input utterance from each interaction.
The input text from each interaction can be viewed as component modifications
to the previous SQL statements, which could be further extracted as the
modification patterns. Since these modification patterns could also be combined
with other SQL statements, the models are supposed to have the compositional
generalization to these novel combinations. This work is the first exploration
of compositional generalization in context-dependent Text-to-SQL scenarios. To
facilitate related studies, we constructed two challenging benchmarks named
\textsc{CoSQL-CG} and \textsc{SParC-CG} by recombining the modification
patterns and existing SQL statements. The following experiments show that all
current models struggle on our proposed benchmarks. Furthermore, we found that
better aligning the previous SQL statements with the input utterance could give
models better compositional generalization ability. Based on these
observations, we propose a method named \texttt{p-align} to improve the
compositional generalization of Text-to-SQL models. Further experiments
validate the effectiveness of our method. Source code and data are available.Comment: Accepted to ACL 2023 (Findings), Long Paper, 11 page
Few-Shot Semantic Parsing for New Predicates
In this work, we investigate the problems of semantic parsing in a few-shot
learning setting. In this setting, we are provided with utterance-logical form
pairs per new predicate. The state-of-the-art neural semantic parsers achieve
less than 25% accuracy on benchmark datasets when k= 1. To tackle this problem,
we proposed to i) apply a designated meta-learning method to train the model;
ii) regularize attention scores with alignment statistics; iii) apply a
smoothing technique in pre-training. As a result, our method consistently
outperforms all the baselines in both one and two-shot settings.Comment: Accepted to EACL 202
Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing
Cross-lingual semantic parsing transfers parsing capability from a
high-resource language (e.g., English) to low-resource languages with scarce
training data. Previous work has primarily considered silver-standard data
augmentation or zero-shot methods, however, exploiting few-shot gold data is
comparatively unexplored. We propose a new approach to cross-lingual semantic
parsing by explicitly minimizing cross-lingual divergence between probabilistic
latent variables using Optimal Transport. We demonstrate how this direct
guidance improves parsing from natural languages using fewer examples and less
training. We evaluate our method on two datasets, MTOP and MultiATIS++SQL,
establishing state-of-the-art results under a few-shot cross-lingual regime.
Ablation studies further reveal that our method improves performance even
without parallel input translations. In addition, we show that our model better
captures cross-lingual structure in the latent space to improve semantic
representation similarity.Comment: Accepted to TACL 2023. Pre-MIT Press publication. 17 pages, 3
figures, 6 table
- …