259 research outputs found
Fine-Tuning BERT Models for Intent Recognition Using a Frequency Cut-Off Strategy for Domain-Specific Vocabulary Extension
The work leading to these results was supported by the Spanish Ministry of Science and Innovation through the R& D&i projects GOMINOLA (PID2020-118112RB-C21 and PID2020118112RB-C22, funded by MCIN/AEI/10.13039/501100011033), CAVIAR (TEC2017-84593-C2-1-R, funded by MCIN/ AEI/10.13039/501100011033/FEDER "Una manera de hacer Europa"), and AMICPoC (PDC2021-120846-C42, funded by MCIN/AEI/10.13039/501100011033 and by "the European Union "NextGenerationEU/PRTR"). This research also received funding from the European Union's Horizon2020 research and innovation program under grant agreement No 823907 (http://menhirproject.eu, accessed on 2 February 2022). Furthermore, R.K.'s research was supported by the Spanish Ministry of Education (FPI grant PRE2018-083225).Intent recognition is a key component of any task-oriented conversational system. The
intent recognizer can be used first to classify the user’s utterance into one of several predefined classes
(intents) that help to understand the user’s current goal. Then, the most adequate response can be
provided accordingly. Intent recognizers also often appear as a form of joint models for performing
the natural language understanding and dialog management tasks together as a single process, thus
simplifying the set of problems that a conversational system must solve. This happens to be especially
true for frequently asked question (FAQ) conversational systems. In this work, we first present an
exploratory analysis in which different deep learning (DL) models for intent detection and classification
were evaluated. In particular, we experimentally compare and analyze conventional recurrent
neural networks (RNN) and state-of-the-art transformer models. Our experiments confirmed that
best performance is achieved by using transformers. Specifically, best performance was achieved by
fine-tuning the so-called BETO model (a Spanish pretrained bidirectional encoder representations
from transformers (BERT) model from the Universidad de Chile) in our intent detection task. Then, as
the main contribution of the paper, we analyze the effect of inserting unseen domain words to extend
the vocabulary of the model as part of the fine-tuning or domain-adaptation process. Particularly,
a very simple word frequency cut-off strategy is experimentally shown to be a suitable method for
driving the vocabulary learning decisions over unseen words. The results of our analysis show that
the proposed method helps to effectively extend the original vocabulary of the pretrained models.
We validated our approach with a selection of the corpus acquired with the Hispabot-Covid19 system
obtaining satisfactory results.Spanish Ministry of Science and Innovation (MCIN/AEI) PID2020-118112RB-C21
PID2020118112RB-C22
PDC2021-120846-C42Spanish Ministry of Science and Innovation (MCIN/AEI/FEDER "Una manera de hacer Europa") TEC2017-84593-C2-1-RSpanish Ministry of Science and Innovation (European Union "NextGenerationEU/PRTR") PDC2021-120846-C42European Commission 823907German Research Foundation (DFG) PRE2018-08322
Multilingual Name Entity Recognition and Intent Classification Employing Deep Learning Architectures
Named Entity Recognition and Intent Classification are among the most
important subfields of the field of Natural Language Processing. Recent
research has lead to the development of faster, more sophisticated and
efficient models to tackle the problems posed by those two tasks. In this work
we explore the effectiveness of two separate families of Deep Learning networks
for those tasks: Bidirectional Long Short-Term networks and Transformer-based
networks. The models were trained and tested on the ATIS benchmark dataset for
both English and Greek languages. The purpose of this paper is to present a
comparative study of the two groups of networks for both languages and showcase
the results of our experiments. The models, being the current state-of-the-art,
yielded impressive results and achieved high performance.Comment: 24 pages, 5 figures, 11 tables, dataset availabl
A Dynamic Graph Interactive Framework with Label-Semantic Injection for Spoken Language Understanding
Multi-intent detection and slot filling joint models are gaining increasing
traction since they are closer to complicated real-world scenarios. However,
existing approaches (1) focus on identifying implicit correlations between
utterances and one-hot encoded labels in both tasks while ignoring explicit
label characteristics; (2) directly incorporate multi-intent information for
each token, which could lead to incorrect slot prediction due to the
introduction of irrelevant intent. In this paper, we propose a framework termed
DGIF, which first leverages the semantic information of labels to give the
model additional signals and enriched priors. Then, a multi-grain interactive
graph is constructed to model correlations between intents and slots.
Specifically, we propose a novel approach to construct the interactive graph
based on the injection of label semantics, which can automatically update the
graph to better alleviate error propagation. Experimental results show that our
framework significantly outperforms existing approaches, obtaining a relative
improvement of 13.7% over the previous best model on the MixATIS dataset in
overall accuracy.Comment: Submitted to ICASSP 202
End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics
The advances in language-based Artificial Intelligence (AI) technologies
applied to build educational applications can present AI for social-good
opportunities with a broader positive impact. Across many disciplines,
enhancing the quality of mathematics education is crucial in building critical
thinking and problem-solving skills at younger ages. Conversational AI systems
have started maturing to a point where they could play a significant role in
helping students learn fundamental math concepts. This work presents a
task-oriented Spoken Dialogue System (SDS) built to support play-based learning
of basic math concepts for early childhood education. The system has been
evaluated via real-world deployments at school while the students are
practicing early math concepts with multimodal interactions. We discuss our
efforts to improve the SDS pipeline built for math learning, for which we
explore utilizing MathBERT representations for potential enhancement to the
Natural Language Understanding (NLU) module. We perform an end-to-end
evaluation using real-world deployment outputs from the Automatic Speech
Recognition (ASR), Intent Recognition, and Dialogue Manager (DM) components to
understand how error propagation affects the overall performance in real-world
scenarios.Comment: Proceedings of the 1st Workshop on Mathematical Natural Language
Processing (MathNLP) at EMNLP 202
- …