9,049 research outputs found
Leveraging Sentence-level Information with Encoder LSTM for Semantic Slot Filling
Recurrent Neural Network (RNN) and one of its specific architectures, Long
Short-Term Memory (LSTM), have been widely used for sequence labeling. In this
paper, we first enhance LSTM-based sequence labeling to explicitly model label
dependencies. Then we propose another enhancement to incorporate the global
information spanning over the whole input sequence. The latter proposed method,
encoder-labeler LSTM, first encodes the whole input sequence into a fixed
length vector with the encoder LSTM, and then uses this encoded vector as the
initial state of another LSTM for sequence labeling. Combining these methods,
we can predict the label sequence with considering label dependencies and
information of whole input sequence. In the experiments of a slot filling task,
which is an essential component of natural language understanding, with using
the standard ATIS corpus, we achieved the state-of-the-art F1-score of 95.66%.Comment: Accepted in EMNLP 201
Domain Adaptation of Recurrent Neural Networks for Natural Language Understanding
The goal of this paper is to use multi-task learning to efficiently scale
slot filling models for natural language understanding to handle multiple
target tasks or domains. The key to scalability is reducing the amount of
training data needed to learn a model for a new task. The proposed multi-task
model delivers better performance with less data by leveraging patterns that it
learns from the other tasks. The approach supports an open vocabulary, which
allows the models to generalize to unseen words, which is particularly
important when very little training data is used. A newly collected
crowd-sourced data set, covering four different domains, is used to demonstrate
the effectiveness of the domain adaptation and open vocabulary techniques.Comment: Interspeech 201
An Efficient Approach to Encoding Context for Spoken Language Understanding
In task-oriented dialogue systems, spoken language understanding, or SLU,
refers to the task of parsing natural language user utterances into semantic
frames. Making use of context from prior dialogue history holds the key to more
effective SLU. State of the art approaches to SLU use memory networks to encode
context by processing multiple utterances from the dialogue at each turn,
resulting in significant trade-offs between accuracy and computational
efficiency. On the other hand, downstream components like the dialogue state
tracker (DST) already keep track of the dialogue state, which can serve as a
summary of the dialogue history. In this work, we propose an efficient approach
to encoding context from prior utterances for SLU. More specifically, our
architecture includes a separate recurrent neural network (RNN) based encoding
module that accumulates dialogue context to guide the frame parsing sub-tasks
and can be shared between SLU and DST. In our experiments, we demonstrate the
effectiveness of our approach on dialogues from two domains.Comment: Submitted to INTERSPEECH 201
Enhancing Chinese Intent Classification by Dynamically Integrating Character Features into Word Embeddings with Ensemble Techniques
Intent classification has been widely researched on English data with deep
learning approaches that are based on neural networks and word embeddings. The
challenge for Chinese intent classification stems from the fact that, unlike
English where most words are made up of 26 phonologic alphabet letters, Chinese
is logographic, where a Chinese character is a more basic semantic unit that
can be informative and its meaning does not vary too much in contexts. Chinese
word embeddings alone can be inadequate for representing words, and pre-trained
embeddings can suffer from not aligning well with the task at hand. To account
for the inadequacy and leverage Chinese character information, we propose a
low-effort and generic way to dynamically integrate character embedding based
feature maps with word embedding based inputs, whose resulting word-character
embeddings are stacked with a contextual information extraction module to
further incorporate context information for predictions. On top of the proposed
model, we employ an ensemble method to combine single models and obtain the
final result. The approach is data-independent without relying on external
sources like pre-trained word embeddings. The proposed model outperforms
baseline models and existing methods
Seq2Biseq: Bidirectional Output-wise Recurrent Neural Networks for Sequence Modelling
During the last couple of years, Recurrent Neural Networks (RNN) have reached
state-of-the-art performances on most of the sequence modelling problems. In
particular, the "sequence to sequence" model and the neural CRF have proved to
be very effective in this domain. In this article, we propose a new RNN
architecture for sequence labelling, leveraging gated recurrent layers to take
arbitrarily long contexts into account, and using two decoders operating
forward and backward. We compare several variants of the proposed solution and
their performances to the state-of-the-art. Most of our results are better than
the state-of-the-art or very close to it and thanks to the use of recent
technologies, our architecture can scale on corpora larger than those used in
this work.Comment: Slightly improved version of the paper accepted to the CICling 2019
conferenc
Towards end-to-end spoken language understanding
Spoken language understanding system is traditionally designed as a pipeline
of a number of components. First, the audio signal is processed by an automatic
speech recognizer for transcription or n-best hypotheses. With the recognition
results, a natural language understanding system classifies the text to
structured data as domain, intent and slots for down-streaming consumers, such
as dialog system, hands-free applications. These components are usually
developed and optimized independently. In this paper, we present our study on
an end-to-end learning system for spoken language understanding. With this
unified approach, we can infer the semantic meaning directly from audio
features without the intermediate text representation. This study showed that
the trained model can achieve reasonable good result and demonstrated that the
model can capture the semantic attention directly from the audio features.Comment: submitted to ICASSP 201
Combining Word Feature Vector Method with the Convolutional Neural Network for Slot Filling in Spoken Language Understanding
Slot filling is an important problem in Spoken Language Understanding (SLU)
and Natural Language Processing (NLP), which involves identifying a user's
intent and assigning a semantic concept to each word in a sentence. This paper
presents a word feature vector method and combines it into the convolutional
neural network (CNN). We consider 18 word features and each word feature is
constructed by merging similar word labels. By introducing the concept of
external library, we propose a feature set approach that is beneficial for
building the relationship between a word from the training dataset and the
feature. Computational results are reported using the ATIS dataset and
comparisons with traditional CNN as well as bi-directional sequential CNN are
also presented
Sequential Convolutional Neural Networks for Slot Filling in Spoken Language Understanding
We investigate the usage of convolutional neural networks (CNNs) for the slot
filling task in spoken language understanding. We propose a novel CNN
architecture for sequence labeling which takes into account the previous
context words with preserved order information and pays special attention to
the current word with its surrounding context. Moreover, it combines the
information from the past and the future words for classification. Our proposed
CNN architecture outperforms even the previously best ensembling recurrent
neural network model and achieves state-of-the-art results with an F1-score of
95.61% on the ATIS benchmark dataset without using any additional linguistic
knowledge and resources.Comment: Accepted at Interspeech 201
A Survey on Dialogue Systems: Recent Advances and New Frontiers
Dialogue systems have attracted more and more attention. Recent advances on
dialogue systems are overwhelmingly contributed by deep learning techniques,
which have been employed to enhance a wide range of big data applications such
as computer vision, natural language processing, and recommender systems. For
dialogue systems, deep learning can leverage a massive amount of data to learn
meaningful feature representations and response generation strategies, while
requiring a minimum amount of hand-crafting. In this article, we give an
overview to these recent advances on dialogue systems from various perspectives
and discuss some possible research directions. In particular, we generally
divide existing dialogue systems into task-oriented and non-task-oriented
models, then detail how deep learning techniques help them with representative
algorithms and finally discuss some appealing research directions that can
bring the dialogue system research into a new frontier.Comment: 13 pages. arXiv admin note: text overlap with arXiv:1703.01008 by
other author
Multi-Domain Adversarial Learning for Slot Filling in Spoken Language Understanding
The goal of this paper is to learn cross-domain representations for slot
filling task in spoken language understanding (SLU). Most of the recently
published SLU models are domain-specific ones that work on individual task
domains. Annotating data for each individual task domain is both financially
costly and non-scalable. In this work, we propose an adversarial training
method in learning common features and representations that can be shared
across multiple domains. Model that produces such shared representations can be
combined with models trained on individual domain SLU data to reduce the amount
of training samples required for developing a new domain. In our experiments
using data sets from multiple domains, we show that adversarial training helps
in learning better domain-general SLU models, leading to improved slot filling
F1 scores. We further show that applying adversarial learning on domain-general
model also helps in achieving higher slot filling performance when the model is
jointly optimized with domain-specific models
- …