136 research outputs found
On Tree-Based Neural Sentence Modeling
Neural networks with tree-based sentence encoders have shown better results
on many downstream tasks. Most of existing tree-based encoders adopt syntactic
parsing trees as the explicit structure prior. To study the effectiveness of
different tree structures, we replace the parsing trees with trivial trees
(i.e., binary balanced tree, left-branching tree and right-branching tree) in
the encoders. Though trivial trees contain no syntactic information, those
encoders get competitive or even better results on all of the ten downstream
tasks we investigated. This surprising result indicates that explicit syntax
guidance may not be the main contributor to the superior performances of
tree-based neural sentence modeling. Further analysis show that tree modeling
gives better results when crucial words are closer to the final representation.
Additional experiments give more clues on how to design an effective tree-based
encoder. Our code is open-source and available at
https://github.com/ExplorerFreda/TreeEnc.Comment: To Appear at EMNLP 201
Dialogue Act Recognition via CRF-Attentive Structured Network
Dialogue Act Recognition (DAR) is a challenging problem in dialogue
interpretation, which aims to attach semantic labels to utterances and
characterize the speaker's intention. Currently, many existing approaches
formulate the DAR problem ranging from multi-classification to structured
prediction, which suffer from handcrafted feature extensions and attentive
contextual structural dependencies. In this paper, we consider the problem of
DAR from the viewpoint of extending richer Conditional Random Field (CRF)
structural dependencies without abandoning end-to-end training. We incorporate
hierarchical semantic inference with memory mechanism on the utterance
modeling. We then extend structured attention network to the linear-chain
conditional random field layer which takes into account both contextual
utterances and corresponding dialogue acts. The extensive experiments on two
major benchmark datasets Switchboard Dialogue Act (SWDA) and Meeting Recorder
Dialogue Act (MRDA) datasets show that our method achieves better performance
than other state-of-the-art solutions to the problem. It is a remarkable fact
that our method is nearly close to the human annotator's performance on SWDA
within 2% gap.Comment: 10 pages, 4figure
Backpropagating through Structured Argmax using a SPIGOT
We introduce the structured projection of intermediate gradients optimization
technique (SPIGOT), a new method for backpropagating through neural networks
that include hard-decision structured predictions (e.g., parsing) in
intermediate layers. SPIGOT requires no marginal inference, unlike structured
attention networks (Kim et al., 2017) and some reinforcement learning-inspired
solutions (Yogatama et al., 2017). Like so-called straight-through estimators
(Hinton, 2012), SPIGOT defines gradient-like quantities associated with
intermediate nondifferentiable operations, allowing backpropagation before and
after them; SPIGOT's proxy aims to ensure that, after a parameter update, the
intermediate structure will remain well-formed.
We experiment on two structured NLP pipelines: syntactic-then-semantic
dependency parsing, and semantic parsing followed by sentiment classification.
We show that training with SPIGOT leads to a larger improvement on the
downstream task than a modularly-trained pipeline, the straight-through
estimator, and structured attention, reaching a new state of the art on
semantic dependency parsing.Comment: ACL 201
Character-level Intra Attention Network for Natural Language Inference
Natural language inference (NLI) is a central problem in language
understanding. End-to-end artificial neural networks have reached
state-of-the-art performance in NLI field recently.
In this paper, we propose Character-level Intra Attention Network (CIAN) for
the NLI task. In our model, we use the character-level convolutional network to
replace the standard word embedding layer, and we use the intra attention to
capture the intra-sentence semantics. The proposed CIAN model provides improved
results based on a newly published MNLI corpus.Comment: EMNLP Workshop RepEval 2017: The Second Workshop on Evaluating Vector
Space Representations for NL
Towards Neural Machine Translation with Latent Tree Attention
Building models that take advantage of the hierarchical structure of language
without a priori annotation is a longstanding goal in natural language
processing. We introduce such a model for the task of machine translation,
pairing a recurrent neural network grammar encoder with a novel attentional
RNNG decoder and applying policy gradient reinforcement learning to induce
unsupervised tree structures on both the source and target. When trained on
character-level datasets with no explicit segmentation or parse annotation, the
model learns a plausible segmentation and shallow parse, obtaining performance
close to an attentional baseline.Comment: Presented at SPNLP 201
Forward Attention in Sequence-to-sequence Acoustic Modelling for Speech Synthesis
This paper proposes a forward attention method for the sequenceto- sequence
acoustic modeling of speech synthesis. This method is motivated by the nature
of the monotonic alignment from phone sequences to acoustic sequences. Only the
alignment paths that satisfy the monotonic condition are taken into
consideration at each decoder timestep. The modified attention probabilities at
each timestep are computed recursively using a forward algorithm. A transition
agent for forward attention is further proposed, which helps the attention
mechanism to make decisions whether to move forward or stay at each decoder
timestep. Experimental results show that the proposed forward attention method
achieves faster convergence speed and higher stability than the baseline
attention method. Besides, the method of forward attention with transition
agent can also help improve the naturalness of synthetic speech and control the
speed of synthetic speech effectively.Comment: 5 pages, 3 figures, 2 tables. Published in IEEE International
Conference on Acoustics, Speech and Signal Processing 2018 (ICASSP2018
- …