14,182 research outputs found
Order-Preserving Abstractive Summarization for Spoken Content Based on Connectionist Temporal Classification
Connectionist temporal classification (CTC) is a powerful approach for
sequence-to-sequence learning, and has been popularly used in speech
recognition. The central ideas of CTC include adding a label "blank" during
training. With this mechanism, CTC eliminates the need of segment alignment,
and hence has been applied to various sequence-to-sequence learning problems.
In this work, we applied CTC to abstractive summarization for spoken content.
The "blank" in this case implies the corresponding input data are less
important or noisy; thus it can be ignored. This approach was shown to
outperform the existing methods in term of ROUGE scores over Chinese Gigaword
and MATBN corpora. This approach also has the nice property that the ordering
of words or characters in the input documents can be better preserved in the
generated summaries.Comment: Accepted by Interspeech 201
Bilateral Multi-Perspective Matching for Natural Language Sentences
Natural language sentence matching is a fundamental technology for a variety
of tasks. Previous approaches either match sentences from a single direction or
only apply single granular (word-by-word or sentence-by-sentence) matching. In
this work, we propose a bilateral multi-perspective matching (BiMPM) model
under the "matching-aggregation" framework. Given two sentences and ,
our model first encodes them with a BiLSTM encoder. Next, we match the two
encoded sentences in two directions and . In
each matching direction, each time step of one sentence is matched against all
time-steps of the other sentence from multiple perspectives. Then, another
BiLSTM layer is utilized to aggregate the matching results into a fix-length
matching vector. Finally, based on the matching vector, the decision is made
through a fully connected layer. We evaluate our model on three tasks:
paraphrase identification, natural language inference and answer sentence
selection. Experimental results on standard benchmark datasets show that our
model achieves the state-of-the-art performance on all tasks.Comment: To appear in Proceedings of IJCAI 201
Teaching Machines to Read and Comprehend
Teaching machines to read natural language documents remains an elusive
challenge. Machine reading systems can be tested on their ability to answer
questions posed on the contents of documents that they have seen, but until now
large scale training and test datasets have been missing for this type of
evaluation. In this work we define a new methodology that resolves this
bottleneck and provides large scale supervised reading comprehension data. This
allows us to develop a class of attention based deep neural networks that learn
to read real documents and answer complex questions with minimal prior
knowledge of language structure.Comment: Appears in: Advances in Neural Information Processing Systems 28
(NIPS 2015). 14 pages, 13 figure
Non-Autoregressive Sentence Ordering
Existing sentence ordering approaches generally employ encoder-decoder
frameworks with the pointer net to recover the coherence by recurrently
predicting each sentence step-by-step. Such an autoregressive manner only
leverages unilateral dependencies during decoding and cannot fully explore the
semantic dependency between sentences for ordering. To overcome these
limitations, in this paper, we propose a novel Non-Autoregressive Ordering
Network, dubbed \textit{NAON}, which explores bilateral dependencies between
sentences and predicts the sentence for each position in parallel. We claim
that the non-autoregressive manner is not just applicable but also particularly
suitable to the sentence ordering task because of two peculiar characteristics
of the task: 1) each generation target is in deterministic length, and 2) the
sentences and positions should match exclusively. Furthermore, to address the
repetition issue of the naive non-autoregressive Transformer, we introduce an
exclusive loss to constrain the exclusiveness between positions and sentences.
To verify the effectiveness of the proposed model, we conduct extensive
experiments on several common-used datasets and the experimental results show
that our method outperforms all the autoregressive approaches and yields
competitive performance compared with the state-of-the-arts. The codes are
available at:
\url{https://github.com/steven640pixel/nonautoregressive-sentence-ordering}.Comment: Accepted at Findings of EMNLP202
Language Modeling with Deep Transformers
We explore deep autoregressive Transformer models in language modeling for
speech recognition. We focus on two aspects. First, we revisit Transformer
model configurations specifically for language modeling. We show that well
configured Transformer models outperform our baseline models based on the
shallow stack of LSTM recurrent neural network layers. We carry out experiments
on the open-source LibriSpeech 960hr task, for both 200K vocabulary word-level
and 10K byte-pair encoding subword-level language modeling. We apply our
word-level models to conventional hybrid speech recognition by lattice
rescoring, and the subword-level models to attention based encoder-decoder
models by shallow fusion. Second, we show that deep Transformer language models
do not require positional encoding. The positional encoding is an essential
augmentation for the self-attention mechanism which is invariant to sequence
ordering. However, in autoregressive setup, as is the case for language
modeling, the amount of information increases along the position dimension,
which is a positional signal by its own. The analysis of attention weights
shows that deep autoregressive self-attention models can automatically make use
of such positional information. We find that removing the positional encoding
even slightly improves the performance of these models.Comment: To appear in the proceedings of INTERSPEECH 201
- …