146 research outputs found
Attention-Based End-to-End Speech Recognition on Voice Search
Recently, there has been a growing interest in end-to-end speech recognition
that directly transcribes speech to text without any predefined alignments. In
this paper, we explore the use of attention-based encoder-decoder model for
Mandarin speech recognition on a voice search task. Previous attempts have
shown that applying attention-based encoder-decoder to Mandarin speech
recognition was quite difficult due to the logographic orthography of Mandarin,
the large vocabulary and the conditional dependency of the attention model. In
this paper, we use character embedding to deal with the large vocabulary.
Several tricks are used for effective model training, including L2
regularization, Gaussian weight noise and frame skipping. We compare two
attention mechanisms and use attention smoothing to cover long context in the
attention model. Taken together, these tricks allow us to finally achieve a
character error rate (CER) of 3.58% and a sentence error rate (SER) of 7.43% on
the MiTV voice search dataset. While together with a trigram language model,
CER and SER reach 2.81% and 5.77%, respectively
An Approximate Bayesian Long Short-Term Memory Algorithm for Outlier Detection
Long Short-Term Memory networks trained with gradient descent and
back-propagation have received great success in various applications. However,
point estimation of the weights of the networks is prone to over-fitting
problems and lacks important uncertainty information associated with the
estimation. However, exact Bayesian neural network methods are intractable and
non-applicable for real-world applications. In this study, we propose an
approximate estimation of the weights uncertainty using Ensemble Kalman Filter,
which is easily scalable to a large number of weights. Furthermore, we optimize
the covariance of the noise distribution in the ensemble update step using
maximum likelihood estimation. To assess the proposed algorithm, we apply it to
outlier detection in five real-world events retrieved from the Twitter
platform
Emulating Human Developmental Stages with Bayesian Neural Networks
We compare the acquisition of knowledge in humans and machines. Research from
the field of developmental psychology indicates, that human-employed hypothesis
are initially guided by simple rules, before evolving into more complex
theories. This observation is shared across many tasks and domains. We
investigate whether stages of development in artificial learning systems are
based on the same characteristics. We operationalize developmental stages as
the size of the data-set, on which the artificial system is trained. For our
analysis we look at the developmental progress of Bayesian Neural Networks on
three different data-sets, including occlusion, support and quantity comparison
tasks. We compare the results with prior research from developmental psychology
and find agreement between the family of optimized models and pattern of
development observed in infants and children on all three tasks, indicating
common principles for the acquisition of knowledge
Deep AutoRegressive Networks
We introduce a deep, generative autoencoder capable of learning hierarchies
of distributed representations from data. Successive deep stochastic hidden
layers are equipped with autoregressive connections, which enable the model to
be sampled from quickly and exactly via ancestral sampling. We derive an
efficient approximate parameter estimation method based on the minimum
description length (MDL) principle, which can be seen as maximising a
variational lower bound on the log-likelihood, with a feedforward neural
network implementing approximate inference. We demonstrate state-of-the-art
generative performance on a number of classic data sets: several UCI data sets,
MNIST and Atari 2600 games.Comment: Appears in Proceedings of the 31st International Conference on
Machine Learning (ICML), Beijing, China, 201
- …