7,985 research outputs found
Learning to Adaptively Scale Recurrent Neural Networks
Recent advancements in recurrent neural network (RNN) research have
demonstrated the superiority of utilizing multiscale structures in learning
temporal representations of time series. Currently, most of multiscale RNNs use
fixed scales, which do not comply with the nature of dynamical temporal
patterns among sequences. In this paper, we propose Adaptively Scaled Recurrent
Neural Networks (ASRNN), a simple but efficient way to handle this problem.
Instead of using predefined scales, ASRNNs are able to learn and adjust scales
based on different temporal contexts, making them more flexible in modeling
multiscale patterns. Compared with other multiscale RNNs, ASRNNs are bestowed
upon dynamical scaling capabilities with much simpler structures, and are easy
to be integrated with various RNN cells. The experiments on multiple sequence
modeling tasks indicate ASRNNs can efficiently adapt scales based on different
sequence contexts and yield better performances than baselines without
dynamical scaling abilities
A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction
The Nonlinear autoregressive exogenous (NARX) model, which predicts the
current value of a time series based upon its previous values as well as the
current and past values of multiple driving (exogenous) series, has been
studied for decades. Despite the fact that various NARX models have been
developed, few of them can capture the long-term temporal dependencies
appropriately and select the relevant driving series to make predictions. In
this paper, we propose a dual-stage attention-based recurrent neural network
(DA-RNN) to address these two issues. In the first stage, we introduce an input
attention mechanism to adaptively extract relevant driving series (a.k.a.,
input features) at each time step by referring to the previous encoder hidden
state. In the second stage, we use a temporal attention mechanism to select
relevant encoder hidden states across all time steps. With this dual-stage
attention scheme, our model can not only make predictions effectively, but can
also be easily interpreted. Thorough empirical studies based upon the SML 2010
dataset and the NASDAQ 100 Stock dataset demonstrate that the DA-RNN can
outperform state-of-the-art methods for time series prediction.Comment: International Joint Conference on Artificial Intelligence (IJCAI),
201
Self-Adaptive Hierarchical Sentence Model
The ability to accurately model a sentence at varying stages (e.g.,
word-phrase-sentence) plays a central role in natural language processing. As
an effort towards this goal we propose a self-adaptive hierarchical sentence
model (AdaSent). AdaSent effectively forms a hierarchy of representations from
words to phrases and then to sentences through recursive gated local
composition of adjacent segments. We design a competitive mechanism (through
gating networks) to allow the representations of the same sentence to be
engaged in a particular learning task (e.g., classification), therefore
effectively mitigating the gradient vanishing problem persistent in other
recursive models. Both qualitative and quantitative analysis shows that AdaSent
can automatically form and select the representations suitable for the task at
hand during training, yielding superior classification performance over
competitor models on 5 benchmark data sets.Comment: 8 pages, 7 figures, accepted as a full paper at IJCAI 201
- …