1,363 research outputs found
A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction
The Nonlinear autoregressive exogenous (NARX) model, which predicts the
current value of a time series based upon its previous values as well as the
current and past values of multiple driving (exogenous) series, has been
studied for decades. Despite the fact that various NARX models have been
developed, few of them can capture the long-term temporal dependencies
appropriately and select the relevant driving series to make predictions. In
this paper, we propose a dual-stage attention-based recurrent neural network
(DA-RNN) to address these two issues. In the first stage, we introduce an input
attention mechanism to adaptively extract relevant driving series (a.k.a.,
input features) at each time step by referring to the previous encoder hidden
state. In the second stage, we use a temporal attention mechanism to select
relevant encoder hidden states across all time steps. With this dual-stage
attention scheme, our model can not only make predictions effectively, but can
also be easily interpreted. Thorough empirical studies based upon the SML 2010
dataset and the NASDAQ 100 Stock dataset demonstrate that the DA-RNN can
outperform state-of-the-art methods for time series prediction.Comment: International Joint Conference on Artificial Intelligence (IJCAI),
201
Ti-MAE: Self-Supervised Masked Time Series Autoencoders
Multivariate Time Series forecasting has been an increasingly popular topic
in various applications and scenarios. Recently, contrastive learning and
Transformer-based models have achieved good performance in many long-term
series forecasting tasks. However, there are still several issues in existing
methods. First, the training paradigm of contrastive learning and downstream
prediction tasks are inconsistent, leading to inaccurate prediction results.
Second, existing Transformer-based models which resort to similar patterns in
historical time series data for predicting future values generally induce
severe distribution shift problems, and do not fully leverage the sequence
information compared to self-supervised methods. To address these issues, we
propose a novel framework named Ti-MAE, in which the input time series are
assumed to follow an integrate distribution. In detail, Ti-MAE randomly masks
out embedded time series data and learns an autoencoder to reconstruct them at
the point-level. Ti-MAE adopts mask modeling (rather than contrastive learning)
as the auxiliary task and bridges the connection between existing
representation learning and generative Transformer-based methods, reducing the
difference between upstream and downstream forecasting tasks while maintaining
the utilization of original time series data. Experiments on several public
real-world datasets demonstrate that our framework of masked autoencoding could
learn strong representations directly from the raw data, yielding better
performance in time series forecasting and classification tasks.Comment: 20 pages, 7 figure
Distributional Drift Adaptation with Temporal Conditional Variational Autoencoder for Multivariate Time Series Forecasting
Due to the nonstationary nature, the distribution of real-world multivariate
time series (MTS) changes over time, which is known as distribution drift. Most
existing MTS forecasting models greatly suffer from distribution drift and
degrade the forecasting performance over time. Existing methods address
distribution drift via adapting to the latest arrived data or self-correcting
per the meta knowledge derived from future data. Despite their great success in
MTS forecasting, these methods hardly capture the intrinsic distribution
changes, especially from a distributional perspective. Accordingly, we propose
a novel framework temporal conditional variational autoencoder (TCVAE) to model
the dynamic distributional dependencies over time between historical
observations and future data in MTSs and infer the dependencies as a temporal
conditional distribution to leverage latent variables. Specifically, a novel
temporal Hawkes attention mechanism represents temporal factors subsequently
fed into feed-forward networks to estimate the prior Gaussian distribution of
latent variables. The representation of temporal factors further dynamically
adjusts the structures of Transformer-based encoder and decoder to distribution
changes by leveraging a gated attention mechanism. Moreover, we introduce
conditional continuous normalization flow to transform the prior Gaussian to a
complex and form-free distribution to facilitate flexible inference of the
temporal conditional distribution. Extensive experiments conducted on six
real-world MTS datasets demonstrate the TCVAE's superior robustness and
effectiveness over the state-of-the-art MTS forecasting baselines. We further
illustrate the TCVAE applicability through multifaceted case studies and
visualization in real-world scenarios.Comment: 13 pages, 6 figures, submitted to IEEE Transactions on Neural
Networks and Learning Systems (TNNLS
- …