566 research outputs found
Learning Long-Term Dependencies is Not as Difficult With NARX Recurrent Neural Networks
It has recently been shown that gradient descent learning algorithms for
recurrent neural networks can perform poorly on tasks that involve long-
term dependencies, i.e. those problems for which the desired output
depends on inputs presented at times far in the past.
In this paper we explore the long-term dependencies problem for a class of
architectures called NARX recurrent neural networks, which have power
ful representational capabilities. We have previously reported that gradient
descent learning is more effective in NARX networks than in recurrent
neural network architectures that have ``hidden states'' on problems includ
ing grammatical inference and nonlinear system identification. Typically,
the network converges much faster and generalizes better than other net
works. The results in this paper are an attempt to explain this phenomenon.
We present some experimental results which show that NARX networks
can often retain information for two to three times as long as conventional
recurrent neural networks. We show that although NARX networks do not
circumvent the problem of long-term dependencies, they can greatly
improve performance on long-term dependency problems.
We also describe in detail some of the assumption regarding what it means
to latch information robustly and suggest possible ways to loosen these
assumptions.
(Also cross-referenced as UMIACS-TR-95-78
A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction
The Nonlinear autoregressive exogenous (NARX) model, which predicts the
current value of a time series based upon its previous values as well as the
current and past values of multiple driving (exogenous) series, has been
studied for decades. Despite the fact that various NARX models have been
developed, few of them can capture the long-term temporal dependencies
appropriately and select the relevant driving series to make predictions. In
this paper, we propose a dual-stage attention-based recurrent neural network
(DA-RNN) to address these two issues. In the first stage, we introduce an input
attention mechanism to adaptively extract relevant driving series (a.k.a.,
input features) at each time step by referring to the previous encoder hidden
state. In the second stage, we use a temporal attention mechanism to select
relevant encoder hidden states across all time steps. With this dual-stage
attention scheme, our model can not only make predictions effectively, but can
also be easily interpreted. Thorough empirical studies based upon the SML 2010
dataset and the NASDAQ 100 Stock dataset demonstrate that the DA-RNN can
outperform state-of-the-art methods for time series prediction.Comment: International Joint Conference on Artificial Intelligence (IJCAI),
201
Behavioural pattern identification and prediction in intelligent environments
In this paper, the application of soft computing techniques in prediction of an occupant's behaviour in an inhabited intelligent environment is addressed. In this research, daily activities of elderly people who live in their own homes suffering from dementia are studied. Occupancy sensors are used to extract the movement patterns of the occupant. The occupancy data is then converted into temporal sequences of activities which are eventually used to predict the occupant behaviour. To build the prediction model, different dynamic recurrent neural networks are investigated. Recurrent neural networks have shown a great ability in finding the temporal relationships of input patterns. The experimental results show that non-linear autoregressive network with exogenous inputs model correctly extracts the long term prediction patterns of the occupant and outperformed the Elman network. The results presented here are validated using data generated from a simulator and real environments
Energy rating of a water pumping station using multivariate analysis
Among water management policies, the preservation and the saving of energy demand in water supply and treatment systems play key roles. When focusing on energy, the customary metric to determine the performance of water supply systems is linked to the definition of component-based energy indicators. This approach is unfit to account for interactions occurring among system elements or between the system and its environment. On the other hand, the development of information technology has led to the availability of increasing large amount of data, typically gathered from distributed sensor networks in so-called smart grids. In this context, data intensive methodologies address the possibility of using complex network modeling approaches, and advocate the issues related to the interpretation and analysis of large amount of data produced by smart sensor networks.
In this perspective, the present work aims to use data intensive techniques in the energy analysis of a water management network.
The purpose is to provide new metrics for the energy rating of the system and to be able to provide insights into the dynamics of its operations. The study applies neural network as a tool to predict energy demand, when using flowrate and vibration data as predictor variables
A Comparative Study of Reservoir Computing for Temporal Signal Processing
Reservoir computing (RC) is a novel approach to time series prediction using
recurrent neural networks. In RC, an input signal perturbs the intrinsic
dynamics of a medium called a reservoir. A readout layer is then trained to
reconstruct a target output from the reservoir's state. The multitude of RC
architectures and evaluation metrics poses a challenge to both practitioners
and theorists who study the task-solving performance and computational power of
RC. In addition, in contrast to traditional computation models, the reservoir
is a dynamical system in which computation and memory are inseparable, and
therefore hard to analyze. Here, we compare echo state networks (ESN), a
popular RC architecture, with tapped-delay lines (DL) and nonlinear
autoregressive exogenous (NARX) networks, which we use to model systems with
limited computation and limited memory respectively. We compare the performance
of the three systems while computing three common benchmark time series:
H{\'e}non Map, NARMA10, and NARMA20. We find that the role of the reservoir in
the reservoir computing paradigm goes beyond providing a memory of the past
inputs. The DL and the NARX network have higher memorization capability, but
fall short of the generalization power of the ESN
- …