10 research outputs found
The Power of Linear Recurrent Neural Networks
Recurrent neural networks are a powerful means to cope with time series. We
show how a type of linearly activated recurrent neural networks, which we call
predictive neural networks, can approximate any time-dependent function f(t)
given by a number of function values. The approximation can effectively be
learned by simply solving a linear equation system; no backpropagation or
similar methods are needed. Furthermore, the network size can be reduced by
taking only most relevant components. Thus, in contrast to others, our approach
not only learns network weights but also the network architecture. The networks
have interesting properties: They end up in ellipse trajectories in the long
run and allow the prediction of further values and compact representations of
functions. We demonstrate this by several experiments, among them multiple
superimposed oscillators (MSO), robotic soccer, and predicting stock prices.
Predictive neural networks outperform the previous state-of-the-art for the MSO
task with a minimal number of units.Comment: 22 pages, 14 figures and tables, revised implementatio
Learning to Adaptively Scale Recurrent Neural Networks
Recent advancements in recurrent neural network (RNN) research have
demonstrated the superiority of utilizing multiscale structures in learning
temporal representations of time series. Currently, most of multiscale RNNs use
fixed scales, which do not comply with the nature of dynamical temporal
patterns among sequences. In this paper, we propose Adaptively Scaled Recurrent
Neural Networks (ASRNN), a simple but efficient way to handle this problem.
Instead of using predefined scales, ASRNNs are able to learn and adjust scales
based on different temporal contexts, making them more flexible in modeling
multiscale patterns. Compared with other multiscale RNNs, ASRNNs are bestowed
upon dynamical scaling capabilities with much simpler structures, and are easy
to be integrated with various RNN cells. The experiments on multiple sequence
modeling tasks indicate ASRNNs can efficiently adapt scales based on different
sequence contexts and yield better performances than baselines without
dynamical scaling abilities
Leaning Robust Sequence Features via Dynamic Temporal Pattern Discovery
As a major type of data, time series possess invaluable latent knowledge for describing the real world and human society. In order to improve the ability of intelligent systems for understanding the world and people, it is critical to design sophisticated machine learning algorithms for extracting robust time series features from such latent knowledge. Motivated by the successful applications of deep learning in computer vision, more and more machine learning researchers put their attentions on the topic of applying deep learning techniques to time series data. However, directly employing current deep models in most time series domains could be problematic. A major reason is that temporal pattern types that current deep models are aiming at are very limited, which cannot meet the requirement of modeling different underlying patterns of data coming from various sources. In this study we address this problem by designing different network structures explicitly based on specific domain knowledge such that we can extract features via most salient temporal patterns. More specifically, we mainly focus on two types of temporal patterns: order patterns and frequency patterns. For order patterns, which are usually related to brain and human activities, we design a hashing-based neural network layer to globally encode the ordinal pattern information into the resultant features. It is further generalized into a specially designed Recurrent Neural Networks (RNN) cell which can learn order patterns in an online fashion. On the other hand, we believe audio-related data such as music and speech can benefit from modeling frequency patterns. Thus, we do so by developing two types of RNN cells. The first type tries to directly learn the long-term dependencies on frequency domain rather than time domain. The second one aims to dynamically filter out the noise frequencies based on temporal contexts. By proposing various deep models based on different domain knowledge and evaluating them on extensive time series tasks, we hope this work can provide inspirations for others and increase the community\u27s interests on the problem of applying deep learning techniques to more time series tasks
A Survey on Deep Learning based Time Series Analysis with Frequency Transformation
Recently, frequency transformation (FT) has been increasingly incorporated
into deep learning models to significantly enhance state-of-the-art accuracy
and efficiency in time series analysis. The advantages of FT, such as high
efficiency and a global view, have been rapidly explored and exploited in
various time series tasks and applications, demonstrating the promising
potential of FT as a new deep learning paradigm for time series analysis.
Despite the growing attention and the proliferation of research in this
emerging field, there is currently a lack of a systematic review and in-depth
analysis of deep learning-based time series models with FT. It is also unclear
why FT can enhance time series analysis and what its limitations in the field
are. To address these gaps, we present a comprehensive review that
systematically investigates and summarizes the recent research advancements in
deep learning-based time series analysis with FT. Specifically, we explore the
primary approaches used in current models that incorporate FT, the types of
neural networks that leverage FT, and the representative FT-equipped models in
deep time series analysis. We propose a novel taxonomy to categorize the
existing methods in this field, providing a structured overview of the diverse
approaches employed in incorporating FT into deep learning models for time
series analysis. Finally, we highlight the advantages and limitations of FT for
time series modeling and identify potential future research directions that can
further contribute to the community of time series analysis
State-Frequency Memory Recurrent Neural Networks
Modeling temporal sequences plays a fundamental role in various modern applications and has drawn more and more attentions in the machine learning community. Among those efforts on improving the capability to represent temporal data, the Long Short-Term Memory (LSTM) has achieved great success in many areas. Although the LSTM can capture long-range dependency in the time domain, it does not explicitly model the pattern occurrences in the frequency domain that plays an important role in tracking and predicting data points over various time cycles. We propose the State-Frequency Memory (SFM), a novel recurrent architecture that allows to separate dynamic patterns across different frequency components and their impacts on modeling the temporal contexts of input sequences. By jointly decomposing memorized dynamics into state-frequency components, the SFM is able to offer a fine-grained analysis of temporal sequences by capturing the dependency of uncovered patterns in both time and frequency domains. Evaluations on several temporal modeling tasks demonstrate the SFM can yield competitive performances, in particular as compared with the state-of-the-art LSTM models