Temporal Spatial Decomposition and Fusion Network for Time Series Forecasting

Feature engineering is required to obtain better results for time series forecasting, and decomposition is a crucial one. One decomposition approach often cannot be used for numerous forecasting tasks since the standard time series decomposition lacks flexibility and robustness. Traditional feature selection relies heavily on preexisting domain knowledge, has no generic methodology, and requires a lot of labor. However, most time series prediction models based on deep learning typically suffer from interpretability issue, so the "black box" results lead to a lack of confidence. To deal with the above issues forms the motivation of the thesis. In the paper we propose TSDFNet as a neural network with self-decomposition mechanism and an attentive feature fusion mechanism, It abandons feature engineering as a preprocessing convention and creatively integrates it as an internal module with the deep model. The self-decomposition mechanism empowers TSDFNet with extensible and adaptive decomposition capabilities for any time series, users can choose their own basis functions to decompose the sequence into temporal and generalized spatial dimensions. Attentive feature fusion mechanism has the ability to capture the importance of external variables and the causality with target variables. It can automatically suppress the unimportant features while enhancing the effective ones, so that users do not have to struggle with feature selection. Moreover, TSDFNet is easy to look into the "black box" of the deep neural network by feature visualization and analyze the prediction results. We demonstrate performance improvements over existing widely accepted models on more than a dozen datasets, and three experiments showcase the interpretability of TSDFNet.Comment: 10 page

Long Term Predictive Modeling on Big Spatio-Temporal Data

In the era of massive data, one of the most promising research fields involves the analysis of large-scale Spatio-temporal databases to discover exciting and previously unknown but potentially useful patterns from data collected over time and space. A modeling process in this domain must take temporal and spatial correlations into account, but with the dimensionality of the time and space measurements increasing, the number of elements potentially contributing to a target sharply grows, making the target\u27s long-term behavior highly complex, chaotic, highly dynamic, and hard to predict. Therefore, two different considerations are taken into account in this work: one is about how to identify the most relevant and meaningful features from the original Spatio-temporal feature space; the other is about how to model complex space-time dynamics with sensitive dependence on initial and boundary conditions. First, identifying strongly related features and removing the irrelevant or less important features with respect to a target feature from large-scale Spatio-temporal data sets is a critical and challenging issue in many fields, including the evolutionary history of crime hot spots, uncovering weather patterns, predicting floodings, earthquakes, and hurricanes, and determining global warming trends. The optimal sub-feature-set that contains all the valuable information is called the Markov Boundary. Unfortunately, the existing feature selection methods often focus on identifying a single Markov Boundary when real-world data could have many feature subsets that are equally good boundaries. In our work, we design a new multiple-Markov-boundary-based predictive model, Galaxy, to identify the precursors to heavy precipitation event clusters and predict heavy rainfall with a long lead time. We applied Galaxy to an extremely high-dimensional meteorological data set and finally determined 15 Markov boundaries related to heavy rainfall events in the Des Moines River Basin in Iowa. Our model identified the cold surges along the coast of Asia as an essential precursor to the surface weather over the United States, a finding which was later corroborated by climate experts. Second, chaotic behavior exists in many nonlinear Spatio-temporal systems, such as climate dynamics, weather prediction, and the space-time dynamics of virus spread. A reliable solution for these systems must handle their complex space-time dynamics and sensitive dependence on initial and boundary conditions. Deep neural networks\u27 hierarchical feature learning capabilities in both spatial and temporal domains are helpful for nonlinear Spatio-temporal dynamics modeling. However, sensitive dependence on initial and boundary conditions is still challenging for theoretical research and many critical applications. This study proposes a new recurrent architecture, error trajectory tracing, and accompanying training regime, Horizon Forcing, for prediction in chaotic systems. These methods have been validated on real-world Spatio-temporal data sets, including one meteorological dataset, three classics, chaotic systems, and four real-world time series prediction tasks with chaotic characteristics. Experiments\u27 results show that each proposed model could outperform the performance of current baseline approaches

Machine learning for scientific data mining and solar eruption prediction

This dissertation explores new machine learning techniques and adapts them to mine scientific data, specifically data from solar physics and space weather studies. The dissertation tackles three important problems in heliophysics: solar flare prediction, coronal mass ejection (CME) prediction and Stokes inversion. First, the dissertation presents a long short-term memory (LSTM) network for predicting whether an active region (AR) would produce a certain class of solar flare within the next 24 hours. The essence of this approach is to model data samples in an AR as time series and use LSTMs to capture temporal information of the data samples. The LSTM network consists of an LSTM layer, an attention layer, two fully connected layers and an output layer. The attention layer is designed to allow the LSTM network to automatically search for parts of the data samples that are related to the prediction of solar flares. Second, the dissertation presents two recurrent neural networks (RNNs), one based on gated recurrent units and the other based on LSTM, for predicting whether an AR that produces a significant flare will also initiate a CME. Again, data samples in an AR are modeled as time series and the RNNs are used to capture temporal dependencies in the time series. A feature selection technique is employed to enhance prediction accuracy. Third, the dissertation approaches the Stokes inversion problem using a novel convolutional neural network (CNN). This CNN method is faster, and produces cleaner magnetic maps, than a widely used physics-based tool. Furthermore, the CNN method outperforms other machine learning algorithms such as multiple support vector regression and multilayer perceptrons. Findings reported here have been validated by substantial experiments based on different datasets. The dissertation concludes with a fully operational database system containing real-time flare forecasting results produced by the proposed LSTM method. This is the first cyberinfrastructure capable of continuous learning and forecasting of solar flares based on deep learning

Short-Term Forecasting of Passenger Demand under On-Demand Ride Services: A Spatio-Temporal Deep Learning Approach

Short-term passenger demand forecasting is of great importance to the on-demand ride service platform, which can incentivize vacant cars moving from over-supply regions to over-demand regions. The spatial dependences, temporal dependences, and exogenous dependences need to be considered simultaneously, however, which makes short-term passenger demand forecasting challenging. We propose a novel deep learning (DL) approach, named the fusion convolutional long short-term memory network (FCL-Net), to address these three dependences within one end-to-end learning architecture. The model is stacked and fused by multiple convolutional long short-term memory (LSTM) layers, standard LSTM layers, and convolutional layers. The fusion of convolutional techniques and the LSTM network enables the proposed DL approach to better capture the spatio-temporal characteristics and correlations of explanatory variables. A tailored spatially aggregated random forest is employed to rank the importance of the explanatory variables. The ranking is then used for feature selection. The proposed DL approach is applied to the short-term forecasting of passenger demand under an on-demand ride service platform in Hangzhou, China. Experimental results, validated on real-world data provided by DiDi Chuxing, show that the FCL-Net achieves better predictive performance than traditional approaches including both classical time-series prediction models and neural network based algorithms (e.g., artificial neural network and LSTM). This paper is one of the first DL studies to forecast the short-term passenger demand of an on-demand ride service platform by examining the spatio-temporal correlations.Comment: 39 pages, 10 figure

Toward Automating and Systematizing the Use of Domain Knowledge in Feature Selection

University of Minnesota Ph.D. dissertation. August 2015. Major: Computer Science. Advisor: Maria Gini. 1 computer file (PDF); xi, 185 pages.Constructing prediction models for real-world domains often involves practical complexities that must be addressed to achieve good prediction results. Often, there are too many sources of data (features). Limiting the set of features in the prediction model is essential for good performance, but prediction accuracy may be degraded by the inadvertent removal of relevant features. The problem is even more acute in situations where the number of training instances is limited, as limited sample size and domain complexity are often attributes of real-world problems. This thesis explores the practical challenges of building regression models in large multivariate time-series domains with known relationships between variables. Further, we explore the conventional wisdom related to preparing datasets for model calibration in machine learning, and discuss best practices for learning time-varying concepts from data. The core contribution of this work is a novel wrapper-based feature selection framework called Developer-Guided Feature Selection (DGFS). It systematically incorporates domain knowledge for domains characterized by a large number of observable features. The observable features may be related to each other by logical, temporal, or spatial relationships, some of which are known to the model developer a priori. The approach relies on limited domain-specific knowledge but can replace or improve upon more elaborate domain specific models and on fully automated feature selection for many applications. As a wrapper-based approach, DGFS can augment existing multivariate techniques used in high-dimensional domains to produce improved modeling results particularly in situations where the volume of training data is limited. We demonstrate the viability of our method in several complex domains (natural and synthetic) that have significant temporal aspects and many observable features

Using Echo State Networks for Robot Navigation Behavior Acquisition

International audienceRobot Behavior Learning by Demonstration deals with the ability for a robot to learn a behavior from one or several demonstrations provided by a human teacher, possibly through tele-operation or imitation. This implies controllers that can address both (1) the feature selection problem related to a great amount of mostly irrelevant sensory data and (2) dealing with temporal sequences of demonstrations. Echo State Networks have been proposed recently for time series prediction and have been shown to perform remarkably well on this kind of data. In this paper, we introduce ESN to robot behavior acquisition in the scope of a mobile robot performing navigation tasks. ESN actually show comparable and even better performance with that of other algorithms from the literature in similar experimental conditions. Moreover, some properties regarding dynamics of ESN in the context of learning by demonstration are investigated

A long short-temory relation network for real-time prediction of patient-specific ventilator parameters

Accurate prediction of patient-specific ventilator parameters is crucial for optimizing patient-ventilator interaction. Current approaches encounter difficulties in concurrently observing long-term, time-series dependencies and capturing complex, significant features that influence the ventilator treatment process, thereby hindering the achievement of accurate prediction of ventilator parameters. To address these challenges, we propose a novel approach called the long short-term memory relation network (LSTMRnet). Our approach uses a long, short-term memory bank to store rich information and an important feature selection step to extract relevant features related to respiratory parameters. This information is obtained from the prior knowledge of the follow up model. We also concatenate the embeddings of both information types to maintain the joint learning of spatio-temporal features. Our LSTMRnet effectively preserves both time-series and complex spatial-critical feature information, enabling an accurate prediction of ventilator parameters. We extensively validate our approach using the publicly available medical information mart for intensive care (MIMIC-III) dataset and achieve superior results, which can be potentially utilized for ventilator treatment (i.e., sleep apnea-hypopnea syndrome ventilator treatment and intensive care units ventilator treatment

Neural Network Ensembles for Time Series Prediction

Rapidly evolving businesses generate massive amounts of time-stamped data sequences and defy a demand for massively multivariate time series analysis. For such data the predictive engine shifts from the historical auto-regression to modelling complex non-linear relationships between multidimensional features and the time series outputs. In order to exploit these time-disparate relationships for the improved time series forecasting, the system requires a flexible methodology of combining multiple prediction models applied to multiple versions of the temporal data under significant noise component and variable temporal depth of predictions. In reply to this challenge a composite time series prediction model is proposed which combines the strength of multiple neural network (NN) regressors applied to the temporally varied feature subsets and the postprocessing smoothing of outputs developed to further reduce noise. The key strength of the model is its excellent adaptability and generalisation ability achieved through a highly diversified set of complementary NN models. The model has been evaluated within NISIS Competition 2006 and NN3 Competition 2007 concerning prediction of univariate and multivariate time-series. It showed the best predictive performance among 12 competitive models in the NISIS 2006 and is under evaluation within NN3 2007 Competition