Recurrent Gaussian processes

We define Recurrent Gaussian Processes (RGP) models, a general family of Bayesian nonparametric models with recurrent GP priors which are able to learn dynamical patterns from sequential data. Similar to Recurrent Neural Networks (RNNs), RGPs can have different formulations for their internal states, distinct inference methods and be extended with deep structures. In such context, we propose a novel deep RGP model whose autoregressive states are latent, thereby performing representation and dynamical learning simultaneously. To fully exploit the Bayesian nature of the RGP model we develop the Recurrent Variational Bayes (REVARB) framework, which enables efficient inference and strong regularization through coherent propagation of uncertainty across the RGP layers and states. We also introduce a RGP extension where variational parameters are greatly reduced by being reparametrized through RNN-based sequential recognition models. We apply our model to the tasks of nonlinear system identification and human motion modeling. The promising obtained results indicate that our RGP model maintains its highly flexibility while being able to avoid overfitting and being applicable even when larger datasets are not available

Recurrent Gaussian Processes

We define Recurrent Gaussian Processes (RGP) models, a general family of Bayesian nonparametric models with recurrent GP priors which are able to learn dynamical patterns from sequential data. Similar to Recurrent Neural Networks (RNNs), RGPs can have different formulations for their internal states, distinct inference methods and be extended with deep structures. In such context, we propose a novel deep RGP model whose autoregressive states are latent, thereby performing representation and dynamical learning simultaneously. To fully exploit the Bayesian nature of the RGP model we develop the Recurrent Variational Bayes (REVARB) framework, which enables efficient inference and strong regularization through coherent propagation of uncertainty across the RGP layers and states. We also introduce a RGP extension where variational parameters are greatly reduced by being reparametrized through RNN-based sequential recognition models. We apply our model to the tasks of nonlinear system identification and human motion modeling. The promising obtained results indicate that our RGP model maintains its highly flexibility while being able to avoid overfitting and being applicable even when larger datasets are not available

Defining and applying prediction performance metrics on a recurrent NARX time series model.

International audienceNonlinear autoregressive moving average with exogenous inputs (NARMAX) models have been successfully demonstrated for modeling the input-output behavior of many complex systems. This paper deals with the proposition of a scheme to provide time series prediction. The approach is based on a recurrent NARX model obtained by linear combination of a recurrent neural network (RNN) output and the real data output. Some prediction metrics are also proposed to assess the quality of predictions. This metrics enable to compare different prediction schemes and provide an objective way to measure how changes in training or prediction model (Neural network architecture) affect the quality of predictions. Results show that the proposed NARX approach consistently outperforms the prediction obtained by the RNN neural network

Behavioural pattern identification and prediction in intelligent environments

In this paper, the application of soft computing techniques in prediction of an occupant's behaviour in an inhabited intelligent environment is addressed. In this research, daily activities of elderly people who live in their own homes suffering from dementia are studied. Occupancy sensors are used to extract the movement patterns of the occupant. The occupancy data is then converted into temporal sequences of activities which are eventually used to predict the occupant behaviour. To build the prediction model, different dynamic recurrent neural networks are investigated. Recurrent neural networks have shown a great ability in finding the temporal relationships of input patterns. The experimental results show that non-linear autoregressive network with exogenous inputs model correctly extracts the long term prediction patterns of the occupant and outperformed the Elman network. The results presented here are validated using data generated from a simulator and real environments

Improving the prediction accuracy of recurrent neural network by a PID controller.

International audienceIn maintenance field, prognostic is recognized as a key feature as the prediction of the remaining useful life of a system which allows avoiding inopportune maintenance spending. Assuming that it can be difficult to provide models for that purpose, artificial neural networks appear to be well suited. In this paper, an approach combining a Recurrent Radial Basis Function network (RRBF) and a proportional integral derivative controller (PID) is proposed in order to improve the accuracy of predictions. The PID controller attempts to correct the error between the real process variable and the neural network predictions

Model structure selection using an integrated forward orthogonal search algorithm interfered with squared correlation and mutual information

Model structure selection plays a key role in nonlinear system identification. The first step in nonlinear system identification is to determine which model terms should be included in the model. Once significant model terms have been determined, a model selection criterion can then be applied to select a suitable model subset. The well known orthogonal least squares type algorithms are one of the most efficient and commonly used techniques for model structure selection. However, it has been observed that the orthogonal least squares type algorithms may occasionally select incorrect model terms or yield a redundant model subset in the presence of particular noise structures or input signals. A very efficient integrated forward orthogonal searching (IFOS) algorithm, which is interfered with squared correlation and mutual information, and which incorporates a general cross-validation (GCV) criterion and hypothesis tests, is introduced to overcome these limitations in model structure selection

Data driven nonlinear dynamic models for predicting heavy-duty diesel engine torque and combustion emissions

Diesel engines' reliable and durable structures, high torque generation capabilities at low speeds, and fuel consumption efficiencies make them irreplaceable for heavy-duty vehicles in the market. However, ine ciencies in the combustion process result in the release of emissions to the environment. In addition to the restrictive international regulations for emissions, the competitive demands for more powerful engines and increasing fuel prices obligate heavy-duty engine and vehicle manufacturers to seek for solutions to reduce the emissions while meeting the performance requirements. In line with these objectives, remarkable progress has been made in modern diesel engine systems such as air handling, fuel injection, combustion, and after-treatment. However, such systems utilize quite sophisticated equipment with a large number of calibratable parameters that increases the experimentation time and effort to find the optimal operating points. Therefore, a dynamic model-based transient calibration is required for an e cient combustion optimization which obeys the emission limits, and meets the desired power and efficiency requirements. This thesis is about developing optimizationoriented high delity nonlinear dynamic models for predicting heavy-duty diesel engine torque and combustion emissions. Contributions of the thesis are: (i) A new design of experiments is proposed where air-path and fuel-path input channels are excited by chirp signals with varying frequency pro les in terms of the number and directions of the sweeps. The proposed approach is a strong alternative to the steady-state experiment based approaches to reduce the testing time considerably and improve the modeling accuracy in both steady-state and transient conditions. (ii) A nonlinear nite impulse response (NFIR) model is developed to predict indicated torque by including the estimations of friction, pumping and inertia torques in addition to the torque measured from the engine dynamometer. (iii) Two different nonlinear autoregressive with exogenous input (NARX) models are proposed to predict NOx emissions. In the first structure, input regressor set for the nonlinear part of the model is reduced by an orthogonal least square (OLS) algorithm to increase the robustness and decrease the sensitivity to parameter changes, and linear output feedback is employed. In the second structure, only the previous output is used as the output regressor in the model due to the stability considerations. (iv) An analysis of model sensitivities to parameter changes is conducted and an easy-tointerpret map is introduced to select the best modeling parameters with limited testing time in powertrain development. (v) Soot (particulated matter) emission is predicted using LSTM type networks which provide more accurate and smoother predictions than NARX models. Experimental results obtained from the engine dynamometer tests show the e ectiveness of the proposed models in terms of prediction accuracies in both NEDC (New European Driving Cycle) and WHTC (World Harmonized Transient Cycle) cycle

Modeling Dynamic Systems for Multi-Step Prediction with Recurrent Neural Networks

This thesis investigates the applicability of Recurrent Neural Networks (RNNs) and Deep Learning methods for multi-step prediction of robotic systems. The unmodeled dynamics and simplifying assumptions in classic modeling methods result in models that yield rapidly diverging predictions when the model is used in an iterative fashion, i.e., for multi-step prediction. However, the effect of the unmodeled dynamics can be captured by collecting datasets of the system. Deep Learning provides a strong set of tools to extract patterns from data, however, large datasets are commonly required for the methods to work well. Collecting a large amount of data from a robotic system can be a cumbersome and expensive approach. In this work, Deep Learning methods, particularly RNNs, are studied and employed for the purpose of learning models of two aerial vehicles from experimental data. The feasibility of employing RNNs is first studied to learn a model of a quadrotor based on a simulated dataset, which yields a Multi Layer Fully Connected (MLFC) architecture. Models can be learned for multi-step prediction, recovering excellent predictions over 500 timesteps in the presence of simulated disturbances to the robot and noise on the measurements. To learn models from experimental data, the RNN state initialization problem is defined and formulated. It is shown that the RNN state initialization problem can be addressed by creating and training an initialization network jointly with the multi-step prediction network, and the combination can be used in a black-box modeling approach such that the model produces predictions which are immediately accurate. The RNN based black-box methods are trained on an experimental dataset gathered from a quadrotor and a publicly available helicopter dataset. The quadrotor dataset, which encompasses approximately 4 hours of flight data in various regimes, has been released and is now available publicly online. Finally, a hybrid network, which combines the proposed RNN based black-box models with a physics based quadrotor model into a single RNN-based modeling system is introduced. The proposed hybrid network solves many of the limitations of the existing state of the art in long-term prediction for robotics systems. Trained on the quadrotor dataset, the hybrid model provides accurate body angular rate and velocity predictions of the vehicle over almost 2 seconds which is suitable to be used in a variety of model-based controller applications