Recognizing recurrent neural networks (rRNN): Bayesian inference for recurrent neural networks

Recurrent neural networks (RNNs) are widely used in computational neuroscience and machine learning applications. In an RNN, each neuron computes its output as a nonlinear function of its integrated input. While the importance of RNNs, especially as models of brain processing, is undisputed, it is also widely acknowledged that the computations in standard RNN models may be an over-simplification of what real neuronal networks compute. Here, we suggest that the RNN approach may be made both neurobiologically more plausible and computationally more powerful by its fusion with Bayesian inference techniques for nonlinear dynamical systems. In this scheme, we use an RNN as a generative model of dynamic input caused by the environment, e.g. of speech or kinematics. Given this generative RNN model, we derive Bayesian update equations that can decode its output. Critically, these updates define a 'recognizing RNN' (rRNN), in which neurons compute and exchange prediction and prediction error messages. The rRNN has several desirable features that a conventional RNN does not have, for example, fast decoding of dynamic stimuli and robustness to initial conditions and noise. Furthermore, it implements a predictive coding scheme for dynamic inputs. We suggest that the Bayesian inversion of recurrent neural networks may be useful both as a model of brain function and as a machine learning tool. We illustrate the use of the rRNN by an application to the online decoding (i.e. recognition) of human kinematics

Outcome Prediction for Unipolar Depression

Although effective drug and non-drug treatment for unipolar depressive illness exist, different individuals respond differently to different treatments. It is not uncommon for a given patient to lw switched several times from one treatment to another until an effective remedy for that particular patient is found. This process is costly in terms of time, money and suffering. It is thus desirable to determine at the outset the likdy response of a patient to the available treatments, so that the optimal one can be selected. Although prior attempts at outcome prediction with linear regression models have failed, recent work on this problem has indicated that the nonlinear predictive techniques of backpropagation and quadratic regression call account for a significant proportion of the variance in the data. The present research applies the nonlinear predictive technique of kernel regression to this problcrn, and employs cross-validation to test the ability of the resulting model to extract, from extremely noisy dinical data, information with predictive value. The importance of comparison with a suitable null hypothesis is illustrated.Office of Naval Research (N00014-95-1-0409

Geographical General Regression Neural Network (GGRNN) tool for geographically weighted regression analysis

This paper presents a new geographically weighted regression analysis tool, based upon a modified version of a General Regression Neural Network (GRNN). The new Geographic General Regression Neural Network (GGRNN) tool allows for local variations in the regression analysis. The algorithm of the GRNN has been extended to allow for both globally independent variables and local variables, restricted to a given spatial kernel. This mimics the results of Geographically Weighted Regression (GWR) analysis in a given geographical space. The GGRNN tool allows the user to load geographic data from the Shapefile into the underlying neural networks data structure. The spatial kernel can be either a fixed radius or adaptive, by using a given number of neighboring regions. The Holdout Method has been used to compare the fitness of a given model. An application of the tool has been presented using the benchmark working-age deaths in the Tokyo metropolitan area, Japan. Standardized residual maps produced by the GGRNN tool have been compared with those produced by the GWR4 tool for validation. The tool has been developed in the .Net C# programming language using the DotSpatial open source library. The tool is valuable because it allows the user to investigate the influence of spatially non-stationary processes in the regression analysis. The tool can also be used for prediction or interpolation purposes for a range of environmental, socioeconomic and public health applications

A novel improved model for building energy consumption prediction based on model integration

Building energy consumption prediction plays an irreplaceable role in energy planning, management, and conservation. Constantly improving the performance of prediction models is the key to ensuring the efficient operation of energy systems. Moreover, accuracy is no longer the only factor in revealing model performance, it is more important to evaluate the model from multiple perspectives, considering the characteristics of engineering applications. Based on the idea of model integration, this paper proposes a novel improved integration model (stacking model) that can be used to forecast building energy consumption. The stacking model combines advantages of various base prediction algorithms and forms them into “meta-features” to ensure that the final model can observe datasets from different spatial and structural angles. Two cases are used to demonstrate practical engineering applications of the stacking model. A comparative analysis is performed to evaluate the prediction performance of the stacking model in contrast with existing well-known prediction models including Random Forest, Gradient Boosted Decision Tree, Extreme Gradient Boosting, Support Vector Machine, and K-Nearest Neighbor. The results indicate that the stacking method achieves better performance than other models, regarding accuracy (improvement of 9.5%–31.6% for Case A and 16.2%–49.4% for Case B), generalization (improvement of 6.7%–29.5% for Case A and 7.1%-34.6% for Case B), and robustness (improvement of 1.5%–34.1% for Case A and 1.8%–19.3% for Case B). The proposed model enriches the diversity of algorithm libraries of empirical models

Neural Extensions to Robust Parameter Design

Robust parameter design (RPD) is implemented in systems in which a user wants to minimize the variance of a system response caused by uncontrollable factors while obtaining a consistent and reliable system response over time. We propose the use of artificial neural networks to compensate for highly non-linear problems that quadratic regression fails to accurately model. RPD is conducted under the assumption that the relationship between system response and controllable and uncontrollable variables does not change over time. We propose a methodology to find a new set of settings that will be robust to moderate system degradation while remaining robust to noise variables within the system RPD has been well developed on single response problems. Sparse literature exists on dealing with multiple responses in RPD and most methods utilize a subjective weighting scheme. To account for multiple responses, we examine the use of factor analysis on the response data. All the proposed techniques are applied to textbook applications to demonstrate their utility. An Air Force application problem is examined to demonstrate the new technique’s potential on a real-world problem that is highly non-linear. The application is a detector developed to detect anomalies within hyper-spectral imagery