Optimized parameter search for large datasets of the regularization parameter and feature selection for ridge regression

In this paper we propose mathematical optimizations to select the optimal regularization parameter for ridge regression using cross-validation. The resulting algorithm is suited for large datasets and the computational cost does not depend on the size of the training set. We extend this algorithm to forward or backward feature selection in which the optimal regularization parameter is selected for each possible feature set. These feature selection algorithms yield solutions with a sparse weight matrix using a quadratic cost on the norm of the weights. A naive approach to optimizing the ridge regression parameter has a computational complexity of the order with the number of applied regularization parameters, the number of folds in the validation set, the number of input features and the number of data samples in the training set. Our implementation has a computational complexity of the order . This computational cost is smaller than that of regression without regularization for large datasets and is independent of the number of applied regularization parameters and the size of the training set. Combined with a feature selection algorithm the algorithm is of complexity and for forward and backward feature selection respectively, with the number of selected features and the number of removed features. This is an order faster than and for the naive implementation, with for large datasets. To show the performance and reduction in computational cost, we apply this technique to train recurrent neural networks using the reservoir computing approach, windowed ridge regression, least-squares support vector machines (LS-SVMs) in primal space using the fixed-size LS-SVM approximation and extreme learning machines

European exchange trading funds trading with locally weighted support vector regression

In this paper, two different Locally Weighted Support Vector Regression (wSVR) algorithms are generated and applied to the task of forecasting and trading five European Exchange Traded Funds. The trading application covers the recent European Monetary Union debt crisis. The performance of the proposed models is benchmarked against traditional Support Vector Regression (SVR) models. The Radial Basis Function, the Wavelet and the Mahalanobis kernel are explored and tested as SVR kernels. Finally, a novel statistical SVR input selection procedure is introduced based on a principal component analysis and the Hansen, Lunde, and Nason (2011) model confidence test. The results demonstrate the superiority of the wSVR models over the traditional SVRs and of the v-SVR over the ε-SVR algorithms. We note that the performance of all models varies and considerably deteriorates in the peak of the debt crisis. In terms of the kernels, our results do not confirm the belief that the Radial Basis Function is the optimum choice for financial series

A review on Day-Ahead Solar Energy Prediction

Accurate day-ahead prediction of solar energy plays a vital role in the planning of supply and demand in a power grid system. The previous study shows predictions based on weather forecasts composed of numerical text data. They can reflect temporal factors therefore the data versus the result might not always give the most accurate and precise results. That is why incorporating different methods and techniques which enhance accuracy is an important topic. An in-depth review of current deep learning-based forecasting models for renewable energy is provided in this paper

Wind Power Forecasting Based on Echo State Networks and Long Short-Term Memory

Wind power generation has presented an important development around the world. However, its integration into electrical systems presents numerous challenges due to the variable nature of the wind. Therefore, to maintain an economical and reliable electricity supply, it is necessary to accurately predict wind generation. The Wind Power Prediction Tool (WPPT) has been proposed to solve this task using the power curve associated with a wind farm. Recurrent Neural Networks (RNNs) model complex non-linear relationships without requiring explicit mathematical expressions that relate the variables involved. In particular, two types of RNN, Long Short-Term Memory (LSTM) and Echo State Network (ESN), have shown good results in time series forecasting. In this work, we present an LSTM+ESN architecture that combines the characteristics of both networks. An architecture similar to an ESN is proposed, but using LSTM blocks as units in the hidden layer. The training process of this network has two key stages: (i) the hidden layer is trained with a descending gradient method online using one epoch; (ii) the output layer is adjusted with a regularized regression. In particular, the case is proposed where Step (i) is used as a target for the input signal, in order to extract characteristics automatically as the autoencoder approach; and in the second stage (ii), a quantile regression is used in order to obtain a robust estimate of the expected target. The experimental results show that LSTM+ESN using the autoencoder and quantile regression outperforms the WPPT model in all global metrics used

A Radial Basis Function Neural Network using biologically plausible activation functions

Este proyecto se centra en el diseño, la implementación y la evaluación de Redes Neuronales de Función de Base Radial (RBFNN), comparando el modelo gaussiano con una nueva versión que utiliza la función de activación Ricker. La forma de esta función ha sido observada en las señales de neuronas de distintas partes del cerebro humano, a menudo produciendo una señal negativa (inhibitoria) conocida como inhibición lateral. Se han desarrollado dos modelos de RBFNN, incorporando técnicas de Machine Learning (ML) y estadística como la regularización L2 y el algoritmo sigest para mejorar su rendimiento. También se implementan técnicas adicionales, como la estimación de un parámetro k sobredimensionado y la AIC backward selection, para mejorar la eficiencia. En este estudio, los modelos desarrollados se prueban con conjuntos de datos de diferente naturaleza, evaluando su rendimiento con datos sintéticos y realistas, y midiendo sus resultados con problemas de varios niveles de ruido y dificultad. Además, también se realiza una comparación de los modelos para observar qué RBFNN funciona mejor en determinadas condiciones, así como para analizar la diferencia en el número de neuronas y el parámetro de suavizado estimado. La evaluación experimental confirma la eficacia de los modelos RBFNN, proporcionando estimaciones precisas y demostrando su adaptabilidad con problemas de dificultad variable. El análisis comparativo revela que el modelo Ricker tiende a exhibir un rendimiento superior en presencia de altos niveles de ruido, mientras que ambos modelos tienen un rendimiento similar en condiciones de bajo ruido. Estos resultados sugieren la potencial influencia de la inhibición lateral, que podría ser explorada en más profundidad en futuros estudios.This project focuses on the design, implementation and evaluation of Radial Basis Function Neural Networks (RBFNN), comparing the gaussian model with a new version using the Ricker Wavelet activation function. The shape of this wavelet has been observed in the signals of neurons from different parts of the human brain, often producing a negative (inhibitory) signal known as lateral inhibition. Two RBFNN models have been developed, incorporating Machine Learning (ML) and statistical techniques such as L2 regularization and the sigest algorithm for improved performance. Additional techniques, such as estimating an oversized k parameter and using AIC backward selection, are implemented to enhance efficiency. In this study, the developed models are tested with datasets of different nature, evaluating their performance with synthetic and realistic data and measuring their results with problems of various levels of noise and difficulty. Furthermore, a comparison of the models is also made in order to observe which RBFNN performs better on certain conditions, as well as to analyze the difference in the number of neurons and the estimated smoothing parameter. The experimental evaluation confirms the effectiveness of the RBFNN models, yielding accurate estimations and demonstrating their adaptability to problems of varying difficulty. Comparative analysis reveals that the Ricker model tends to exhibit superior performance in the presence of high levels of noise, while both models perform similarly under low noise conditions. These results suggest the potential influence of lateral inhibition, which could be explored further in future studies

Bio-signal based control in assistive robots: a survey

Recently, bio-signal based control has been gradually deployed in biomedical devices and assistive robots for improving the quality of life of disabled and elderly people, among which electromyography (EMG) and electroencephalography (EEG) bio-signals are being used widely. This paper reviews the deployment of these bio-signals in the state of art of control systems. The main aim of this paper is to describe the techniques used for (i) collecting EMG and EEG signals and diving these signals into segments (data acquisition and data segmentation stage), (ii) dividing the important data and removing redundant data from the EMG and EEG segments (feature extraction stage), and (iii) identifying categories from the relevant data obtained in the previous stage (classification stage). Furthermore, this paper presents a summary of applications controlled through these two bio-signals and some research challenges in the creation of these control systems. Finally, a brief conclusion is summarized

Training Echo State Networks with Regularization through Dimensionality Reduction

In this paper we introduce a new framework to train an Echo State Network to predict real valued time-series. The method consists in projecting the output of the internal layer of the network on a space with lower dimensionality, before training the output layer to learn the target task. Notably, we enforce a regularization constraint that leads to better generalization capabilities. We evaluate the performances of our approach on several benchmark tests, using different techniques to train the readout of the network, achieving superior predictive performance when using the proposed framework. Finally, we provide an insight on the effectiveness of the implemented mechanics through a visualization of the trajectory in the phase space and relying on the methodologies of nonlinear time-series analysis. By applying our method on well known chaotic systems, we provide evidence that the lower dimensional embedding retains the dynamical properties of the underlying system better than the full-dimensional internal states of the network

Synthesis of neural networks for spatio-temporal spike pattern recognition and processing

The advent of large scale neural computational platforms has highlighted the lack of algorithms for synthesis of neural structures to perform predefined cognitive tasks. The Neural Engineering Framework offers one such synthesis, but it is most effective for a spike rate representation of neural information, and it requires a large number of neurons to implement simple functions. We describe a neural network synthesis method that generates synaptic connectivity for neurons which process time-encoded neural signals, and which makes very sparse use of neurons. The method allows the user to specify, arbitrarily, neuronal characteristics such as axonal and dendritic delays, and synaptic transfer functions, and then solves for the optimal input-output relationship using computed dendritic weights. The method may be used for batch or online learning and has an extremely fast optimization process. We demonstrate its use in generating a network to recognize speech which is sparsely encoded as spike times.Comment: In submission to Frontiers in Neuromorphic Engineerin

Recurrent error-based ridge polynomial neural networks for time series forecasting

Time series forecasting has attracted much attention due to its impact on many practical applications. Neural networks (NNs) have been attracting widespread interest as a promising tool for time series forecasting. The majority of NNs employ only autoregressive (AR) inputs (i.e., lagged time series values) when forecasting time series. Moving-average (MA) inputs (i.e., errors) however have not adequately considered. The use of MA inputs, which can be done by feeding back forecasting errors as extra network inputs, alongside AR inputs help to produce more accurate forecasts. Among numerous existing NNs architectures, higher order neural networks (HONNs), which have a single layer of learnable weights, were considered in this research work as they have demonstrated an ability to deal with time series forecasting and have an simple architecture. Based on two HONNs models, namely the feedforward ridge polynomial neural network (RPNN) and the recurrent dynamic ridge polynomial neural network (DRPNN), two recurrent error-based models were proposed. These models were called the ridge polynomial neural network with error feedback (RPNN-EF) and the ridge polynomial neural network with error-output feedbacks (RPNN-EOF). Extensive simulations covering ten time series were performed. Besides RPNN and DRPNN, a pi-sigma neural network and a Jordan pi-sigma neural network were used for comparison. Simulation results showed that introducing error feedback to the models lead to significant forecasting performance improvements. Furthermore, it was found that the proposed models outperformed many state-of-the-art models. It was concluded that the proposed models have the capability to efficiently forecast time series and that practitioners could benefit from using these forecasting models