Cluster-based Input Weight Initialization for Echo State Networks

Echo State Networks (ESNs) are a special type of recurrent neural networks (RNNs), in which the input and recurrent connections are traditionally generated randomly, and only the output weights are trained. Despite the recent success of ESNs in various tasks of audio, image and radar recognition, we postulate that a purely random initialization is not the ideal way of initializing ESNs. The aim of this work is to propose an unsupervised initialization of the input connections using the K-Means algorithm on the training data. We show that this initialization performs equivalently or superior than a randomly initialized ESN whilst needing significantly less reservoir neurons (2000 vs. 4000 for spoken digit recognition, and 300 vs. 8000 neurons for f0 extraction) and thus reducing the amount of training time. Furthermore, we discuss that this approach provides the opportunity to estimate the suitable size of the reservoir based on the prior knowledge about the data.Comment: Submitted to IEEE Transactions on Neural Network and Learning System (TNNLS), 202

On the application of reservoir computing networks for noisy image recognition

Reservoir Computing Networks (RCNs) are a special type of single layer recurrent neural networks, in which the input and the recurrent connections are randomly generated and only the output weights are trained. Besides the ability to process temporal information, the key points of RCN are easy training and robustness against noise. Recently, we introduced a simple strategy to tune the parameters of RCNs. Evaluation in the domain of noise robust speech recognition proved that this method was effective. The aim of this work is to extend that study to the field of image processing, by showing that the proposed parameter tuning procedure is equally valid in the field of image processing and conforming that RCNs are apt at temporal modeling and are robust with respect to noise. In particular, we investigate the potential of RCNs in achieving competitive performance on the well-known MNIST dataset by following the aforementioned parameter optimizing strategy. Moreover, we achieve good noise robust recognition by utilizing such a network to denoise images and supplying them to a recognizer that is solely trained on clean images. The experiments demonstrate that the proposed RCN-based handwritten digit recognizer achieves an error rate of 0.81 percent on the clean test data of the MNIST benchmark and that the proposed RCN-based denoiser can effectively reduce the error rate on the various types of noise. (c) 2017 Elsevier B.V. All rights reserved

PyRCN: A Toolbox for Exploration and Application of Reservoir Computing Networks

Reservoir Computing Networks belong to a group of machine learning techniques that project the input space non-linearly into a high-dimensional feature space, where the underlying task can be solved linearly. Popular variants of RCNs, e.g.\ Extreme Learning Machines (ELMs), Echo State Networks (ESNs) and Liquid State Machines (LSMs) are capable of solving complex tasks equivalently to widely used deep neural networks, but with a substantially simpler training paradigm based on linear regression. In this paper, we introduce the Python toolbox PyRCN (Python Reservoir Computing Networks) for optimizing, training and analyzing Reservoir Computing Networks (RCNs) on arbitrarily large datasets. The tool is based on widely-used scientific packages, such as numpy and scipy and complies with the scikit-learn interface specification. It provides a platform for educational and exploratory analyses of RCNs, as well as a framework to apply RCNs on complex tasks including sequence processing. With only a small number of basic components, the framework allows the implementation of a vast number of different RCN architectures. We provide extensive code examples on how to set up RCNs for a time series prediction and for a sequence classification task.Comment: Preprint submitted to Engineering Applications of Artificial Intelligenc

Radar signal processing for human identification by means of reservoir computing networks

Along with substantial advances in the area of image processing and, consequently, video-based surveillance systems, concerns about preserving the privacy of people have also deepened. Therefore, replacing conventional video cameras in surveillance systems with less-intrusive and yet effective alternatives, such as micro-wave radars, is of high interest. The aim of this work is to explore the application of Reservoir Computing Networks (RCNs) to the problem of identifying a limited number of people in an indoor environment, leveraging gait information captured by micro-wave radar measurements. These measurements are done using a commercial low-power linear frequency-modulated continuous-wave (FMCW) radar. Besides the low quality of the outputs of such a radar sensor, walking spontaneously as opposed to controlled situations adds another level of complexity to the targeted use case. In this context, RCNs are interesting tools, given that they have shown a high effectiveness in capturing temporal information and handling noise, while at the same time being easy to setup and train. Using Micro-Doppler features as inputs, we follow a structured procedure towards optimizing the parameters of our RCN-based approach, showing that RCNs have a great potential in processing the noisy features provided by a low-power radar

Improved acoustic modeling for automatic piano music transcription using echo state networks

Automatic music transcription (AMT) is one of the challenging problems in Music Information Retrieval with the goal of generating a score-like representation of a polyphonic audio signal. Typically, the starting point of AMT is an acoustic model that computes note likelihoods from feature vectors. In this work, we evaluate the capabilities of Echo State Networks (ESNs) in acoustic modeling of piano music. Our experiments show that the ESN-based models outperform state-of-the-art Convolutional Neural Networks (CNNs) by an absolute improvement of 0.5 F-1-score without using an extra language model. We also discuss that a two-layer ESN, which mimics a hybrid acoustic and language model, achieves better results than the best reference approach that combines Invertible Neural Networks (INNs) with a biGRU language model by an absolute improvement of 0.91 F-1-score

Noise robust continuous digit recognition with reservoir-based acoustic models

Notwithstanding the many years of research, more work is needed to create automatic speech recognition (ASR) systems with a close-to-human robustness against confounding factors such as ambient noise, channel distortion, etc. Whilst most work thus far focused on the improvement of ASR systems embedding Gaussian Mixture Models (GMM)s to compute the acoustic likelihoods in the states of a Hidden Markov Model (HMM), the present work focuses on the noise robustness of systems employing Reservoir Computing (RC) as an alternative acoustic modeling technique. Previous work already demonstrated good noise robustness for continuous digit recognition (CDR). The present paper investigates whether further progress can be achieved by driving reservoirs with noise-robust inputs that have been shown to raise the robustness of GMM-based systems, by introducing bi-directional reservoirs and by combining reservoirs with GMMs in a single system. Experiments on Aurora-2 demonstrate that it is indeed possible to raise the noise robustness without significantly increasing the system complexity