Echo State Networks (ESNs) are a special type of recurrent neural networks
(RNNs), in which the input and recurrent connections are traditionally
generated randomly, and only the output weights are trained. Despite the recent
success of ESNs in various tasks of audio, image and radar recognition, we
postulate that a purely random initialization is not the ideal way of
initializing ESNs. The aim of this work is to propose an unsupervised
initialization of the input connections using the K-Means algorithm on the
training data. We show that this initialization performs equivalently or
superior than a randomly initialized ESN whilst needing significantly less
reservoir neurons (2000 vs. 4000 for spoken digit recognition, and 300 vs. 8000
neurons for f0 extraction) and thus reducing the amount of training time.
Furthermore, we discuss that this approach provides the opportunity to estimate
the suitable size of the reservoir based on the prior knowledge about the data.Comment: Submitted to IEEE Transactions on Neural Network and Learning System
(TNNLS), 202