31,999 research outputs found
The difference between memory and prediction in linear recurrent networks
Recurrent networks are trained to memorize their input better, often in the
hopes that such training will increase the ability of the network to predict.
We show that networks designed to memorize input can be arbitrarily bad at
prediction. We also find, for several types of inputs, that one-node networks
optimized for prediction are nearly at upper bounds on predictive capacity
given by Wiener filters, and are roughly equivalent in performance to randomly
generated five-node networks. Our results suggest that maximizing memory
capacity leads to very different networks than maximizing predictive capacity,
and that optimizing recurrent weights can decrease reservoir size by half an
order of magnitude
Photonic Delay Systems as Machine Learning Implementations
Nonlinear photonic delay systems present interesting implementation platforms
for machine learning models. They can be extremely fast, offer great degrees of
parallelism and potentially consume far less power than digital processors. So
far they have been successfully employed for signal processing using the
Reservoir Computing paradigm. In this paper we show that their range of
applicability can be greatly extended if we use gradient descent with
backpropagation through time on a model of the system to optimize the input
encoding of such systems. We perform physical experiments that demonstrate that
the obtained input encodings work well in reality, and we show that optimized
systems perform significantly better than the common Reservoir Computing
approach. The results presented here demonstrate that common gradient descent
techniques from machine learning may well be applicable on physical
neuro-inspired analog computers
- …