Towards Optimally Distributed Computation

This paper introduces the concept of "optimally distributed computation" in feed-forward neural networks via regularization of weight saliency. By constraining the relative importance of the parameters, computation can be distributed thinly and evenly throughout the network. We propose that this will have beneficial effects on fault tolerance performance and generalization ability in large network architectures. These theoretical predictions are verified by simulation experiments on two problems --- one artificial and the other a "real world" task. In summary, this paper presents regularization terms for distributing neural computation optimally. 1 Introduction Optimally distributed computation leads to fault tolerance and follows the spirit of Occam's razor even in large networks to yield good generalization ability. In neural network models computation is more or less distributed in the weighted connections. However, the "distributedness" is in no way optimal and poor performance, ..

Towards Optimally Distributed Computation in Augmented Networks

This paper introduces the concept of "optimally distributed computation" in feed-forward neural networks via regularisation of weight saliency. By constraining the relative importance of the parameters, computation can be distributed thinly and evenly throughout the network. We propose that this will have beneficial effects on fault tolerance performance and generalisation ability in augmented network architectures. These theoretical predictions are verified by simulation experiments on two problems --- one artificial and the other a "real world" task. In summary, this paper presents regularisation terms for distributing neural computation optimally. 1 Introduction Optimally distributed computation leads to fault tolerance and follows the spirit of Occam's razor even in augmented networks to yield good generalisation ability. In neural network models computation is more or less distributed in the weighted connections. However, the "distributedness" is in no way optimal and poor perfo..