An Improved Algorithm for Neural Network Classification of Imbalanced Training Sets

In this paper, we analyze the reason for the slow rate of convergence of net output error when using the backpropagation algorithm to train neural networks for a two-class problems in which the numbers of exemplars for the two classes differ greatly. This occurs because the negative gradient vector computed by backpropagation for an imbalanced training set does not point initially in a downhill direction for the class with the smaller number of exemplars. Consequently, in the initial iteration, the net error for the exemplars in this class increases significantly. The subsequent rate of convergence of the net error is very low. We suggest a modified technique for calculating a direction in weight-space which is downhill for both classes. Using this algorithm, we have been able to accelerate the rate of learning for two-class classification problems by an order of magnitude

A fast and robust learning algorithm for feedforward neural networks

The back propagation algorithm caused a tremendous breakthrough in the application of multilayer perceptrons. However, it has some important drawbacks: long training times and sensitivity to the presence of local minima. Another problem is the network topology; the exact number of units in a particular hidden layer, as well as the number of hidden layers need to be known in advance. A lot of time is often spent in finding the optimal topology. In this article, we consider multilayer networks with one hidden layer of Gaussian units and an output layer of conventional units. We show that for this kind of networks, it is possible to perform a fast dimensionality analysis, by analyzing only a small fraction of the input patterns. Moreover, as a result of this approach, it is possible to initialize the weights of the network before starting the back propagation training. Several classification problems are taken as examples

Using constraints to improve generalisation and training of feedforward neural networks : constraint based decomposition and complex backpropagation

Neural networks can be analysed from two points of view: training and generalisation. The training is characterised by a trade-off between the 'goodness' of the training algorithm itself (speed, reliability, guaranteed convergence) and the 'goodness' of the architecture (the difficulty of the problems the network can potentially solve). Good training algorithms are available for simple architectures which cannot solve complicated problems. More complex architectures, which have been shown to be able to solve potentially any problem do not have in general simple and fast algorithms with guaranteed convergence and high reliability. A good training technique should be simple, fast and reliable, and yet also be applicable to produce a network able to solve complicated problems. The thesis presents Constraint Based Decomposition (CBD) as a technique which satisfies the above requirements well. CBD is shown to build a network able to solve complicated problems in a simple, fast and reliable manner. Furthermore, the user is given a better control over the generalisation properties of the trained network with respect to the control offered by other techniques. The generalisation issue is addressed, as well. An analysis of the meaning of the term "good generalisation" is presented and a framework for assessing generalisation is given: the generalisation can be assessed only with respect to a known or desired underlying function. The known properties of the underlying function can be embedded into the network thus ensuring a better generalisation for the given problem. This is the fundamental idea of the complex backpropagation network. This network can associate signals through associating some of their parameters using complex weights. It is shown that such a network can yield better generalisation results than a standard backpropagation network associating instantaneous values