Artificial neural networks, also called neural networks, have been used successfully in many fields including engineering, science and business. This paper presents the implementation of several neural network simulators and their applications in character recognition and other engineering area

Jain, L.C.

Veelenturf, L.P.J.

Vonk, E.

English

NARCIS 

Neural Network Applications

University of Twente Research Information

Neural Network Applications E. Vonk *. L.C. Jain **, and L.P.J. Veelenturf' *Control, Systems and Computer Engineering Group (BSC), Laboratory for Network Theory, Department of Electrical Engineering, University of Twente, Postbus 217, 7500 AE Enschede, The Netherlands. TeL : +61 8 302 3984 Fax : +61 8 302 3384 Email: 9S0021a81ux.levels.unisa.edu.au Keywords neural network, Back Propagation, Radial Basis Function, Kohonen, ART1, Hopfield, Bidirechonal Assodative Memory, character recogrution Abstract Artificial neural networks, also called neural networks, have been used successfully in many fields including engineering. science and business. This paper presents the implementation of several neural network simulators and their applications in chnracter recognition and other engineering areas. 1. Introduction In the last five years the reported applications of neural networks have multiplied exponentially. These networks have proven to be successful1 in many applications including fault diagnosis, financial control, forecasting, plant control, industrial process control and pattern recognition. Despite tremendous interest in neural networks, there are still many research problems which need careful attention. For example, in the back propagation network, parameters such as learning rate and momentum term are found by trial and error. The mathematical basis for finding the number of neurons in the hidden layer %e reported in the literature [2], but there are still some problems associated with it. Often the researchers adjust the neurons in the hidden layer by trial and error. There are not enough guidelines pertaining to which architecture i s  a suitable for a given application. This paper presents the implemenration of back propagabon, radial basis function, Kohonen net, ARTl, Hopfield net and BAM. Other applications are also presented. 2. Neural network simulations **Knowledge-based Engineering Systems Group, School of Electronic Engineering, University of South Australia. Adelaide, The Levels, 5095, Austtalia. Tel. : +61 8 302 3315 Fax : +61 8 302 3384 Email: etlcj 8 1v.levels.unisa.edu.au This section describes the neural network simulators and their respective software implementations used in this study. 2.1: The Back propagation network (BP). The back propagation network consists of an input layer of neurons (not shown). an output layer of neurons and at least one hidden layer as shown in Figure 1. output 5 Inputs Fig. I:A '3-2-3' Back Propagation Network. The neurons perform a weighted sum of their inputs and use this sum as the input of an activation function which is usually the sigmoid function. A supervised learning algorithm [2 ]  [8] is used to teach the network. It consists of updating all the weights of a network until1 the output of the network reaches the pre-specified desired output. The factors responsible for the training and performance of the network include: - the initial values of the weights (usually - the number of training cycles - the number of hidden neurons - the training set - the values of the teaching parameters random) (learning rate, momentum term) 0-8186-7085-1/95 $04.00 0 1995 IEEE 63 2.2: The Radial basis function (RBF) network [41 [GI 171. The Radial Basis Function network consists of three layers. The hidden layer is used to cluster inputs of he network; neurons in this layer are therefore called cluster centres. Though its architecture is similar to a three layer back propagation network, its working is different. It uses a Gaussian kemcl function to calculate the activations of the neurons i n  the first layer. 'The neurons in the output layer perform an ordinary linear weighted sum of these activations. The learning in this network is done i n  two stages. Fii~tly, the input-training pattern? :ue clustered (unsupervised) to their ntrarest cluster centres by means of a clustering algorithm. The K-Means clustering algorithm i s  used. After this, the spread of each cluster centre is determined. This can be done in several ways; they can be made equal to the average distance between the cluster centre and the training patterns clustered with that centre [4], or they can be set equitl to the average distance between the cluster centre and its nearest cluster centres (often two) 161. 'The second stage of learning is the supervised learning of the weights of the output layer. This is usually done by the least mean square algorithm. The inital value of the output weights are usually set to small random values. The perfonnance of the Radial Basis Function network depends on the way the inputs are clustered. As with the back propagation network, the number of neurons in the firit layer greatly influences the performance of the network. The training time of this network is typically orders of magnitude smaller than that for the back propagation network as training is split up in two parts, both of which c:in be done quite fast. In our program the only teaching parameter is the learning rate. The calculation of training arid test errors are identical to the back propagation program. 23: 'I'he Kohonen self organising feature map r51. The Kohonen self organising feature map has two layers of neurons. The signals from input ncurons (also called inputs) are fed to every single ncuron. While learning (unsupervised) the network makes a two dimensional representation of the input space. The multi-dimensional input space is transfcrred to a two-dimensional grid using this network. Fig. 2 shows a Kohonen network that has two inputs and a grid (or feature map) consisting of 3 by 3 = 9 output neurons. I"P"tS Fig. 2.: A '2-3-3'Kohonen Nerwork. Learning in the Kohonen network is performed by selecting the winning neuron for a certain training input. This ncuron has the weights that most closely resemble the values of the inputs. The weights of this neuron as well as of its neighbours are updated to resemble like the input vector. The neighbours are dctermincd by a neighbourhood function. The radius of this neighbourhood function determines how many neurons around the winning neuron are included in the neighbourhood. There are several shapes possible for the neighbourhood function. A square shape, a bubble (circle) shape or a hexagonal shape are the ones usually used. In ordcr to train the network properly both the learning rate and the radius of the neighbourhood function should decrease in time. The initial weights of the network should be different from each other and are often set to random values in the range of the possible input values. fhe learning process consists of two phases. In the first phase, co"x leaning, the learning rate is quite Imge and the neighbourhood radius decreases from encompassing a large part of the grid to encompassing only the winning neuron itself (radius = 0) or the winning neuron and its closest neighbours (radius = 1). In the second phase the learning rate slowly decrcases to zero (this is usually the stopping criterion) while tlic neighbourhood radius remains 0 or  1. I'he learning rate as well as the neighbourhood radius can decrease by a number of functions; e.g. linearly, exponential. According to [ 5 ]  this is not very critical. The total number of training steps (= the number of limes a single input training pattern is presented to the network) is typically 500 times the numbcr of iietwork units or less, with the first phase of learning listing only about 1000 cyclcs. 2.4: I'he ARTl clustering algorithm [SI The ARTl network consists of two layers; a laycr to which the inputs are applied and a top layer of iieurons (outputs). The two layers are fully 64 interconnected with feedforward (from input-layer to top-layer) weights (the W-weights') as well as feedback weights (the 'V-weights'). The 'W-weights', or clustering weights, are used to select the winning output neuron (the cluster) and serve as the long term memory of the network. The winning neuron is the neuron with the maximum response (= weighted sum) for a certain input pattern. The 'v or vigilance weights' are used for the vigilance test and serve as the short term memory of the network. When a mining input is similar enough to the last pattem clustered with Ihe winning neuron it will be clustered with this neuron. Otherwise the neuron with the second largest response is tried. The required similarity is determined by the vigilance threshold. If the input does not cluster with any of the existing clusters (output neurons), then a new cluster Centre is made. The vigilance threshold (0 to 1) determines how similar two patterns should be bcrore they are clustered together. The larger the threshold, the less patterns are clustered together and therefore the more clusters will be made. When an input pattem is clustered with an existing or a new cluster centre, its weights are updated. The 'V' weights are set equal to this input vector and the 'W' weights are set to the normaliLed input vector. While Icarning this way, the network works as a 'follow the leader' system. Testing or recall of ARTl is done by feeding a pattern into the input layer and selecting the neuron with maximum response (the winning neuron) using the feedforward weights. The output of the network (i.e. the winning neuron) is the one whose weights are most similar (the smallest Hamming distance) to the input (test) pattem. Figure 3 shows an ART1 network with 4 inputs and 3 output neurons (clusters); a '4-3' network. Only the weights for one input neuron are shown in  the figure. Output 1;tyi.r I 1 Fig. 3 A '4-3'ARTI network. The ARTl network can only be u\ed for binary iiiput values. An enhanced version of the algorithm exists, ART2, that can handle continuous valued inputs 25: The Hopfield network [31 [41 [SI. The Ilopfield network is a fully connected single layer network. After an input pattem is presented to the network, it will converge by means of a state update rule until  i t  resides in a stable pattern. The state of a neuron is simply its activation (-1 or + I )  and the starting state of the network is just the input. The network can only handle bipolar inputs or binary inputs, with a slightly altered update rule. This is the test or recall procedure. The learning or storing is done simply by setting the weights in the network according to wij=s.*s., where wij is the weight bctwcen neurons i and j. and si, s, are the inputs of neurons i and j. Figure 4 shows a 3-neuron Hopfield network. 1 J  (-Weights I I \ In 1 Outputs Fig. 4 A 3-neuron Hopfield nehvork. Usually the weights from a neuron back to itself (i.e. all the weights wii) are set to zero. When an input is presented to rhe network, the output of the network will be the stable (trained) pattem that has the minimum Hamming distance from the input pattem. Bccause of the limited storage capacity, the Hopfield network is not very popular in pattem recognition applications. In [ I ]  a neural network system is used that consists of a single IIopfield network for every training pattern. There is virtually no such thing as training time for the Ik)plield network. The weights are calculated after the training patterns are presented and are set to their appropriate values. There is no updating of weights. The recall phase does take time because the network has to undergo several convergence steps before the network has found its stable state. Two 'versions' of the Hopfield network have been implemented. One program is just the ordinary IIopfield network. The other consists of a system of multiple Ilopfield network. After the network is trained, i t  generates as many separate Hopfield networks as thcrc are Lraining patterns and stores one training pattern in every one of them. In the recall phase, the test input is presented to each of the networks. The output (after convergence) of each network is compared with the input pattem and the final output is that output pattern which most closely 65 (smallest Hamming distance) resembles the input pattem. distorted: . o  1 2.6: The Bidirectional associative memory (BAM) network. Hopfield 0.006 0.00013 - 0.031 0.180 - The Bidirectional Associative Memory is quite similar to the Hopfield network. This type of network has two layers of neurons, which are fully interconnected. In the recall phase, a input (tcst) pattern is presented to one of Lhe two layers. After the recall (lest) phase, the network has come to a stable state so that each of the two layers has a stable pattem as output. The updating of the slates of the neurons (-l/+l) is done for the two layers alternatively, which means that there is a bidirectional data flow while the network is converging . The store (learning) procedure is similar to the Hopfield network except that the BAM network needs inputs to both layers. Figure 9 shows a BAM network which has a layer of three neurons (layer A) and a layer of two neurons (layer B). outputs Inputs Fig. 5 A 3 *2-neuron Bidireclional Associative Memory. For the BAM network, just like the Hopfielc network, the storage capacity is probably its mair limitation. From [8] it follows that an optimistic value of the maximum storage capacity is: min(m,n) where m,n are the number of  neurons in each of the layers. Thus at the most the smallest of m and r properties such as speed and generalisation capability (i.e. the performancc on distortcd input patterns). The training set consisted of ten digits, '0 to '9' in a 5 x 7 grid as shown below. 00100 01110 1 1 1 1 1  00010 1 1 1 1 1  01100 10001 00010 00110 10000 00100 00001 00100 01010 llll0 00100 00010 00010 10010 00001 00100 00100 00001 1 1 1 1 1  00001 00100 01000 10001 00010 10001 01110 1 1 1 1 1  01110 00010 01110 00110 1 1 1 1 1  01110 01110 01110 01000 00001 10001 10001 10001 10000 00010 100Ol l o o 0 1  1 0 0 0 1  . . . . . . . . . . . . . . . - . . . - . . . . 11110 00100 01110 01111 10001 10001 00100 10001 00001 10001 10001 00100 10001 00010 10001 O I l l O  00100 01110 01100 01110 The network architectures which were determined after various trial runs are: a 35x10~10 BP network, a 35x10~10 RBF network, a 2-dimensional 6x6 Kohonen network and a multiple Hopfield network consisting of 10 separate networks. The single Hopfield network and a 35x10 B A M  network were not able to learn all the training pattems. Due to the limited storage capacity of these networks only five pattems could be stored. Fig. 6 shows the cumulative training and test errors as well as the number of incorrectly classified facts for all the test sets. Since only one test set was made for each level of distortion, conclusions from this should be drawn carefully. For the Kohonen. ARTl and Hopfield networks no cumulative errors exist. pixels I BP I RBF 1 Kohonen I ARTl& I --I 0.103 0.742 - I 0 1  1 1  01 0 4 I 0.334 I 1.08 1 -  1 1  1 1  21 0 5 I 1.04 I 1.60 patterns can be stored in the network. A more conservative value is [I]: dmin(m,n). 3. Case studies 3.1 Character recognition It follows that the ARTl and the multiple Hopfield network give the least misclassifications on the test A simple comparative study was performed on the sets. These two networks also showed the best neural network simulators described in section 2. The performance in speed; learning took a few seconds as various simulation programs were compared on compared to 4 minutes for the RBF network, 10 Fig. 6 Cumulative errors (top leJt) and number of incorrectly classified patterns (bottom right) on training and test sets. 66 minutes for the BP network and about 20 minutes for the Kohonen network. In all, the ART1 showed the best performance on this simple test as it is less computationally expensive than the multiple Hopfield network. 3.2 Further applications A nuniber of projects are undertaken on the application of neural networks in the Department of Electrical Engineering of the University of Twente. These projects involve handwritten digit recognition, automatic recognition of registration plate numbers of vehicles, identification of the dynamic pattern of signatures, speech recognition, machine condition monitoring, signal source sepration, electric power load forecasting. automatic detection of spike-wave- complexes in EEG-signals. detection of cancer tumours in X-ray photographs, automatic convol of a metal sheet bending machine and process parameter identification in plastic production. 4. Conclusions This paper has described the implementation of neural network simulators in brief. It is obvious from the applictions that neural networks are suitable for a wide range of applications in business, science and engineering. Acknowledgements E. Vonk implemented the networks during his 10 weeks work experience training in the Knowledge- based Engineering Systems group. University of South Australia, Adelaide, in 1993. Thanks are due to the department of electrical engineering, University of Twente, for permitting to undertake this research work. References Chen, Liang-Chia, Fan, Ju-Yi, and Chen, Yung- Sheng, "A Modified High Speed Hopfield Neural Network And Its Character Recognition System", Proc. ISIC-91. pp. 135-140, (1991). Mirchandani, Gagan, "On Hidden Nodes for Neural Nets", IEEE Transactions on Circuits & Systems, Vol. 36, No. 5, pp. 661-6, May (1989). Hopfield, John. J., "Artificial Neural Networks", IEEE Circuits and Devices Magazine, p p ~  3-10, (1988). Hush, D.R. and IIome, B.G., "Progress in  Supervised Neural Networks. What'sNew Since Lipmann", IEEE Signal Processing Magazine, pp. 8-39, (1993). [SI Kohonen, Teuvo, "The Self-Organizing Map", Proceedings of the IEEE, Vol. 78, No. 9, [6] Moody. John and Darken, Christian J., "Fast Learning in Networks of Locally-Tuned Processing Units", Neural Computation, Vol. 1, 171 Musavi, M.T. ct al.. "On the Training of Radial Base Function Classifiers", Neural Networks, [8] Zurada, Jacck M., "(Introduction to) Artificial Neural Systems", West Publishing Company, St. Paul, (1 992). pp. 1464-1480, (1990). pp. 281-294. (1989). Vol. 5 ,  pp. 595-603. (1992). 67 

A Modified High Speed Hopfield Neural Network And Its Character Recognition System&quot;,

Artificial Neural Networks&quot;,

On Hidden Nodes for Neural Nets&quot;,

Progress in Supervised Neural Networks. What'sNew Since Lipmann&quot;,

The Self-Organizing Map&quot;,

https://research.utwente.nl/en/publications/1380f63a-dc5c-4578-a01c-0cdb7dae002e

Neural Network Applications

Abstract

Similar works

Full text

Available Versions

NARCIS

University of Twente Research Information