An instruction systolic array architecture for multiple neural network types

Modern electronic systems, especially sensor and imaging systems, are beginning to incorporate their own neural network subsystems. In order for these neural systems to learn in real-time they must be implemented using VLSI technology, with as much of the learning processes incorporated on-chip as is possible. The majority of current VLSI implementations literally implement a series of neural processing cells, which can be connected together in an arbitrary fashion. Many do not perform the entire neural learning process on-chip, instead relying on other external systems to carry out part of the computation requirements of the algorithm. The work presented here utilises two dimensional instruction systolic arrays in an attempt to define a general neural architecture which is closer to the biological basis of neural networks - it is the synapses themselves, rather than the neurons, that have dedicated processing units. A unified architecture is described which can be programmed at the microcode level in order to facilitate the processing of multiple neural network types. An essential part of neural network processing is the neuron activation function, which can range from a sequential algorithm to a discrete mathematical expression. The architecture presented can easily carry out the sequential functions, and introduces a fast method of mathematical approximation for the more complex functions. This can be evaluated on-chip, thus implementing the entire neural process within a single system. VHDL circuit descriptions for the chip have been generated, and the systolic processing algorithms and associated microcode instruction set for three different neural paradigms have been designed. A software simulator of the architecture has been written, giving results for several common applications in the field

Applications of neural networks to control systems

Tese de dout., Engenharia Electrónica, School of Electronic Engineering Science, Univ. of Wales, Bangor, 1992This work investigates the applicability of artificial neural networks to control systems. The following properties of neural networks are identified as of major interest to this field: their ability to implement nonlinear mappings, their massively parallel structure and their capacity to adapt. Exploiting the first feature, a new method is proposed for PID autotuning. Based on integral measures of the open or closed loop step response, multilayer perceptrons (MLPs) are used to supply PID parameter values to a standard PID controller. Before being used on-line, the MLPs are trained offline, to provide PID parameter values based on integral performance criteria. Off-line simulations, where a plant with time-varying parameters and time varying transfer function is considered, show that well damped responses are obtained. The neural PID autotuner is subsequently implemented in real-time. Extensive experimentation confirms the good results obtained in the off-line simulations. To reduce the training time incurred when using the error back-propagation algorithm, three possibilities are investigated. A comparative study of higherorder methods of optimization identifies the Levenberg-Marquardt (LM)algorithm as the best method. When used for function approximation purposes, the neurons in the output layer of the MLPs have a linear activation function. Exploiting this linearity, the standard training criterion can be replaced by a new, yet equivalent, criterion. Using the LM algorithm to minimize this new criterion, together with an alternative form of Jacobian matrix, a new learning algorithm is obtained. This algorithm is subsequently parallelized. Its main blocks of computation are identified, separately parallelized, and finally connected together. The training time of MLPs is reduced by a factor greater than 70 executing the new learning algorithm on 7 Inmos transputers

Design and implementation of a digital neural processor for detection applications

The main focus of this research is to develop a digital neural network (processor) and hardware (VLSI) implementation of the same for detection applications, for example in the distance protection of power transmission lines. Using a hardware neural processor will improve the protection system performance over software implementations in terms of speed of operation, response time for faults etc. The main aspects of this research are software design, performance analysis, hardware design and hardware implementation of the digital neural processor. The software design is carried out by developing an object oriented neural network simulator with backpropagation training using C++ language. A preliminary analysis shows that the inputs to the neural network need to be preprocessed. Two filters have been developed for this purpose, based on the analysis of the training data available. The performance analysis involves studying quantization effects (determination of precision requirements) in the network. -- The hardware design involves design of the neural network and the preprocessors. The neural processor consists of three types of processing elements (neurons): input, hidden and output neurons. The input neurons form the input layer of the processor which receive input from the preprocessors. The input layer can be configured to directly receive external input by changing the mode of operation. The output layer gives the signal to the relay for tripping the line under fault. Each neuron consists of datapath and local control unit. Datapath consists of the components for forward and backward passes of the processor and the register file. The local control unit controls the flow of data within a neuron and co-ordinates with the global control unit which controls the flow of data between layers. The neurons and the layers are pipelined for improving the throughput of the processor. The neural processor and the filters are implemented in VLSI using hardware description language (VHDL) and Synopsys / Cadence CAD tools. All the components are individually verified and tested for their functionality and implemented using 0.5 μ CMOS technology

Coding Strategies for Genetic Algorithms and Neural Nets

The interaction between coding and learning rules in neural nets (NNs), and between coding and genetic operators in genetic algorithms (GAs) is discussed. The underlying principle advocated is that similar things in "the world" should have similar codes. Similarity metrics are suggested for the coding of images and numerical quantities in neural nets, and for the coding of neural network structures in genetic algorithms. A principal component analysis of natural images yields receptive fields resembling horizontal and vertical edge and bar detectors. The orientation sensitivity of the "bar detector" components is found to match a psychophysical model, suggesting that the brain may make some use of principal components in its visual processing. Experiments are reported on the effects of different input and output codings on the accuracy of neural nets handling numeric data. It is found that simple analogue and interpolation codes are most successful. Experiments on the coding of image data demonstrate the sensitivity of final performance to the internal structure of the net. The interaction between the coding of the target problem and reproduction operators of mutation and recombination in GAs are discussed and illustrated. The possibilities for using GAs to adapt aspects of NNs are considered. The permutation problem, which affects attempts to use GAs both to train net weights and adapt net structures, is illustrated and methods to reduce it suggested. Empirical tests using a simulated net design problem to reduce evaluation times indicate that the permutation problem may not be as severe as has been thought, but suggest the utility of a sorting recombination operator, that matches hidden units according to the number of connections they have in common. A number of experiments using GAs to design network structures are reported, both to specify a net to be trained from random weights, and to prune a pre-trained net. Three different coding methods are tried, and various sorting recombination operators evaluated. The results indicate that appropriate sorting can be beneficial, but the effects are problem-dependent. It is shown that the GA tends to overfit the net to the particular set of test criteria, to the possible detriment of wider generalisation ability. A method of testing the ability of a GA to make progress in the presence of noise, by adding a penalty flag, is described

The development of artificial neural networks for the analysis of market research and electronic nose data

This thesis details research carried out into the application of unsupervised neural network and statistical clustering techniques to market research interview survey analysis. The objective of the research was to develop mathematical mechanisms to locate and quantify internal clusters within the data sets with definite commonality. As the data sets being used were binary, this commonality was expressed in terms of identical question answers. Unsupervised neural network paradigms are investigated, along with statistical clustering techniques. The theory of clustering in a binary space is also looked at. Attempts to improve the clarity of output of Self-Organising Maps (SOM) consisted of several stages of investigation culminating in the conception of the Interrogative Memory Structure (lMS). IMS proved easy to use, fast in operation and consistently produced results with the highest degree of commonality when tested against SOM, Adaptive Resonance Theory (ART!) and FASTCLUS. ARTl performed well when clusters were measured using general metrics. During the course of the research a supervised technique, the Vector Memory Array (VMA), was developed. VMA was tested against Back Propagation (BP) (using data sets provided by the Warwick electronic nose project) and consistently produced higher classification accuracies. The main advantage of VMA is its speed of operation - in testing it produced results in minutes compared to hours for the BP method, giving speed increases in the region of 100: 1