Connectivity and Performance Tradeoffs in the Cascade Correlation Learning Architecture

The Cascade Correlation [1] is a very flexible, efficient and fast algorithm for supervised learning. It incrementally builds the network by adding hidden units one at a time, until the desired input/output mapping is achieved. It connects all the previously installed units to the new unit being added. Consequently, each new unit in effect adds a new layer and the fan–in of the hidden and output units keeps on increasing as more units get added. The resulting structure could be hard to implement in VLSI, because the connections are irregular and the fan-in is unbounded. Moreover, the depth or the propagation delay through the resulting network is directly proportional to the number of units and can be excessive. We have modified the algorithm to generate networks with restricted fan-in and small depth (propagation delay) by controlling the connectivity. Our results reveal that there is a tradeoff between connectivity and other performance attributes like depth, total number of independent parameters, learning time, etc. When the number of inputs or outputs is small relative to the size of the training set, a higher connectivity usually leads to faster learning, and fewer independent parameters, but it also results in unbounded fan-in and depth. Strictly layered architectures with restricted connectivity, on the other hand, need more epochs to learn and use more parameters, but generate more regular structures, with smaller, limited fan-in and significantly smaller depth (propagation delay), and may be better suited for VLSI implementations. When the number of inputs or outputs is not very small compared to the size of the training set, however, a strictly layered topology is seen to yield an overall better performance

The Cascade Orthogonal Neural Network

In the paper new non-conventional growing neural network is proposed. It coincides with the Cascade- Correlation Learning Architecture structurally, but uses ortho-neurons as basic structure units, which can be adjusted using linear tuning procedures. As compared with conventional approximating neural networks proposed approach allows significantly to reduce time required for weight coefficients adjustment and the training dataset size

The Cascade Neo-Fuzzy Architecture and its Online Learning Algorithm

In the paper learning algorithm for adjusting weight coefficients of the Cascade Neo-Fuzzy Neural Network (CNFNN) in sequential mode is introduced. Concerned architecture has the similar structure with the Cascade-Correlation Learning Architecture proposed by S.E. Fahlman and C. Lebiere, but differs from it in type of artificial neurons. CNFNN consists of neo-fuzzy neurons, which can be adjusted using high-speed linear learning procedures. Proposed CNFNN is characterized by high learning rate, low size of learning sample and its operations can be described by fuzzy linguistic “if-then” rules providing “transparency” of received results, as compared with conventional neural networks. Using of online learning algorithm allows to process input data sequentially in real time mode

Improving the performance of cascade correlation neural networks on multimodal functions

Intrinsic qualities of the cascade correlation algorithm make it a popular choice for many researchers wishing to utilize neural networks. Problems arise when the outputs required are highly multimodal over the input domain. The mean squared error of the approximation increases significantly as the number of modes increases. By applying ensembling and early stopping, we show that this error can be reduced by a factor of three. We also present a new technique based on subdivision that we call patchworking. When used in combination with early stopping and ensembling the mean improvement in error is over 10 in some cases

A study of early stopping, ensembling, and patchworking for cascade correlation neural networks

The constructive topology of the cascade correlation algorithm makes it a popular choice for many researchers wishing to utilize neural networks. However, for multimodal problems, the mean squared error of the approximation increases significantly as the number of modes increases. The components of this error will comprise both bias and variance and we provide formulae for estimating these values from mean squared errors alone. We achieve a near threefold reduction in the overall error by using early stopping and ensembling. Also described is a new subdivision technique that we call patchworking. Patchworking, when used in combination with early stopping and ensembling, can achieve an order of magnitude improvement in the error. Also presented is an approach for validating the quality of a neural network’s training, without the explicit use of a testing dataset

Stacking-based Deep Neural Network: Deep Analytic Network on Convolutional Spectral Histogram Features

Stacking-based deep neural network (S-DNN), in general, denotes a deep neural network (DNN) resemblance in terms of its very deep, feedforward network architecture. The typical S-DNN aggregates a variable number of individually learnable modules in series to assemble a DNN-alike alternative to the targeted object recognition tasks. This work likewise devises an S-DNN instantiation, dubbed deep analytic network (DAN), on top of the spectral histogram (SH) features. The DAN learning principle relies on ridge regression, and some key DNN constituents, specifically, rectified linear unit, fine-tuning, and normalization. The DAN aptitude is scrutinized on three repositories of varying domains, including FERET (faces), MNIST (handwritten digits), and CIFAR10 (natural objects). The empirical results unveil that DAN escalates the SH baseline performance over a sufficiently deep layer.Comment: 5 page

Input window size and neural network predictors

Neural network approaches to time series prediction are briefly discussed, and the need to specify an appropriately sized input window identified. Relevant theoretical results from dynamic systems theory are briefly introduced, and heuristics for finding the correct embedding dimension, and hence window size, are discussed. The method is applied to two time series and the resulting generalisation performance of the trained feedforward neural network predictors is analysed. It is shown that the heuristics can provide useful information in defining the appropriate network architectur

Optical imaging of cloud-to-stratosphere/mesosphere lightning over the Amazon Basin (CS/LAB)

The purpose of the CS/LAB project was to obtain images of cloud to stratosphere lightning discharges from aboard NASA's DC-8 Airborne Laboratory while flying in the vicinity of thunderstorms over the Amazon Basin. We devised a low light level imaging package as an add-on experiment to an airborne Laboratory deployment to South America during May-June, 1993. We were not successful in obtaining the desired images during the South American deployment. However, in a follow up flight over the American Midwest during the night of July 8-9, 1993 we recorded nineteen examples of the events over intense thunderstorms. From the observations were estimated absolute brightness, terminal altitudes, flash duration, horizontal extents, emission volumes, and frequencies relative to negative and positive ground strokes