24,459 research outputs found
PROPOSED METHODOLOGY FOR OPTIMIZING THE TRAINING PARAMETERS OF A MULTILAYER FEED-FORWARD ARTIFICIAL NEURAL NETWORKS USING A GENETIC ALGORITHM
An artificial neural network (ANN), or shortly "neural network" (NN), is a powerful
mathematical or computational model that is inspired by the structure and/or
functional characteristics of biological neural networks. Despite the fact that ANN has
been developing rapidly for many years, there are still some challenges concerning
the development of an ANN model that performs effectively for the problem at hand.
ANN can be categorized into three main types: single layer, recurrent network and
multilayer feed-forward network. In multilayer feed-forward ANN, the actual
performance is highly dependent on the selection of architecture and training
parameters. However, a systematic method for optimizing these parameters is still an
active research area. This work focuses on multilayer feed-forward ANNs due to their
generalization capability, simplicity from the viewpoint of structure, and ease of
mathematical analysis. Even though, several rules for the optimization of multilayer
feed-forward ANN parameters are available in the literature, most networks are still
calibrated via a trial-and-error procedure, which depends mainly on the type of
problem, and past experience and intuition of the expert. To overcome these
limitations, there have been attempts to use genetic algorithm (GA) to optimize some
of these parameters. However most, if not all, of the existing approaches are focused
partially on the part of architecture and training parameters. On the contrary, the GAANN
approach presented here has covered most aspects of multilayer feed-forward
ANN in a more comprehensive way. This research focuses on the use of binaryencoded
genetic algorithm (GA) to implement efficient search strategies for the
optimal architecture and training parameters of a multilayer feed-forward ANN.
Particularly, GA is utilized to determine the optimal number of hidden layers, number
of neurons in each hidden layer, type of training algorithm, type of activation function
of hidden and output neurons, initial weight, learning rate, momentum term, and
epoch size of a multilayer feed-forward ANN. In this thesis, the approach has been
analyzed and algorithms that simulate the new approach have been mapped out
PROPOSED METHODOLOGY FOR OPTIMIZING THE TRAINING PARAMETERS OF A MULTILAYER FEED-FORWARD ARTIFICIAL NEURAL NETWORKS USING A GENETIC ALGORITHM
An artificial neural network (ANN), or shortly "neural network" (NN), is a powerful
mathematical or computational model that is inspired by the structure and/or
functional characteristics of biological neural networks. Despite the fact that ANN has
been developing rapidly for many years, there are still some challenges concerning
the development of an ANN model that performs effectively for the problem at hand.
ANN can be categorized into three main types: single layer, recurrent network and
multilayer feed-forward network. In multilayer feed-forward ANN, the actual
performance is highly dependent on the selection of architecture and training
parameters. However, a systematic method for optimizing these parameters is still an
active research area. This work focuses on multilayer feed-forward ANNs due to their
generalization capability, simplicity from the viewpoint of structure, and ease of
mathematical analysis. Even though, several rules for the optimization of multilayer
feed-forward ANN parameters are available in the literature, most networks are still
calibrated via a trial-and-error procedure, which depends mainly on the type of
problem, and past experience and intuition of the expert. To overcome these
limitations, there have been attempts to use genetic algorithm (GA) to optimize some
of these parameters. However most, if not all, of the existing approaches are focused
partially on the part of architecture and training parameters. On the contrary, the GAANN
approach presented here has covered most aspects of multilayer feed-forward
ANN in a more comprehensive way. This research focuses on the use of binaryencoded
genetic algorithm (GA) to implement efficient search strategies for the
optimal architecture and training parameters of a multilayer feed-forward ANN.
Particularly, GA is utilized to determine the optimal number of hidden layers, number
of neurons in each hidden layer, type of training algorithm, type of activation function
of hidden and output neurons, initial weight, learning rate, momentum term, and
epoch size of a multilayer feed-forward ANN. In this thesis, the approach has been
analyzed and algorithms that simulate the new approach have been mapped out
Limited Evaluation Cooperative Co-evolutionary Differential Evolution for Large-scale Neuroevolution
Many real-world control and classification tasks involve a large number of
features. When artificial neural networks (ANNs) are used for modeling these
tasks, the network architectures tend to be large. Neuroevolution is an
effective approach for optimizing ANNs; however, there are two bottlenecks that
make their application challenging in case of high-dimensional networks using
direct encoding. First, classic evolutionary algorithms tend not to scale well
for searching large parameter spaces; second, the network evaluation over a
large number of training instances is in general time-consuming. In this work,
we propose an approach called the Limited Evaluation Cooperative
Co-evolutionary Differential Evolution algorithm (LECCDE) to optimize
high-dimensional ANNs.
The proposed method aims to optimize the pre-synaptic weights of each
post-synaptic neuron in different subpopulations using a Cooperative
Co-evolutionary Differential Evolution algorithm, and employs a limited
evaluation scheme where fitness evaluation is performed on a relatively small
number of training instances based on fitness inheritance. We test LECCDE on
three datasets with various sizes, and our results show that cooperative
co-evolution significantly improves the test error comparing to standard
Differential Evolution, while the limited evaluation scheme facilitates a
significant reduction in computing time
Adaptive Momentum for Neural Network Optimization
In this thesis, we develop a novel and efficient algorithm for optimizing neural networks inspired by a recently proposed geodesic optimization algorithm. Our algorithm, which we call Stochastic Geodesic Optimization (SGeO), utilizes an adaptive coefficient on top of Polyaks Heavy Ball method effectively controlling the amount of weight put on the previous update to the parameters based on the change of direction in the optimization path. Experimental results on strongly convex functions with Lipschitz gradients and deep Autoencoder benchmarks show that SGeO reaches lower errors than established first-order methods and competes well with lower or similar errors to a recent second-order method called K-FAC (Kronecker-Factored Approximate Curvature). We also incorporate Nesterov style lookahead gradient into our algorithm (SGeO-N) and observe notable improvements. We believe that our research will open up new directions for high-dimensional neural network optimization where combining the efficiency of first-order methods and the effectiveness of second-order methods proves a promising avenue to explore
- …