Neural networks in geophysical applications

Neural networks are increasingly popular in geophysics. Because they are universal approximators, these tools can approximate any continuous function with an arbitrary precision. Hence, they may yield important contributions to finding solutions to a variety of geophysical applications. However, knowledge of many methods and techniques recently developed to increase the performance and to facilitate the use of neural networks does not seem to be widespread in the geophysical community. Therefore, the power of these tools has not yet been explored to their full extent. In this paper, techniques are described for faster training, better overall performance, i.e., generalization,and the automatic estimation of network size and architecture

Techniques of replica symmetry breaking and the storage problem of the McCulloch-Pitts neuron

In this article the framework for Parisi's spontaneous replica symmetry breaking is reviewed, and subsequently applied to the example of the statistical mechanical description of the storage properties of a McCulloch-Pitts neuron. The technical details are reviewed extensively, with regard to the wide range of systems where the method may be applied. Parisi's partial differential equation and related differential equations are discussed, and a Green function technique introduced for the calculation of replica averages, the key to determining the averages of physical quantities. The ensuing graph rules involve only tree graphs, as appropriate for a mean-field-like model. The lowest order Ward-Takahashi identity is recovered analytically and is shown to lead to the Goldstone modes in continuous replica symmetry breaking phases. The need for a replica symmetry breaking theory in the storage problem of the neuron has arisen due to the thermodynamical instability of formerly given solutions. Variational forms for the neuron's free energy are derived in terms of the order parameter function x(q), for different prior distribution of synapses. Analytically in the high temperature limit and numerically in generic cases various phases are identified, among them one similar to the Parisi phase in the Sherrington-Kirkpatrick model. Extensive quantities like the error per pattern change slightly with respect to the known unstable solutions, but there is a significant difference in the distribution of non-extensive quantities like the synaptic overlaps and the pattern storage stability parameter. A simulation result is also reviewed and compared to the prediction of the theory.Comment: 103 Latex pages (with REVTeX 3.0), including 15 figures (ps, epsi, eepic), accepted for Physics Report

Meta-Heuristic Optimization Methods for Quaternion-Valued Neural Networks

In recent years, real-valued neural networks have demonstrated promising, and often striking, results across a broad range of domains. This has driven a surge of applications utilizing high-dimensional datasets. While many techniques exist to alleviate issues of high-dimensionality, they all induce a cost in terms of network size or computational runtime. This work examines the use of quaternions, a form of hypercomplex numbers, in neural networks. The constructed networks demonstrate the ability of quaternions to encode high-dimensional data in an efficient neural network structure, showing that hypercomplex neural networks reduce the number of total trainable parameters compared to their real-valued equivalents. Finally, this work introduces a novel training algorithm using a meta-heuristic approach that bypasses the need for analytic quaternion loss or activation functions. This algorithm allows for a broader range of activation functions over current quaternion networks and presents a proof-of-concept for future work

Techniques of replica symmetry breaking and the storage problem of the McCulloch-Pitts neuron

In this article the framework for Parisi's spontaneous replica symmetry breaking is reviewed, and subsequently applied to the example of the statistical mechanical description of the storage properties of a McCulloch-Pitts neuron. The technical details are reviewed extensively, with regard to the wide range of systems where the method may be applied. Parisi's partial differential equation and related differential equations are discussed, and a Green function technique introduced for the calculation of replica averages, the key to determining the averages of physical quantities. The ensuing graph rules involve only tree graphs, as appropriate for a mean-field-like model. The lowest order Ward-Takahashi identity is recovered analytically and is shown to lead to the Goldstone modes in continuous replica symmetry breaking phases. The need for a replica symmetry breaking theory in the storage problem of the neuron has arisen due to the thermodynamical instability of formerly given solutions. Variational forms for the neuron's free energy are derived in terms of the order parameter function x(q), for different prior distribution of synapses. Analytically in the high temperature limit and numerically in generic cases various phases are identified, among them one similar to the Parisi phase in the Sherrington-Kirkpatrick model. Extensive quantities like the error per pattern change slightly with respect to the known unstable solutions, but there is a significant difference in the distribution of non-extensive quantities like the synaptic overlaps and the pattern storage stability parameter. A simulation result is also reviewed and compared to the prediction of the theory.Comment: 103 Latex pages (with REVTeX 3.0), including 15 figures (ps, epsi, eepic), accepted for Physics Report

A Theory of Neural Computation with Clifford Algebras

The present thesis introduces Clifford Algebra as a framework for neural computation. Neural computation with Clifford algebras is model-based. This principle is established by constructing Clifford algebras from quadratic spaces. Then the subspace grading inherent to any Clifford algebra is introduced. The above features of Clifford algebras are then taken as motivation for introducing the Basic Clifford Neuron (BCN). As a second type of Clifford neuron the Spinor Clifford Neuron is presented. A systematic basis for Clifford neural computation is provided by the important notions of isomorphic Clifford neurons and isomorphic representations. After the neuron level is established, the discussion continues with (Spinor) Clifford Multilayer Perceptrons. First, (Spinor) Clifford Multilayer Perceptrons with real-valued activation functions ((S)CMLPs) are studied. A generic Backpropagation algorithm for CMLPs is derived. Also, universal approximation theorems for (S)CMLPs are presented. Finally, CMLPs with Clifford-valued activation functions are studied

Novel Artificial Neural Network Application for Prediction of Inverse Kinematics of Robot Manipulator

The robot control problem can be divided into two main areas, kinematics control (the coordination of the links of kinematics chain to produce desire motion of the robot), and dynamic control (driving the actuator of the mechanism to follow the commanded position velocities). In general the control strategies used in robot involves position coordination in Cartesian space by direct or indirect kinematics method. Inverse kinematics comprises the computation need to find the join angles for a given Cartesian position and orientation of the end effectors. This computation is fundamental to control of robot arms but it is very difficult to calculate an inverse kinematics solution of robot manipulator. For this solution most industrial robot arms are designed by using a non-linear algebraic computation to finding the inverse kinematics solution. From the literature it is well described that there is no unique solution for the inverse kinematics. That is why it is significant to apply an artificial neural network models. Here structured artificial neural network (ANN) models an approach has been proposed to control the motion of robot manipulator. In these work two types of ANN models were used. The first kind ANN model is MLP (multi-layer perceptrons) which was famous as back propagation neural network model. In this network gradient descent type of learning rules are applied. The second kind of ANN model is PPN (polynomial poly-processor neural network) where polynomial equation was used. Here, work has been undertaken to find the best ANN configuration for the problem. It was found that between MLP and PPN, MLP gives better result as compared to PPN by considering average percentage error, as the performance index