A Vector Matrix Real Time Backpropagation Algorithm for Recurrent neural networks That Approximate Multi-valued Periodic Functions

Unlike feedforward neural networks (FFNN) which can act as universal function approximators, recursive, or recurrent, neural networks can act as universal approximators for multi-valued functions. In this paper, a real time recursive backpropagation (RTRBP) algorithm in a vector matrix form is developed for a two-layer globally recursive neural network that has multiple delays in its feedback path. This algorithm has been evaluated on two GRNNs that approximate both an analytic and nonanalytic periodic multi-valued function that a feedforward neural network is not capable of approximating

Initialization-Dependent Sample Complexity of Linear Predictors and Neural Networks

We provide several new results on the sample complexity of vector-valued linear predictors (parameterized by a matrix), and more generally neural networks. Focusing on size-independent bounds, where only the Frobenius norm distance of the parameters from some fixed reference matrix $W_0$ is controlled, we show that the sample complexity behavior can be surprisingly different than what we may expect considering the well-studied setting of scalar-valued linear predictors. This also leads to new sample complexity bounds for feed-forward neural networks, tackling some open questions in the literature, and establishing a new convex linear prediction problem that is provably learnable without uniform convergence.Comment: 30 page

Eigenvalues of block structured asymmetric random matrices

We study the spectrum of an asymmetric random matrix with block structured variances. The rows and columns of the random square matrix are divided into $D$ partitions with arbitrary size (linear in $N$). The parameters of the model are the variances of elements in each block, summarized in $g\in\mathbb{R}^{D\times D}_+$. Using the Hermitization approach and by studying the matrix-valued Stieltjes transform we show that these matrices have a circularly symmetric spectrum, we give an explicit formula for their spectral radius and a set of implicit equations for the full density function. We discuss applications of this model to neural networks

Global stability of Clifford-valued Takagi-Sugeno fuzzy neural networks with time-varying delays and impulses

summary:In this study, we consider the Takagi-Sugeno (T-S) fuzzy model to examine the global asymptotic stability of Clifford-valued neural networks with time-varying delays and impulses. In order to achieve the global asymptotic stability criteria, we design a general network model that includes quaternion-, complex-, and real-valued networks as special cases. First, we decompose the $n$-dimensional Clifford-valued neural network into $2^mn$-dimensional real-valued counterparts in order to solve the noncommutativity of Clifford numbers multiplication. Then, we prove the new global asymptotic stability criteria by constructing an appropriate Lyapunov-Krasovskii functionals (LKFs) and employing Jensen's integral inequality together with the reciprocal convex combination method. All the results are proven using linear matrix inequalities (LMIs). Finally, a numerical example is provided to show the effectiveness of the achieved results

Global μ

The impulsive complex-valued neural networks with three kinds of time delays including leakage delay, discrete delay, and distributed delay are considered. Based on the homeomorphism mapping principle of complex domain, a sufficient condition for the existence and uniqueness of the equilibrium point of the addressed complex-valued neural networks is proposed in terms of linear matrix inequality (LMI). By constructing appropriate Lyapunov-Krasovskii functionals, and employing the free weighting matrix method, several delay-dependent criteria for checking the global μ-stability of the complex-valued neural networks are established in LMIs. As direct applications of these results, several criteria on the exponential stability, power-stability, and log-stability are obtained. Two examples with simulations are provided to demonstrate the effectiveness of the proposed criteria

Global μ

The complex-valued neural networks with unbounded time-varying delays are considered. By constructing appropriate Lyapunov-Krasovskii functionals, and employing the free weighting matrix method, several delay-dependent criteria for checking the global μ-stability of the addressed complex-valued neural networks are established in linear matrix inequality (LMI), which can be checked numerically using the effective LMI toolbox in MATLAB. Two examples with simulations are given to show the effectiveness and less conservatism of the proposed criteria

Counterexample of a Claim Pertaining to the Synthesis of a Recurrent Neural Network

Recurrent neural networks have received much attention due to their nonlinear dynamic behavior. One such type of dynamic behavior is that of setting a fixed stable state. This paper shows a counterexample to the claim of A.N. Michel et al. (IEEE Control Systems Magazine, vol. 15, pp. 52-65, Jun. 1995), that sparse constraints on the interconnecting structure for a given neural network are usually expressed as constraints which require that pre-determined elements of T [a real n×n matrix acting on a real n-vector valued function] be zero , for the synthesis of sparsely interconnected recurrent neural networks

Operator recurrent neural network approach to an inverse problem for the wave equation

In my mathematics master's thesis we dive into the wave equation and its inverse problem and try to solve it with neural networks we create in Python. There are different types of artificial neural networks. The basic structure is that there are several layers and each layer contains neurons. The input goes to all the neurons in the first layer, the neurons do calculations and send the output to all the neurons in the next layer. In this way, the input data goes through all the neurons and changes and the last layer outputs this changed data. In our code we use operator recurrent neural network. The biggest difference between the standard neural network and the operator recurrent neural network is, that instead of matrix-vector multiplications we use matrix-matrix multiplications in the neurons. We teach the neural networks for a certain number of times with training data and then we check how well they learned with test data. It is up to us how long and how far we teach the networks. Easy criterion would be when a neural network has learned the inversion completely, but it takes a lot of time and might never happen. So we settle for a situation when the error, the difference between the actual inverse and the inverse calculated by the neural network, is as small as we wanted. We start the coding by studying the matrix inversion. The idea is to teach the neural networks to do the inversion of a given 2-by-2 real valued matrix. First we deal with networks that don't have the activation function ReLU in their layers. We seek a learning rate, a small constant, that speeds up the learning of a neural network the most. After this we start comparing networks that don't have ReLU layers to networks that do have ReLU layers. The hypothesis is that ReLU assists neural networks to learn quicker. After this we study the one-dimensional wave equation and we calculate its general form of solution. The inverse problem of the wave equation is to recover wave speed c(x) when we have boundary terms. Inverse problems in general do not often have a unique solution, but in real life if we have measured data and some additional a priori information, it is possible to find a unique solution. In our case we do know that the inverse problem of the wave equation has a unique solution. When coding the inverse problem of the wave equation we use the same approach as with the matrix inversion. First we seek the best learning rate and then start to compare neural networks with and without ReLU layers. The hypothesis once again is that ReLU supports the learning of the neural networks. This turns out to be true and happens more clearly with wave equation than with matrix inversion. All the teaching was run on one computer. There is a chance to get even better results if a more powerful computer is used