Generalised cellular neural networks (GCNNs) constructed using particle swarm optimisation for spatio-temporal evolutionary pattern identification

Particle swarm optimization (PSO) is introduced to implement a new constructive learning algorithm for training generalized cellular neural networks (GCNNs) for the identification of spatio-temporal evolutionary (STE) systems. The basic idea of the new PSO-based learning algorithm is to successively approximate the desired signal by progressively pursuing relevant orthogonal projections. This new algorithm will thus be referred to as the orthogonal projection pursuit (OPP) algorithm, which is in mechanism similar to the conventional projection pursuit approach. A novel two-stage hybrid training scheme is proposed for constructing a parsimonious GCNN model. In the first stage, the orthogonal projection pursuit algorithm is applied to adaptively and successively augment the network, where adjustable parameters of the associated units are optimized using a particle swarm optimizer. The resultant network model produced at the first stage may be redundant. In the second stage, a forward orthogonal regression (FOR) algorithm, aided by mutual information estimation, is applied to re. ne and improve the initially trained network. The effectiveness and performance of the proposed method is validated by applying the new modeling framework to a spatio-temporal evolutionary system identification problem

Convergence of Discrete-Time Cellular Neural Networks with Application to Image Processing

The paper considers a class of discrete-time cellular neural networks (DT-CNNs) obtained by applying Euler's discretization scheme to standard CNNs. Let T be the DT-CNN interconnection matrix which is defined by the feedback cloning template. The paper shows that a DT-CNN is convergent, i.e. each solution tends to an equilibrium point, when T is symmetric and, in the case where T + En is not positive-semidefinite, the step size of Euler's discretization scheme does not exceed a given bound (En is the n × n unit matrix). It is shown that two relevant properties hold as a consequence of the local and space-invariant interconnecting structure of a DT-CNN, namely: (1) the bound on the step size can be easily estimated via the elements of the DT-CNN feedback cloning template only; (2) the bound is independent of the DT-CNN dimension. These two properties make DT-CNNs very effective in view of computer simulations and for the practical applications to high-dimensional processing tasks. The obtained results are proved via Lyapunov approach and LaSalle's Invariance Principle in combination with some fundamental inequalities enjoyed by the projection operator on a convex set. The results are compared with previous ones in the literature on the convergence of DT-CNNs and also with those obtained for different neural network models as the Brain-State-in-a-Box model. Finally, the results on convergence are illustrated via the application to some relevant 2D and 1D DT-CNNs for image processing tasks

SIRENA: A CAD environment for behavioural modelling and simulation of VLSI cellular neural network chips

This paper presents SIRENA, a CAD environment for the simulation and modelling of mixed-signal VLSI parallel processing chips based on cellular neural networks. SIRENA includes capabilities for: (a) the description of nominal and non-ideal operation of CNN analogue circuitry at the behavioural level; (b) performing realistic simulations of the transient evolution of physical CNNs including deviations due to second-order effects of the hardware; and, (c) evaluating sensitivity figures, and realize noise and Monte Carlo simulations in the time domain. These capabilities portray SIRENA as better suited for CNN chip development than algorithmic simulation packages (such as OpenSimulator, Sesame) or conventional neural networks simulators (RCS, GENESIS, SFINX), which are not oriented to the evaluation of hardware non-idealities. As compared to conventional electrical simulators (such as HSPICE or ELDO-FAS), SIRENA provides easier modelling of the hardware parasitics, a significant reduction in computation time, and similar accuracy levels. Consequently, iteration during the design procedure becomes possible, supporting decision making regarding design strategies and dimensioning. SIRENA has been developed using object-oriented programming techniques in C, and currently runs under the UNIX operating system and X-Windows framework. It employs a dedicated high-level hardware description language: DECEL, fitted to the description of non-idealities arising in CNN hardware. This language has been developed aiming generality, in the sense of making no restrictions on the network models that can be implemented. SIRENA is highly modular and composed of independent tools. This simplifies future expansions and improvements.Comisión Interministerial de Ciencia y Tecnología TIC96-1392-C02-0

Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks

Future wireless networks have a substantial potential in terms of supporting a broad range of complex compelling applications both in military and civilian fields, where the users are able to enjoy high-rate, low-latency, low-cost and reliable information services. Achieving this ambitious goal requires new radio techniques for adaptive learning and intelligent decision making because of the complex heterogeneous nature of the network structures and wireless services. Machine learning (ML) algorithms have great success in supporting big data analytics, efficient parameter estimation and interactive decision making. Hence, in this article, we review the thirty-year history of ML by elaborating on supervised learning, unsupervised learning, reinforcement learning and deep learning. Furthermore, we investigate their employment in the compelling applications of wireless networks, including heterogeneous networks (HetNets), cognitive radios (CR), Internet of things (IoT), machine to machine networks (M2M), and so on. This article aims for assisting the readers in clarifying the motivation and methodology of the various ML algorithms, so as to invoke them for hitherto unexplored services as well as scenarios of future wireless networks.Comment: 46 pages, 22 fig

Convergence of Neural Networks with a Class of Real Memristors with Rectifying Characteristics

The paper considers a neural network with a class of real extended memristors obtained via the parallel connection of an ideal memristor and a nonlinear resistor. The resistor has the same rectifying characteristic for the current as that used in relevant models in the literature to account for diode-like effects at the interface between the memristor metal and insulating material. The paper proves some fundamental results on the trajectory convergence of this class of real memristor neural networks under the assumption that the interconnection matrix satisfies some symmetry conditions. First of all, the paper shows that, while in the case of neural networks with ideal memristors, it is possible to explicitly find functions of the state variables that are invariants of motions, the same functions can be used as Lyapunov functions that decrease along the trajectories in the case of real memristors with rectifying characteristics. This fundamental property is then used to study convergence by means of a reduction-of-order technique in combination with a Lyapunov approach. The theoretical predictions are verified via numerical simulations, and the convergence results are illustrated via the applications of real memristor neural networks to the solution of some image processing tasks in real time

Lattice dynamical wavelet neural networks implemented using particle swarm optimisation for spatio-temporal system identification

Starting from the basic concept of coupled map lattices, a new family of adaptive wavelet neural networks, called lattice dynamical wavelet neural networks (LDWNN), is introduced for spatiotemporal system identification, by combining an efficient wavelet representation with a coupled map lattice model. A new orthogonal projection pursuit (OPP) method, coupled with a particle swarm optimisation (PSO) algorithm, is proposed for augmenting the proposed network. A novel two-stage hybrid training scheme is developed for constructing a parsimonious network model. In the first stage, by applying the orthogonal projection pursuit algorithm, significant wavelet-neurons are adaptively and successively recruited into the network, where adjustable parameters of the associated waveletneurons are optimised using a particle swarm optimiser. The resultant network model, obtained in the first stage, may however be redundant. In the second stage, an orthogonal least squares (OLS) algorithm is then applied to refine and improve the initially trained network by removing redundant wavelet-neurons from the network. The proposed two-stage hybrid training procedure can generally produce a parsimonious network model, where a ranked list of wavelet-neurons, according to the capability of each neuron to represent the total variance in the system output signal is produced. Two spatio-temporal system identification examples are presented to demonstrate the performance of the proposed new modelling framework

Controller design for synchronization of an array of delayed neural networks using a controllable

This is the post-print version of the Article - Copyright @ 2011 ElsevierIn this paper, a controllable probabilistic particle swarm optimization (CPPSO) algorithm is introduced based on Bernoulli stochastic variables and a competitive penalized method. The CPPSO algorithm is proposed to solve optimization problems and is then applied to design the memoryless feedback controller, which is used in the synchronization of an array of delayed neural networks (DNNs). The learning strategies occur in a random way governed by Bernoulli stochastic variables. The expectations of Bernoulli stochastic variables are automatically updated by the search environment. The proposed method not only keeps the diversity of the swarm, but also maintains the rapid convergence of the CPPSO algorithm according to the competitive penalized mechanism. In addition, the convergence rate is improved because the inertia weight of each particle is automatically computed according to the feedback of fitness value. The efficiency of the proposed CPPSO algorithm is demonstrated by comparing it with some well-known PSO algorithms on benchmark test functions with and without rotations. In the end, the proposed CPPSO algorithm is used to design the controller for the synchronization of an array of continuous-time delayed neural networks.This research was partially supported by the National Natural Science Foundation of PR China (Grant No 60874113), the Research Fund for the Doctoral Program of Higher Education (Grant No 200802550007), the Key Creative Project of Shanghai Education Community (Grant No 09ZZ66), the Key Foundation Project of Shanghai(Grant No 09JC1400700), the Engineering and Physical Sciences Research Council EPSRC of the U.K. under Grant No. GR/S27658/01, an International Joint Project sponsored by the Royal Society of the U.K., and the Alexander von Humboldt Foundation of Germany

Complete Stability of Neural Networks With Extended Memristors

The article considers a large class of delayed neural networks (NNs) with extended memristors obeying the Stanford model. This is a widely used and popular model that accurately describes the switching dynamics of real nonvolatile memristor devices implemented in nanotechnology. The article studies via the Lyapunov method complete stability (CS), i.e., convergence of trajectories in the presence of multiple equilibrium points (EPs), for delayed NNs with Stanford memristors. The obtained conditions for CS are robust with respect to variations of the interconnections and they hold for any value of the concentrated delay. Moreover, they can be checked either numerically, via a linear matrix inequality (LMI), or analytically, via the concept of Lyapunov diagonally stable (LDS) matrices. The conditions ensure that at the end of the transient capacitor voltages and NN power vanish. In turn, this leads to advantages in terms of power consumption. This notwithstanding, the nonvolatile memristors can retain the result of computation in accordance with the in-memory computing principle. The results are verified and illustrated via numerical simulations. From a methodological viewpoint, the article faces new challenges to prove CS since due to the presence of nonvolatile memristors the NNs possess a continuum of nonisolated EPs. Also, for physical reasons, the memristor state variables are constrained to lie in some given intervals so that the dynamics of the NNs need to be modeled via a class of differential inclusions named differential variational inequalities