Connectionist Learning Based Numerical Solution of Differential Equations

It is well known that the differential equations are back bone of different physical systems. Many real world problems of science and engineering may be modeled by various ordinary or partial differential equations. These differential equations may be solved by different approximate methods such as Euler, Runge-Kutta, predictor-corrector, finite difference, finite element, boundary element and other numerical techniques when the problems cannot be solved by exact/analytical methods. Although these methods provide good approximations to the solution, they require a discretization of the domain via meshing, which may be challenging in two or higher dimension problems. These procedures provide solutions at the pre-defined points and computational complexity increases with the number of sampling points.In recent decades, various machine intelligence methods in particular connectionist learning or Artificial Neural Network (ANN) models are being used to solve a variety of real-world problems because of its excellent learning capacity. Recently, a lot of attention has been given to use ANN for solving differential equations. The approximate solution of differential equations by ANN is found to be advantageous but it depends upon the ANN model that one considers. Here our target is to solve ordinary as well as partial differential equations using ANN. The approximate solution of differential equations by ANN method has various inherent benefits in comparison with other numerical methods such as (i) the approximate solution is differentiable in the given domain, (ii) computational complexity does not increase considerably with the increase in number of sampling points and dimension of the problem, (iii) it can be applied to solve linear as well as non linear Ordinary Differential Equations (ODEs) and Partial Differential Equations (PDEs). Moreover, the traditional numerical methods are usually iterative in nature, where we fix the step size before the start of the computation. After the solution is obtained, if we want to know the solution in between steps then again the procedure is to be repeated from initial stage. ANN may be one of the ways where we may overcome this repetition of iterations. Also, we may use it as a black box to get numerical results at any arbitrary point in the domain after training of the model.Few authors have solved ordinary and partial differential equations by combining the feed forward neural network and optimization technique. As said above that the objective of this thesis is to solve various types of ODEs and PDEs using efficient neural network. Algorithms are developed where no desired values are known and the output of the model can be generated by training only. The architectures of the existing neural models are usually problem dependent and the number of nodes etc. are taken by trial and error method. Also, the training depends upon the weights of the connecting nodes. In general, these weights are taken as random number which dictates the training. In this investigation, firstly a new method viz. Regression Based Neural Network (RBNN) has been developed to handle differential equations. In RBNN model, the number of nodes in hidden layer may be fixed by using the regression method. For this, the input and output data are fitted first with various degree polynomials using regression analysis and the coefficients involved are taken as initial weights to start with the neural training. Fixing of the hidden nodes depends upon the degree of the polynomial.We have considered RBNN model for solving different ODEs with initial/boundary conditions. Feed forward neural model and unsupervised error back propagation algorithm have been used for minimizing the error function and modification of the parameters (weights and biases) without use of any optimization technique. Next, single layer Functional Link Artificial Neural Network (FLANN) architecture has been developed for solving differential equations for the first time. In FLANN, the hidden layer is replaced by a functional expansion block for enhancement of the input patterns using orthogonal polynomials such as Chebyshev, Legendre, Hermite, etc. The computations become efficient because the procedure does not need to have hidden layer. Thus, the numbers of network parameters are less than the traditional ANN model. Varieties of differential equations are solved here using the above mentioned methods to show the reliability, powerfulness, and easy computer implementation of the methods. As such singular nonlinear initial value problems such as Lane-Emden and Emden-Fowler type equations have been solved using Chebyshev Neural Network (ChNN) model. Single layer Legendre Neural Network (LeNN) model has also been developed to handle Lane-Emden equation, Boundary Value Problem (BVP) and system of coupled ordinary differential equations. Unforced Duffing oscillator and unforced Van der Pol-Duffing oscillator equations are solved by developing Simple Orthogonal Polynomial based Neural Network (SOPNN) model. Further, Hermite Neural Network (HeNN) model is proposed to handle the Van der Pol-Duffing oscillator equation. Finally, a single layer Chebyshev Neural Network (ChNN) model has also been implemented to solve partial differential equations

Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives

Part 2 of this monograph builds on the introduction to tensor networks and their operations presented in Part 1. It focuses on tensor network models for super-compressed higher-order representation of data/parameters and related cost functions, while providing an outline of their applications in machine learning and data analytics. A particular emphasis is on the tensor train (TT) and Hierarchical Tucker (HT) decompositions, and their physically meaningful interpretations which reflect the scalability of the tensor network approach. Through a graphical approach, we also elucidate how, by virtue of the underlying low-rank tensor approximations and sophisticated contractions of core tensors, tensor networks have the ability to perform distributed computations on otherwise prohibitively large volumes of data/parameters, thereby alleviating or even eliminating the curse of dimensionality. The usefulness of this concept is illustrated over a number of applied areas, including generalized regression and classification (support tensor machines, canonical correlation analysis, higher order partial least squares), generalized eigenvalue decomposition, Riemannian optimization, and in the optimization of deep neural networks. Part 1 and Part 2 of this work can be used either as stand-alone separate texts, or indeed as a conjoint comprehensive review of the exciting field of low-rank tensor networks and tensor decompositions.Comment: 232 page

Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives

Part 2 of this monograph builds on the introduction to tensor networks and their operations presented in Part 1. It focuses on tensor network models for super-compressed higher-order representation of data/parameters and related cost functions, while providing an outline of their applications in machine learning and data analytics. A particular emphasis is on the tensor train (TT) and Hierarchical Tucker (HT) decompositions, and their physically meaningful interpretations which reflect the scalability of the tensor network approach. Through a graphical approach, we also elucidate how, by virtue of the underlying low-rank tensor approximations and sophisticated contractions of core tensors, tensor networks have the ability to perform distributed computations on otherwise prohibitively large volumes of data/parameters, thereby alleviating or even eliminating the curse of dimensionality. The usefulness of this concept is illustrated over a number of applied areas, including generalized regression and classification (support tensor machines, canonical correlation analysis, higher order partial least squares), generalized eigenvalue decomposition, Riemannian optimization, and in the optimization of deep neural networks. Part 1 and Part 2 of this work can be used either as stand-alone separate texts, or indeed as a conjoint comprehensive review of the exciting field of low-rank tensor networks and tensor decompositions.Comment: 232 page

Interval and Fuzzy Computing in Neural Network for System Identification Problems

Increase of population and growing of societal and commercial activities with limited land available in a modern city leads to construction up of tall/high-rise buildings. As such, it is important to investigate about the health of the structure after the occurrence of manmade or natural disasters such as earthquakes etc. A direct mathematical expression for parametric study or system identification of these structures is not always possible. Actually System Identification (SI) problems are inverse vibration problems consisting of coupled linear or non-linear differential equations that depend upon the physics of the system. It is also not always possible to get the solutions for these problems by classical methods. Few researchers have used different methods to solve the above mentioned problems. But difficulties are faced very often while finding solution to these problems because inverse problem generally gives non-unique parameter estimates. To overcome these difficulties alternate soft computing techniques such as Artificial Neural Networks (ANNs) are being used by various researchers to handle the above SI problems. It is worth mentioning that traditional neural network methods have inherent advantage because it can model the experimental data (input and output) where good mathematical model is not available. Moreover, inverse problems have been solved by other researchers for deterministic cases only. But while performing experiments it is always not possible to get the data exactly in crisp form. There may be some errors that are due to involvement of human or experiment. Accordingly, those data may actually be in uncertain form and corresponding methodologies need to be developed. It is an important issue about dealing with variables, parameters or data with uncertain value. There are three classes of uncertain models, which are probabilistic, fuzzy and interval. Recently, fuzzy theory and interval analysis are becoming powerful tools for many applications in recent decades. It is known that interval and fuzzy computations are themselves very complex to handle. Having these in mind one has to develop efficient computational models and algorithms very carefully to handle these uncertain problems. As said above, in general we may not obtain the corresponding input and output values (experimental) exactly or in crisp form but we may have only uncertain information of the data. Hence, investigations are needed to handle the SI problems where data is available in uncertain form. Identification methods with crisp (exact) data are known and traditional neural network methods have already been used by various researchers. But when the data are in uncertain form then traditional ANN may not be applied. Accordingly, new ANN models need to be developed which may solve the targeted uncertain SI problems. Hence present investigation targets to develop powerful methods of neural network based on interval and fuzzy theory for the analysis and simulation with respect to the uncertain system identification problems. In this thesis, these uncertain data are assumed as interval and fuzzy numbers. Accordingly, identification methodologies are developed for multistorey shear buildings by proposing new models of Interval Neural Network (INN) and Fuzzy Neural Network (FNN) models which can handle interval and fuzzified data respectively. It may however be noted that the developed methodology not only be important for the mentioned problems but those may very well be used in other application problems too. Few SI problems have been solved in the present thesis using INN and FNN model which are briefly described below. From initial design parameters (namely stiffness and mass in terms of interval and fuzzy) corresponding design frequencies may be obtained for a given structural problem viz. for a multistorey shear structure. The uncertain (interval/fuzzy) frequencies may then be used to estimate the present structural parameter values by the proposed INN and FNN. Next, the identification has been done using vibration response of the structure subject to ambient vibration with interval/fuzzy initial conditions. Forced vibration with horizontal displacement in interval/fuzzified form has also been used to investigate the identification problem. Moreover this study involves SI problems of structures (viz. shear buildings) with respect to earthquake data in order to know the health of a structure. It is well known that earthquake data are both positive and negative. The Interval Neural Network and Fuzzy Neural Network model may not handle the data with negative sign due to the complexity in interval and fuzzy computation. As regards, a novel transformation method have been developed to compute response of a structural system by training the model for Indian earthquakes at Chamoli and Uttarkashi using uncertain (interval/fuzzified) ground motion data. The simulation may give an idea about the safety of the structural system in case of future earthquakes. Further a single layer interval and fuzzy neural network based strategy has been proposed for simultaneous identification of the mass, stiffness and damping of uncertain multi-storey shear buildings using series/cluster of neural networks. It is known that training in MNN and also in INN and FNN are time consuming because these models depend upon the number of nodes in the hidden layer and convergence of the weights during training. As such, single layer Functional Link Neural Network (FLNN) with multi-input and multi-output model has also been proposed to solve the system identification problems for the first time. It is worth mentioning that, single input single output FLNN had been proposed by previous authors. In FLNN, the hidden layer is replaced by a functional expansion block for enhancement of the input patterns using orthogonal polynomials such as Chebyshev, Legendre and Hermite, etc. The computations become more efficient than the traditional or classical multi-layer neural network due to the absence of hidden layer. FLNN has also been used for structural response prediction of multistorey shear buildings subject to earthquake ground motion. It is seen that FLNN can very well predict the structural response of different floors of multi-storey shear building subject to earthquake data. Comparison of results among Multi layer Neural Network (MNN), Chebyshev Neural Network (ChNN), Legendre Neural Network (LeNN), Hermite Neural Network (HNN) and desired are considered and it is found that Functional Link Neural Network models are more effective and takes less computation time than MNN. In order to show the reliability, efficacy and powerfulness of INN, FNN and FLNN models variety of problems have been solved here. Finally FLNN is also extended to interval based FLNN which is again proposed for the first time to the best of our knowledge. This model is implemented to estimate the uncertain stiffness parameters of a multi-storey shear building. The parameters are identified here using uncertain response of the structure subject to ambient and forced vibration with interval initial condition and horizontal displacement also in interval form

Metaheuristic design of feedforward neural networks: a review of two decades of research

Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era

Advanced Strategies for Robot Manipulators

Amongst the robotic systems, robot manipulators have proven themselves to be of increasing importance and are widely adopted to substitute for human in repetitive and/or hazardous tasks. Modern manipulators are designed complicatedly and need to do more precise, crucial and critical tasks. So, the simple traditional control methods cannot be efficient, and advanced control strategies with considering special constraints are needed to establish. In spite of the fact that groundbreaking researches have been carried out in this realm until now, there are still many novel aspects which have to be explored

Reservoir Computing: computation with dynamical systems

In het onderzoeksgebied Machine Learning worden systemen onderzocht die kunnen leren op basis van voorbeelden. Binnen dit onderzoeksgebied zijn de recurrente neurale netwerken een belangrijke deelgroep. Deze netwerken zijn abstracte modellen van de werking van delen van de hersenen. Zij zijn in staat om zeer complexe temporele problemen op te lossen maar zijn over het algemeen zeer moeilijk om te trainen. Recentelijk zijn een aantal gelijkaardige methodes voorgesteld die dit trainingsprobleem elimineren. Deze methodes worden aangeduid met de naam Reservoir Computing. Reservoir Computing combineert de indrukwekkende rekenkracht van recurrente neurale netwerken met een eenvoudige trainingsmethode. Bovendien blijkt dat deze trainingsmethoden niet beperkt zijn tot neurale netwerken, maar kunnen toegepast worden op generieke dynamische systemen. Waarom deze systemen goed werken en welke eigenschappen bepalend zijn voor de prestatie is evenwel nog niet duidelijk. Voor dit proefschrift is onderzoek gedaan naar de dynamische eigenschappen van generieke Reservoir Computing systemen. Zo is experimenteel aangetoond dat de idee van Reservoir Computing ook toepasbaar is op niet-neurale netwerken van dynamische knopen. Verder is een maat voorgesteld die gebruikt kan worden om het dynamisch regime van een reservoir te meten. Tenslotte is een adaptatieregel geïntroduceerd die voor een breed scala reservoirtypes de dynamica van het reservoir kan afregelen tot het gewenste dynamisch regime. De technieken beschreven in dit proefschrift zijn gedemonstreerd op verschillende academische en ingenieurstoepassingen

Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks

Future wireless networks have a substantial potential in terms of supporting a broad range of complex compelling applications both in military and civilian fields, where the users are able to enjoy high-rate, low-latency, low-cost and reliable information services. Achieving this ambitious goal requires new radio techniques for adaptive learning and intelligent decision making because of the complex heterogeneous nature of the network structures and wireless services. Machine learning (ML) algorithms have great success in supporting big data analytics, efficient parameter estimation and interactive decision making. Hence, in this article, we review the thirty-year history of ML by elaborating on supervised learning, unsupervised learning, reinforcement learning and deep learning. Furthermore, we investigate their employment in the compelling applications of wireless networks, including heterogeneous networks (HetNets), cognitive radios (CR), Internet of things (IoT), machine to machine networks (M2M), and so on. This article aims for assisting the readers in clarifying the motivation and methodology of the various ML algorithms, so as to invoke them for hitherto unexplored services as well as scenarios of future wireless networks.Comment: 46 pages, 22 fig

Transition control based on grey, neural states

+hlm.;c

Approximation of the inverse kinematics of a robotic manipulator using a neural network

A fundamental property of a robotic manipulator system is that it is capable of accurately following complex position trajectories in three-dimensional space. An essential component of the robotic control system is the solution of the inverse kinematics problem which allows determination of the joint angle trajectories from the desired trajectory in the Cartesian space. There are several traditional methods based on the known geometry of robotic manipulators to solve the inverse kinematics problem. These methods can become impractical in a robot-vision control system where the environmental parameters can alter. Artificial neural networks with their inherent learning ability can approximate the inverse kinematics function and do not require any knowledge of the manipulator geometry. This thesis concentrates on developing a practical solution using a radial basis function network to approximate the inverse kinematics of a robot manipulator. This approach is distinct from existing approaches as the centres of the hidden-layer units are regularly distributed in the workspace, constrained training data is used and the training phase is performed using either the strict interpolation or the least mean square algorithms. An online retraining approach is also proposed to modify the network function approximation to cope with the situation where the initial training and application environments are different. Simulation results for two and three-link manipulators verify the approach. A novel real-time visual measurement system, based on a video camera and image processing software, has been developed to measure the position of the robotic manipulator in the three-dimensional workspace. Practical experiments have been performed with a Mitsubishi PA10-6CE manipulator and this visual measurement system. The performance of the radial basis function network is analysed for the manipulator operating in two and three-dimensional space and the practical results are compared to the simulation results. Advantages and disadvantages of the proposed approach are discussed