A Theory of Networks for Appxoimation and Learning

Learning an input-output mapping from a set of examples, of the type that many neural networks have been constructed to perform, can be regarded as synthesizing an approximation of a multi-dimensional function, that is solving the problem of hypersurface reconstruction. From this point of view, this form of learning is closely related to classical approximation techniques, such as generalized splines and regularization theory. This paper considers the problems of an exact representation and, in more detail, of the approximation of linear and nolinear mappings in terms of simpler functions of fewer variables. Kolmogorov's theorem concerning the representation of functions of several variables in terms of functions of one variable turns out to be almost irrelevant in the context of networks for learning. We develop a theoretical framework for approximation based on regularization techniques that leads to a class of three-layer networks that we call Generalized Radial Basis Functions (GRBF), since they are mathematically related to the well-known Radial Basis Functions, mainly used for strict interpolation tasks. GRBF networks are not only equivalent to generalized splines, but are also closely related to pattern recognition methods such as Parzen windows and potential functions and to several neural network algorithms, such as Kanerva's associative memory, backpropagation and Kohonen's topology preserving map. They also have an interesting interpretation in terms of prototypes that are synthesized and optimally combined during the learning stage. The paper introduces several extensions and applications of the technique and discusses intriguing analogies with neurobiological data

Bir Gizli Katmanlı Yapay Sinir Ağlarında Optimal Nöron Sayısının İncelenmesi

Bu makalede, bir gizli katmanlı yapay sinir ağları için optimal nöron sayısı araştırılmıştır. Bunun için teorik ve istatiksel çalışmalar yapılmıştır. Optimal nöron sayısını bulmak için global minimum bulmak gereklidir. Ancak yapay sinir ağlarının eğitimi konveks olmayan bir problem olduğundan optimizasyon algoritmaları ile global minimum bulmak zordur. Bu çalışmada global minimumu dolayısıyla optimum nöron sayısını bulmak için baskı maliyet fonksiyonu önerilmiştir. Baskı maliyet fonksiyonu yardımıyla global minimumu veren yapay sinir ağı modelinin nöron sayısının, optimal nöron sayısını verdiği gösterilmiştir. Ayrıca baskı maliyet fonksiyonu XOR veri kümesi ve daire veri kümesi üzerinde test edilmiş ve XOR veri kümesi üzerinde %99, daire veri kümesi üzerinde ise %97 başarı elde edilmiştir. Bu veri kümeleri için optimal nöron sayısı tespit edilmiştir

The Construction of Arbitrary Stable Dynamics in Non-Linear Neural Networks

In this paper, two methods for constructing systems of ordinary differential equations realizing any fixed finite set of equilibria in any fixed finite dimension are introduced; no spurious equilibria are possible for either method. By using the first method, one can construct a system with the fewest number of equilibria, given a fixed set of attractors. Using a strict Lyapunov function for each of these differential equations, a large class of systems with the same set of equilibria is constructed. A method of fitting these nonlinear systems to trajectories is proposed. In addition, a general method which will produce an arbitrary number of periodic orbits of shapes of arbitrary complexity is also discussed. A more general second method is given to construct a differential equation which converges to a fixed given finite set of equilibria. This technique is much more general in that it allows this set of equilibria to have any of a large class of indices which are consistent with the Morse Inequalities. It is clear that this class is not universal, because there is a large class of additional vector fields with convergent dynamics which cannot be constructed by the above method. The easiest way to see this is to enumerate the set of Morse indices which can be obtained by the above method and compare this class with the class of Morse indices of arbitrary differential equations with convergent dynamics. The former set of indices are a proper subclass of the latter, therefore, the above construction cannot be universal. In general, it is a difficult open problem to construct a specific example of a differential equation with a given fixed set of equilibria, permissible Morse indices, and permissible connections between stable and unstable manifolds. A strict Lyapunov function is given for this second case as well. This strict Lyapunov function as above enables construction of a large class of examples consistent with these more complicated dynamics and indices. The determination of all the basins of attraction in the general case for these systems is also difficult and open.Air Force Office of Scientific Research (F49620-86-C-0037

Deep Learning Techniques for Music Generation -- A Survey

This paper is a survey and an analysis of different ways of using deep learning (deep artificial neural networks) to generate musical content. We propose a methodology based on five dimensions for our analysis: Objective - What musical content is to be generated? Examples are: melody, polyphony, accompaniment or counterpoint. - For what destination and for what use? To be performed by a human(s) (in the case of a musical score), or by a machine (in the case of an audio file). Representation - What are the concepts to be manipulated? Examples are: waveform, spectrogram, note, chord, meter and beat. - What format is to be used? Examples are: MIDI, piano roll or text. - How will the representation be encoded? Examples are: scalar, one-hot or many-hot. Architecture - What type(s) of deep neural network is (are) to be used? Examples are: feedforward network, recurrent network, autoencoder or generative adversarial networks. Challenge - What are the limitations and open challenges? Examples are: variability, interactivity and creativity. Strategy - How do we model and control the process of generation? Examples are: single-step feedforward, iterative feedforward, sampling or input manipulation. For each dimension, we conduct a comparative analysis of various models and techniques and we propose some tentative multidimensional typology. This typology is bottom-up, based on the analysis of many existing deep-learning based systems for music generation selected from the relevant literature. These systems are described and are used to exemplify the various choices of objective, representation, architecture, challenge and strategy. The last section includes some discussion and some prospects.Comment: 209 pages. This paper is a simplified version of the book: J.-P. Briot, G. Hadjeres and F.-D. Pachet, Deep Learning Techniques for Music Generation, Computational Synthesis and Creative Systems, Springer, 201

An investigation into adaptive power reduction techniques for neural hardware

In light of the growing applicability of Artificial Neural Network (ANN) in the signal processing field [1] and the present thrust of the semiconductor industry towards lowpower SOCs for mobile devices [2], the power consumption of ANN hardware has become a very important implementation issue. Adaptability is a powerful and useful feature of neural networks. All current approaches for low-power ANN hardware techniques are ‘non-adaptive’ with respect to the power consumption of the network (i.e. power-reduction is not an objective of the adaptation/learning process). In the research work presented in this thesis, investigations on possible adaptive power reduction techniques have been carried out, which attempt to exploit the adaptability of neural networks in order to reduce the power consumption. Three separate approaches for such adaptive power reduction are proposed: adaptation of size, adaptation of network weights and adaptation of calculation precision. Initial case studies exhibit promising results with significantpower reduction

Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusion

Graph neural networks are widely used tools for graph prediction tasks. Motivated by their empirical performance, prior works have developed generalization bounds for graph neural networks, which scale with graph structures in terms of the maximum degree. In this paper, we present generalization bounds that instead scale with the largest singular value of the graph neural network's feature diffusion matrix. These bounds are numerically much smaller than prior bounds for real-world graphs. We also construct a lower bound of the generalization gap that matches our upper bound asymptotically. To achieve these results, we analyze a unified model that includes prior works' settings (i.e., convolutional and message-passing networks) and new settings (i.e., graph isomorphism networks). Our key idea is to measure the stability of graph neural networks against noise perturbations using Hessians. Empirically, we find that Hessian-based measurements correlate with the observed generalization gaps of graph neural networks accurately. Optimizing noise stability properties for fine-tuning pretrained graph neural networks also improves test performance on several graph-level classification tasks.Comment: 36 pages, 2 tables, 3 figures. Appeared in AISTATS 202

Multilevel Power Estimation Of VLSI Circuits Using Efficient Algorithms

New and complex systems are being implemented using highly advanced Electronic Design Automation (EDA) tools. As the complexity increases day by day, the dissipation of power has emerged as one of the very important design constraints. Now low power designs are not only used in small size applications like cell phones and handheld devices but also in high-performance computing applications. Embedded memories have been used extensively in modern SOC designs. In order to estimate the power consumption of the entire design correctly, an accurate memory power model is needed. However, the memory power model commonly used in commercial EDA tools is too simple to estimate the power consumption accurately. For complex digital circuits, building their power models is a popular approach to estimate their power consumption without detailed circuit information. In the literature, most of power models are built with lookup tables. However, building the power models with lookup tables may become infeasible for large circuits because the table size would increase exponentially to meet the accuracy requirement. This thesis involves two parts. In first part it uses the Synopsys power measurement tools together with the use of synthesis and extraction tools to determine power consumed by various macros at different levels of abstraction including the Register Transfer Level (RTL), the gate and the transistor level. In general, it can be concluded that as the level of abstraction goes down the accuracy of power measurement increases depending on the tool used. In second part a novel power modeling approach for complex circuits by using neural networks to learn the relationship between power dissipation and input/output characteristic vector during simulation has been developed. Our neural power model has very low complexity such that this power model can be used for complex circuits. Using such a simple structure, the neural power models can still have high accuracy because they can automatically consider the non-linear power distributions. Unlike the power characterization process in traditional approaches, our characterization process is very simple and straightforward. More importantly, using the neural power model for power estimation does not require any transistor-level or gate-level description of the circuits. The experimental results have shown that the estimations are accurate and efficient for different test sequences with wide range of input distributions