1,637 research outputs found
C-Mantec: A novel constructive neural network algorithm incorporating competition between neurons.
C-Mantec is a novel neural network constructive algorithm that combines competition between neurons
with a stable modified perceptron learning rule. The neuron learning is governed by the thermal
perceptron rule that ensures stability of the acquired knowledge while the architecture grows and while
the neurons compete for new incoming information. Competition makes it possible that even after new
units have been added to the network, existing neurons still can learn if the incoming information is
similar to their stored knowledge, and this constitutes a major difference with existing constructing
algorithms. The new algorithm is tested on two different sets of benchmark problems: a Boolean function
set used in logic circuit design and a well studied set of real world problems. Both sets were used to analyze
the size of the constructed architectures and the generalization ability obtained and to compare the results
with those from other standard and well known classification algorithms. The problem of overfitting is
also analyzed, and a new built-in method to avoid its effects is devised and successfully applied within an
active learning paradigm that filter noisy examples. The results show that the new algorithm generates
very compact neural architectures with state-of-the-art generalization capabilities.The authors acknowledge the support from MICIIN (Spain) through grants TIN2008-04985 and TIN2010-16556 (including FEDER funds) and from Junta de Andalucía through grant P08-TIC-04026
Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions
Despite the widespread success of Transformers on NLP tasks, recent works
have found that they struggle to model several formal languages when compared
to recurrent models. This raises the question of why Transformers perform well
in practice and whether they have any properties that enable them to generalize
better than recurrent models. In this work, we conduct an extensive empirical
study on Boolean functions to demonstrate the following: (i) Random
Transformers are relatively more biased towards functions of low sensitivity.
(ii) When trained on Boolean functions, both Transformers and LSTMs prioritize
learning functions of low sensitivity, with Transformers ultimately converging
to functions of lower sensitivity. (iii) On sparse Boolean functions which have
low sensitivity, we find that Transformers generalize near perfectly even in
the presence of noisy labels whereas LSTMs overfit and achieve poor
generalization accuracy. Overall, our results provide strong quantifiable
evidence that suggests differences in the inductive biases of Transformers and
recurrent models which may help explain Transformer's effective generalization
performance despite relatively limited expressiveness.Comment: Preprin
Applying MDL to Learning Best Model Granularity
The Minimum Description Length (MDL) principle is solidly based on a provably
ideal method of inference using Kolmogorov complexity. We test how the theory
behaves in practice on a general problem in model selection: that of learning
the best model granularity. The performance of a model depends critically on
the granularity, for example the choice of precision of the parameters. Too
high precision generally involves modeling of accidental noise and too low
precision may lead to confusion of models that should be distinguished. This
precision is often determined ad hoc. In MDL the best model is the one that
most compresses a two-part code of the data set: this embodies ``Occam's
Razor.'' In two quite different experimental settings the theoretical value
determined using MDL coincides with the best value found experimentally. In the
first experiment the task is to recognize isolated handwritten characters in
one subject's handwriting, irrespective of size and orientation. Based on a new
modification of elastic matching, using multiple prototypes per character, the
optimal prediction rate is predicted for the learned parameter (length of
sampling interval) considered most likely by MDL, which is shown to coincide
with the best value found experimentally. In the second experiment the task is
to model a robot arm with two degrees of freedom using a three layer
feed-forward neural network where we need to determine the number of nodes in
the hidden layer giving best modeling performance. The optimal model (the one
that extrapolizes best on unseen examples) is predicted for the number of nodes
in the hidden layer considered most likely by MDL, which again is found to
coincide with the best value found experimentally.Comment: LaTeX, 32 pages, 5 figures. Artificial Intelligence journal, To
appea
Deep Learning Techniques for Music Generation -- A Survey
This paper is a survey and an analysis of different ways of using deep
learning (deep artificial neural networks) to generate musical content. We
propose a methodology based on five dimensions for our analysis:
Objective - What musical content is to be generated? Examples are: melody,
polyphony, accompaniment or counterpoint. - For what destination and for what
use? To be performed by a human(s) (in the case of a musical score), or by a
machine (in the case of an audio file).
Representation - What are the concepts to be manipulated? Examples are:
waveform, spectrogram, note, chord, meter and beat. - What format is to be
used? Examples are: MIDI, piano roll or text. - How will the representation be
encoded? Examples are: scalar, one-hot or many-hot.
Architecture - What type(s) of deep neural network is (are) to be used?
Examples are: feedforward network, recurrent network, autoencoder or generative
adversarial networks.
Challenge - What are the limitations and open challenges? Examples are:
variability, interactivity and creativity.
Strategy - How do we model and control the process of generation? Examples
are: single-step feedforward, iterative feedforward, sampling or input
manipulation.
For each dimension, we conduct a comparative analysis of various models and
techniques and we propose some tentative multidimensional typology. This
typology is bottom-up, based on the analysis of many existing deep-learning
based systems for music generation selected from the relevant literature. These
systems are described and are used to exemplify the various choices of
objective, representation, architecture, challenge and strategy. The last
section includes some discussion and some prospects.Comment: 209 pages. This paper is a simplified version of the book: J.-P.
Briot, G. Hadjeres and F.-D. Pachet, Deep Learning Techniques for Music
Generation, Computational Synthesis and Creative Systems, Springer, 201
Storage capacity of correlated perceptrons
We consider an ensemble of single-layer perceptrons exposed to random
inputs and investigate the conditions under which the couplings of these
perceptrons can be chosen such that prescribed correlations between the outputs
occur. A general formalism is introduced using a multi-perceptron costfunction
that allows to determine the maximal number of random inputs as a function of
the desired values of the correlations. Replica-symmetric results for and
are compared with properties of two-layer networks of tree-structure and
fixed Boolean function between hidden units and output. The results show which
correlations in the hidden layer of multi-layer neural networks are crucial for
the value of the storage capacity.Comment: 16 pages, Latex2
- …