43,799 research outputs found
Dynamics of Supervised Learning with Restricted Training Sets
We study the dynamics of supervised learning in layered neural networks, in
the regime where the size of the training set is proportional to the number
of inputs. Here the local fields are no longer described by Gaussian
probability distributions. We show how dynamical replica theory can be used to
predict the evolution of macroscopic observables, including the relevant
performance measures, incorporating the old formalism in the limit
as a special case. For simplicity we restrict ourselves
to single-layer networks and realizable tasks.Comment: 36 pages, latex2e, 12 eps figures (to be publ in: Proc Newton Inst
Workshop on On-Line Learning '97
Ensemble of Single‐Layered Complex‐Valued Neural Networks for Classification Tasks
This paper presents ensemble approaches in single-layered complex-valued
neural network (CVNN) to solve real-valued classification problems. Each
component CVNN of an ensemble uses a recently proposed activation function
for its complex-valued neurons (CVNs). A gradient-descent based learning
algorithm was used to train the component CVNNs. We applied two ensemble
methods, negative correlation learning and bagging, to create the ensembles.
Experimental results on a number of real-world benchmark problems showed a
substantial performance improvement of the ensembles over the individual
single-layered CVNN classifiers. Furthermore, the generalization performances
were nearly equivalent to those obtained by the ensembles of real-valued
multilayer neural networks
Vertical Layering of Quantized Neural Networks for Heterogeneous Inference
Although considerable progress has been obtained in neural network
quantization for efficient inference, existing methods are not scalable to
heterogeneous devices as one dedicated model needs to be trained, transmitted,
and stored for one specific hardware setting, incurring considerable costs in
model training and maintenance. In this paper, we study a new vertical-layered
representation of neural network weights for encapsulating all quantized models
into a single one. With this representation, we can theoretically achieve any
precision network for on-demand service while only needing to train and
maintain one model. To this end, we propose a simple once quantization-aware
training (QAT) scheme for obtaining high-performance vertical-layered models.
Our design incorporates a cascade downsampling mechanism which allows us to
obtain multiple quantized networks from one full precision source model by
progressively mapping the higher precision weights to their adjacent lower
precision counterparts. Then, with networks of different bit-widths from one
source model, multi-objective optimization is employed to train the shared
source model weights such that they can be updated simultaneously, considering
the performance of all networks. By doing this, the shared weights will be
optimized to balance the performance of different quantized models, thus making
the weights transferable among different bit widths. Experiments show that the
proposed vertical-layered representation and developed once QAT scheme are
effective in embodying multiple quantized networks into a single one and allow
one-time training, and it delivers comparable performance as that of quantized
models tailored to any specific bit-width. Code will be available.Comment: Submitted to IEEE for possible publicatio
Neural networks grown and self-organized by noise
Living neural networks emerge through a process of growth and self-organization that begins with a single cell and results in a brain, an organized and functional computational device. Artificial neural networks, however, rely on human-designed, hand-programmed architectures for their remarkable performance. Can we develop artificial computational devices that can grow and self-organize without human intervention? In this paper, we propose a biologically inspired developmental algorithm that can ‘grow’ a functional, layered neural network from a single initial cell. The algorithm organizes inter-layer connections to construct retinotopic pooling layers. Our approach is inspired by the mechanisms employed by the early visual system to wire the retina to the lateral geniculate nucleus (LGN), days before animals open their eyes. The key ingredients for robust self-organization are an emergent spontaneous spatiotemporal activity wave in the first layer and a local learning rule in the second layer that ‘learns’ the underlying activity pattern in the first layer. The algorithm is adaptable to a wide-range of input-layer geometries, robust to malfunctioning units in the first layer, and so can be used to successfully grow and self-organize pooling architectures of different pool-sizes and shapes. The algorithm provides a primitive procedure for constructing layered neural networks through growth and self-organization. We also demonstrate that networks grown from a single unit perform as well as hand-crafted networks on MNIST. Broadly, our work shows that biologically inspired developmental algorithms can be applied to autonomously grow functional `brains' in-silico
Neural networks grown and self-organized by noise
Living neural networks emerge through a process of growth and
self-organization that begins with a single cell and results in a brain, an
organized and functional computational device. Artificial neural networks,
however, rely on human-designed, hand-programmed architectures for their
remarkable performance. Can we develop artificial computational devices that
can grow and self-organize without human intervention? In this paper, we
propose a biologically inspired developmental algorithm that can 'grow' a
functional, layered neural network from a single initial cell. The algorithm
organizes inter-layer connections to construct a convolutional pooling layer, a
key constituent of convolutional neural networks (CNN's). Our approach is
inspired by the mechanisms employed by the early visual system to wire the
retina to the lateral geniculate nucleus (LGN), days before animals open their
eyes. The key ingredients for robust self-organization are an emergent
spontaneous spatiotemporal activity wave in the first layer and a local
learning rule in the second layer that 'learns' the underlying activity pattern
in the first layer. The algorithm is adaptable to a wide-range of input-layer
geometries, robust to malfunctioning units in the first layer, and so can be
used to successfully grow and self-organize pooling architectures of different
pool-sizes and shapes. The algorithm provides a primitive procedure for
constructing layered neural networks through growth and self-organization.
Broadly, our work shows that biologically inspired developmental algorithms can
be applied to autonomously grow functional 'brains' in-silico.Comment: 21 pages (including 11 pages of appendix
- …