938 research outputs found
Deep supervised learning using local errors
Error backpropagation is a highly effective mechanism for learning
high-quality hierarchical features in deep networks. Updating the features or
weights in one layer, however, requires waiting for the propagation of error
signals from higher layers. Learning using delayed and non-local errors makes
it hard to reconcile backpropagation with the learning mechanisms observed in
biological neural networks as it requires the neurons to maintain a memory of
the input long enough until the higher-layer errors arrive. In this paper, we
propose an alternative learning mechanism where errors are generated locally in
each layer using fixed, random auxiliary classifiers. Lower layers could thus
be trained independently of higher layers and training could either proceed
layer by layer, or simultaneously in all layers using local error information.
We address biological plausibility concerns such as weight symmetry
requirements and show that the proposed learning mechanism based on fixed,
broad, and random tuning of each neuron to the classification categories
outperforms the biologically-motivated feedback alignment learning technique on
the MNIST, CIFAR10, and SVHN datasets, approaching the performance of standard
backpropagation. Our approach highlights a potential biological mechanism for
the supervised, or task-dependent, learning of feature hierarchies. In
addition, we show that it is well suited for learning deep networks in custom
hardware where it can drastically reduce memory traffic and data communication
overheads
An event-based architecture for solving constraint satisfaction problems
Constraint satisfaction problems (CSPs) are typically solved using
conventional von Neumann computing architectures. However, these architectures
do not reflect the distributed nature of many of these problems and are thus
ill-suited to solving them. In this paper we present a hybrid analog/digital
hardware architecture specifically designed to solve such problems. We cast
CSPs as networks of stereotyped multi-stable oscillatory elements that
communicate using digital pulses, or events. The oscillatory elements are
implemented using analog non-stochastic circuits. The non-repeating phase
relations among the oscillatory elements drive the exploration of the solution
space. We show that this hardware architecture can yield state-of-the-art
performance on a number of CSPs under reasonable assumptions on the
implementation. We present measurements from a prototype electronic chip to
demonstrate that a physical implementation of the proposed architecture is
robust to practical non-idealities and to validate the theory proposed.Comment: First two authors contributed equally to this wor
Hardware-efficient on-line learning through pipelined truncated-error backpropagation in binary-state networks
Artificial neural networks (ANNs) trained using backpropagation are powerful
learning architectures that have achieved state-of-the-art performance in
various benchmarks. Significant effort has been devoted to developing custom
silicon devices to accelerate inference in ANNs. Accelerating the training
phase, however, has attracted relatively little attention. In this paper, we
describe a hardware-efficient on-line learning technique for feedforward
multi-layer ANNs that is based on pipelined backpropagation. Learning is
performed in parallel with inference in the forward pass, removing the need for
an explicit backward pass and requiring no extra weight lookup. By using binary
state variables in the feedforward network and ternary errors in
truncated-error backpropagation, the need for any multiplications in the
forward and backward passes is removed, and memory requirements for the
pipelining are drastically reduced. Further reduction in addition operations
owing to the sparsity in the forward neural and backpropagating error signal
paths contributes to highly efficient hardware implementation. For
proof-of-concept validation, we demonstrate on-line learning of MNIST
handwritten digit classification on a Spartan 6 FPGA interfacing with an
external 1Gb DDR2 DRAM, that shows small degradation in test error performance
compared to an equivalently sized binary ANN trained off-line using standard
back-propagation and exact errors. Our results highlight an attractive synergy
between pipelined backpropagation and binary-state networks in substantially
reducing computation and memory requirements, making pipelined on-line learning
practical in deep networks.Comment: Now also consider 0/1 binary activations. Memory access statistics
reporte
Effect of particle morphology on film cracking
233 p.With the aim of reducing VOC emissions and producing more sustainable coatings and adhesives, solvent borne polymers are gradually being replaced by water-based alternatives. However, the mechanical properties of water borne systems are typically not as good as those of solvent borne products due to differences in the film formation process.The thesis aimed at shedding some light on film formation and cracking problem to be able to design soft core-hard “shell” latexes that yield VOC and crack free, mechanically strong coatings that can be cast at low temperatures. These are particles consisting of a soft core covered by patches of hard polymer. Different strategies have been used to overcome the film formation and cracking problem like . A mathematical model for cracking prediction and stress calculation during drying of aqueous dispersions of soft core-hard “shell” particles was developed. The model solved numerically the incompressible Navier-Stokes and continuity equations using COMSOL Multiphysics. The model was validated with experimental data. A good agreement between experimental results and model predictions was achieved.The results presented in this work highlight that careful design of soft core-hard “shell" polymer dispersions allows overcoming the film formation dilemma frequently found in water borne coatingsIndustrial Liaison Program in Polymerization in Dispersed Media
POLYMAT
Rhythmic inhibition allows neural networks to search for maximally consistent states
Gamma-band rhythmic inhibition is a ubiquitous phenomenon in neural circuits
yet its computational role still remains elusive. We show that a model of
Gamma-band rhythmic inhibition allows networks of coupled cortical circuit
motifs to search for network configurations that best reconcile external inputs
with an internal consistency model encoded in the network connectivity. We show
that Hebbian plasticity allows the networks to learn the consistency model by
example. The search dynamics driven by rhythmic inhibition enable the described
networks to solve difficult constraint satisfaction problems without making
assumptions about the form of stochastic fluctuations in the network. We show
that the search dynamics are well approximated by a stochastic sampling
process. We use the described networks to reproduce perceptual multi-stability
phenomena with switching times that are a good match to experimental data and
show that they provide a general neural framework which can be used to model
other 'perceptual inference' phenomena
Surrogate Gradient Learning in Spiking Neural Networks
Spiking neural networks are nature's versatile solution to fault-tolerant and
energy efficient signal processing. To translate these benefits into hardware,
a growing number of neuromorphic spiking neural network processors attempt to
emulate biological neural networks. These developments have created an imminent
need for methods and tools to enable such systems to solve real-world signal
processing problems. Like conventional neural networks, spiking neural networks
can be trained on real, domain specific data. However, their training requires
overcoming a number of challenges linked to their binary and dynamical nature.
This article elucidates step-by-step the problems typically encountered when
training spiking neural networks, and guides the reader through the key
concepts of synaptic plasticity and data-driven learning in the spiking
setting. To that end, it gives an overview of existing approaches and provides
an introduction to surrogate gradient methods, specifically, as a particularly
flexible and efficient method to overcome the aforementioned challenges
- …