74,098 research outputs found
Mean Field Bayes Backpropagation: scalable training of multilayer neural networks with binary weights
Significant success has been reported recently using deep neural networks for
classification. Such large networks can be computationally intensive, even
after training is over. Implementing these trained networks in hardware chips
with a limited precision of synaptic weights may improve their speed and energy
efficiency by several orders of magnitude, thus enabling their integration into
small and low-power electronic devices. With this motivation, we develop a
computationally efficient learning algorithm for multilayer neural networks
with binary weights, assuming all the hidden neurons have a fan-out of one.
This algorithm, derived within a Bayesian probabilistic online setting, is
shown to work well for both synthetic and real-world problems, performing
comparably to algorithms with real-valued weights, while retaining
computational tractability
Training Dynamic Exponential Family Models with Causal and Lateral Dependencies for Generalized Neuromorphic Computing
Neuromorphic hardware platforms, such as Intel's Loihi chip, support the
implementation of Spiking Neural Networks (SNNs) as an energy-efficient
alternative to Artificial Neural Networks (ANNs). SNNs are networks of neurons
with internal analogue dynamics that communicate by means of binary time
series. In this work, a probabilistic model is introduced for a generalized
set-up in which the synaptic time series can take values in an arbitrary
alphabet and are characterized by both causal and instantaneous statistical
dependencies. The model, which can be considered as an extension of exponential
family harmoniums to time series, is introduced by means of a hybrid
directed-undirected graphical representation. Furthermore, distributed learning
rules are derived for Maximum Likelihood and Bayesian criteria under the
assumption of fully observed time series in the training set.Comment: Published in IEEE ICASSP 2019. Author's Accepted Manuscrip
PAC-Bayesian Learning of Aggregated Binary Activated Neural Networks with Probabilities over Representations
Considering a probability distribution over parameters is known as an
efficient strategy to learn a neural network with non-differentiable activation
functions. We study the expectation of a probabilistic neural network as a
predictor by itself, focusing on the aggregation of binary activated neural
networks with normal distributions over real-valued weights. Our work leverages
a recent analysis derived from the PAC-Bayesian framework that derives tight
generalization bounds and learning procedures for the expected output value of
such an aggregation, which is given by an analytical expression. While the
combinatorial nature of the latter has been circumvented by approximations in
previous works, we show that the exact computation remains tractable for deep
but narrow neural networks, thanks to a dynamic programming approach. This
leads us to a peculiar bound minimization learning algorithm for binary
activated neural networks, where the forward pass propagates probabilities over
representations instead of activation values. A stochastic counterpart that
scales to wide architectures is proposed
- …