29 research outputs found
Training Dynamic Exponential Family Models with Causal and Lateral Dependencies for Generalized Neuromorphic Computing
Neuromorphic hardware platforms, such as Intel's Loihi chip, support the
implementation of Spiking Neural Networks (SNNs) as an energy-efficient
alternative to Artificial Neural Networks (ANNs). SNNs are networks of neurons
with internal analogue dynamics that communicate by means of binary time
series. In this work, a probabilistic model is introduced for a generalized
set-up in which the synaptic time series can take values in an arbitrary
alphabet and are characterized by both causal and instantaneous statistical
dependencies. The model, which can be considered as an extension of exponential
family harmoniums to time series, is introduced by means of a hybrid
directed-undirected graphical representation. Furthermore, distributed learning
rules are derived for Maximum Likelihood and Bayesian criteria under the
assumption of fully observed time series in the training set.Comment: Published in IEEE ICASSP 2019. Author's Accepted Manuscrip
Learning How to Demodulate from Few Pilots via Meta-Learning
Consider an Internet-of-Things (IoT) scenario in which devices transmit
sporadically using short packets with few pilot symbols. Each device transmits
over a fading channel and is characterized by an amplifier with a unique
non-linear transfer function. The number of pilots is generally insufficient to
obtain an accurate estimate of the end-to-end channel, which includes the
effects of fading and of the amplifier's distortion. This paper proposes to
tackle this problem using meta-learning. Accordingly, pilots from previous IoT
transmissions are used as meta-training in order to learn a demodulator that is
able to quickly adapt to new end-to-end channel conditions from few pilots.
Numerical results validate the advantages of the approach as compared to
training schemes that either do not leverage prior transmissions or apply a
standard learning algorithm on previously received data
Multi-Sample Online Learning for Probabilistic Spiking Neural Networks
Spiking Neural Networks (SNNs) capture some of the efficiency of biological
brains for inference and learning via the dynamic, online, event-driven
processing of binary time series. Most existing learning algorithms for SNNs
are based on deterministic neuronal models, such as leaky integrate-and-fire,
and rely on heuristic approximations of backpropagation through time that
enforce constraints such as locality. In contrast, probabilistic SNN models can
be trained directly via principled online, local, update rules that have proven
to be particularly effective for resource-constrained systems. This paper
investigates another advantage of probabilistic SNNs, namely their capacity to
generate independent outputs when queried over the same input. It is shown that
the multiple generated output samples can be used during inference to robustify
decisions and to quantify uncertainty -- a feature that deterministic SNN
models cannot provide. Furthermore, they can be leveraged for training in order
to obtain more accurate statistical estimates of the log-loss training
criterion, as well as of its gradient. Specifically, this paper introduces an
online learning rule based on generalized expectation-maximization (GEM) that
follows a three-factor form with global learning signals and is referred to as
GEM-SNN. Experimental results on structured output memorization and
classification on a standard neuromorphic data set demonstrate significant
improvements in terms of log-likelihood, accuracy, and calibration when
increasing the number of samples used for inference and training.Comment: Submitte
BiSNN: Training Spiking Neural Networks with Binary Weights via Bayesian Learning
Artificial Neural Network (ANN)-based inference on battery-powered devices
can be made more energy-efficient by restricting the synaptic weights to be
binary, hence eliminating the need to perform multiplications. An alternative,
emerging, approach relies on the use of Spiking Neural Networks (SNNs),
biologically inspired, dynamic, event-driven models that enhance energy
efficiency via the use of binary, sparse, activations. In this paper, an SNN
model is introduced that combines the benefits of temporally sparse binary
activations and of binary weights. Two learning rules are derived, the first
based on the combination of straight-through and surrogate gradient techniques,
and the second based on a Bayesian paradigm. Experiments validate the
performance loss with respect to full-precision implementations, and
demonstrate the advantage of the Bayesian paradigm in terms of accuracy and
calibration.Comment: Submitte
Bayesian Continual Learning via Spiking Neural Networks
Among the main features of biological intelligence are energy efficiency,
capacity for continual adaptation, and risk management via uncertainty
quantification. Neuromorphic engineering has been thus far mostly driven by the
goal of implementing energy-efficient machines that take inspiration from the
time-based computing paradigm of biological brains. In this paper, we take
steps towards the design of neuromorphic systems that are capable of adaptation
to changing learning tasks, while producing well-calibrated uncertainty
quantification estimates. To this end, we derive online learning rules for
spiking neural networks (SNNs) within a Bayesian continual learning framework.
In it, each synaptic weight is represented by parameters that quantify the
current epistemic uncertainty resulting from prior knowledge and observed data.
The proposed online rules update the distribution parameters in a streaming
fashion as data are observed. We instantiate the proposed approach for both
real-valued and binary synaptic weights. Experimental results using Intel's
Lava platform show the merits of Bayesian over frequentist learning in terms of
capacity for adaptation and uncertainty quantification.Comment: Accepted for publication in Frontiers in Computational Neuroscienc
Spiking Neural Networks -- Part III: Neuromorphic Communications
Synergies between wireless communications and artificial intelligence are
increasingly motivating research at the intersection of the two fields. On the
one hand, the presence of more and more wirelessly connected devices, each with
its own data, is driving efforts to export advances in machine learning (ML)
from high performance computing facilities, where information is stored and
processed in a single location, to distributed, privacy-minded, processing at
the end user. On the other hand, ML can address algorithm and model deficits in
the optimization of communication protocols. However, implementing ML models
for learning and inference on battery-powered devices that are connected via
bandwidth-constrained channels remains challenging. This paper explores two
ways in which Spiking Neural Networks (SNNs) can help address these open
problems. First, we discuss federated learning for the distributed training of
SNNs, and then describe the integration of neuromorphic sensing, SNNs, and
impulse radio technologies for low-power remote inference.Comment: Submitte
Spiking Neural Networks -- Part I: Detecting Spatial Patterns
Spiking Neural Networks (SNNs) are biologically inspired machine learning
models that build on dynamic neuronal models processing binary and sparse
spiking signals in an event-driven, online, fashion. SNNs can be implemented on
neuromorphic computing platforms that are emerging as energy-efficient
co-processors for learning and inference. This is the first of a series of
three papers that introduce SNNs to an audience of engineers by focusing on
models, algorithms, and applications. In this first paper, we first cover
neural models used for conventional Artificial Neural Networks (ANNs) and SNNs.
Then, we review learning algorithms and applications for SNNs that aim at
mimicking the functionality of ANNs by detecting or generating spatial patterns
in rate-encoded spiking signals. We specifically discuss ANN-to-SNN conversion
and neural sampling. Finally, we validate the capabilities of SNNs for
detecting and generating spatial patterns through experiments.Comment: Submitte
Learning to Demodulate from Few Pilots via Offline and Online Meta-Learning
This paper considers an Internet-of-Things (IoT) scenario in which devices
sporadically transmit short packets with few pilot symbols over a fading
channel. Devices are characterized by unique transmission non-idealities, such
as I/Q imbalance. The number of pilots is generally insufficient to obtain an
accurate estimate of the end-to-end channel, which includes the effects of
fading and of the transmission-side distortion. This paper proposes to tackle
this problem by using meta-learning. Accordingly, pilots from previous IoT
transmissions are used as meta-training data in order to train a demodulator
that is able to quickly adapt to new end-to-end channel conditions from few
pilots. Various state-of-the-art meta-learning schemes are adapted to the
problem at hand and evaluated, including Model-Agnostic Meta-Learning (MAML),
First-Order MAML (FOMAML), REPTILE, and fast Context Adaptation VIA
meta-learning (CAVIA). Both offline and online solutions are developed. In the
latter case, an integrated online meta-learning and adaptive pilot number
selection scheme is proposed. Numerical results validate the advantages of
meta-learning as compared to training schemes that either do not leverage prior
transmissions or apply a standard joint learning algorithms on previously
received data.Comment: journal paper to appear in IEEE Transactions on Signal Processing,
subsumes (arXiv:1903.02184