9 research outputs found
Inherent Redundancy in Spiking Neural Networks
Spiking Neural Networks (SNNs) are well known as a promising energy-efficient
alternative to conventional artificial neural networks. Subject to the
preconceived impression that SNNs are sparse firing, the analysis and
optimization of inherent redundancy in SNNs have been largely overlooked, thus
the potential advantages of spike-based neuromorphic computing in accuracy and
energy efficiency are interfered. In this work, we pose and focus on three key
questions regarding the inherent redundancy in SNNs. We argue that the
redundancy is induced by the spatio-temporal invariance of SNNs, which enhances
the efficiency of parameter utilization but also invites lots of noise spikes.
Further, we analyze the effect of spatio-temporal invariance on the
spatio-temporal dynamics and spike firing of SNNs. Then, motivated by these
analyses, we propose an Advance Spatial Attention (ASA) module to harness SNNs'
redundancy, which can adaptively optimize their membrane potential distribution
by a pair of individual spatial attention sub-modules. In this way, noise spike
features are accurately regulated. Experimental results demonstrate that the
proposed method can significantly drop the spike firing with better performance
than state-of-the-art SNN baselines. Our code is available in
\url{https://github.com/BICLab/ASA-SNN}.Comment: Accepted by ICCV202
Spike-driven Transformer
Spiking Neural Networks (SNNs) provide an energy-efficient deep learning
option due to their unique spike-based event-driven (i.e., spike-driven)
paradigm. In this paper, we incorporate the spike-driven paradigm into
Transformer by the proposed Spike-driven Transformer with four unique
properties: 1) Event-driven, no calculation is triggered when the input of
Transformer is zero; 2) Binary spike communication, all matrix multiplications
associated with the spike matrix can be transformed into sparse additions; 3)
Self-attention with linear complexity at both token and channel dimensions; 4)
The operations between spike-form Query, Key, and Value are mask and addition.
Together, there are only sparse addition operations in the Spike-driven
Transformer. To this end, we design a novel Spike-Driven Self-Attention (SDSA),
which exploits only mask and addition operations without any multiplication,
and thus having up to lower computation energy than vanilla
self-attention. Especially in SDSA, the matrix multiplication between Query,
Key, and Value is designed as the mask operation. In addition, we rearrange all
residual connections in the vanilla Transformer before the activation functions
to ensure that all neurons transmit binary spike signals. It is shown that the
Spike-driven Transformer can achieve 77.1\% top-1 accuracy on ImageNet-1K,
which is the state-of-the-art result in the SNN field. The source code is
available at https://github.com/BICLab/Spike-Driven-Transformer