495 research outputs found
Direct Learning-Based Deep Spiking Neural Networks: A Review
The spiking neural network (SNN), as a promising brain-inspired computational
model with binary spike information transmission mechanism, rich
spatially-temporal dynamics, and event-driven characteristics, has received
extensive attention. However, its intricately discontinuous spike mechanism
brings difficulty to the optimization of the deep SNN. Since the surrogate
gradient method can greatly mitigate the optimization difficulty and shows
great potential in directly training deep SNNs, a variety of direct
learning-based deep SNN works have been proposed and achieved satisfying
progress in recent years. In this paper, we present a comprehensive survey of
these direct learning-based deep SNN works, mainly categorized into accuracy
improvement methods, efficiency improvement methods, and temporal dynamics
utilization methods. In addition, we also divide these categorizations into
finer granularities further to better organize and introduce them. Finally, the
challenges and trends that may be faced in future research are prospected.Comment: Accepted by Frontiers in Neuroscienc
Internal Language Model Estimation Through Explicit Context Vector Learning for Attention-based Encoder-decoder ASR
An end-to-end (E2E) ASR model implicitly learns a prior Internal Language
Model (ILM) from the training transcripts. To fuse an external LM using Bayes
posterior theory, the log likelihood produced by the ILM has to be accurately
estimated and subtracted. In this paper we propose two novel approaches to
estimate the ILM based on Listen-Attend-Spell (LAS) framework. The first method
is to replace the context vector of the LAS decoder at every time step with a
vector that is learned with training transcripts. Furthermore, we propose
another method that uses a lightweight feed-forward network to directly map
query vector to context vector in a dynamic sense. Since the context vectors
are learned by minimizing the perplexities on training transcripts, and their
estimation is independent of encoder output, hence the ILMs are accurately
learned for both methods. Experiments show that the ILMs achieve the lowest
perplexity, indicating the efficacy of the proposed methods. In addition, they
also significantly outperform the shallow fusion method, as well as two
previously proposed ILM Estimation (ILME) approaches on several datasets.Comment: Proceedings of INTERSPEEC
CILIATE: Towards Fairer Class-based Incremental Learning by Dataset and Training Refinement
Due to the model aging problem, Deep Neural Networks (DNNs) need updates to
adjust them to new data distributions. The common practice leverages
incremental learning (IL), e.g., Class-based Incremental Learning (CIL) that
updates output labels, to update the model with new data and a limited number
of old data. This avoids heavyweight training (from scratch) using conventional
methods and saves storage space by reducing the number of old data to store.
But it also leads to poor performance in fairness. In this paper, we show that
CIL suffers both dataset and algorithm bias problems, and existing solutions
can only partially solve the problem. We propose a novel framework, CILIATE,
that fixes both dataset and algorithm bias in CIL. It features a novel
differential analysis guided dataset and training refinement process that
identifies unique and important samples overlooked by existing CIL and enforces
the model to learn from them. Through this process, CILIATE improves the
fairness of CIL by 17.03%, 22.46%, and 31.79% compared to state-of-the-art
methods, iCaRL, BiC, and WA, respectively, based on our evaluation on three
popular datasets and widely used ResNet models
- …