85 research outputs found
Chaotic Characteristics of Discrete-time Linear Inclusion Dynamical Systems
In this paper, we study the chaotic behavior of a discrete-time linear
inclusion.Comment: 7 pages; submitte
Controllability and Local Accessibility—A Normal Form Approach
Given a system with an uncontrollable linearization at the origin, we study the controllability of the system at equilibria around the origin. If the uncontrollable mode is nonzero, we prove that the system always has other equilibria around the origin. We also prove that these equilibria are linearly controllable provided a coefficient in the normal form is nonzero. Thus, the system is qualitatively changed from being linearly uncontrollable to linearly controllable when the equilibrium point is moved from the origin to a different one. This is called a bifurcation of controllability. As an application of the bifurcation, systems with a positive uncontrollable mode can be stabilized at a nearby equilibrium point. In the last part of this paper, simple sufficient conditions are proved for local accessibility of systems with an uncontrollable mode. Necessary conditions of controllability and local accessibility are also proved for systems with a convergent normal form
Online Training Through Time for Spiking Neural Networks
Spiking neural networks (SNNs) are promising brain-inspired energy-efficient
models. Recent progress in training methods has enabled successful deep SNNs on
large-scale tasks with low latency. Particularly, backpropagation through time
(BPTT) with surrogate gradients (SG) is popularly used to achieve high
performance in a very small number of time steps. However, it is at the cost of
large memory consumption for training, lack of theoretical clarity for
optimization, and inconsistency with the online property of biological learning
and rules on neuromorphic hardware. Other works connect spike representations
of SNNs with equivalent artificial neural network formulation and train SNNs by
gradients from equivalent mappings to ensure descent directions. But they fail
to achieve low latency and are also not online. In this work, we propose online
training through time (OTTT) for SNNs, which is derived from BPTT to enable
forward-in-time learning by tracking presynaptic activities and leveraging
instantaneous loss and gradients. Meanwhile, we theoretically analyze and prove
that gradients of OTTT can provide a similar descent direction for optimization
as gradients based on spike representations under both feedforward and
recurrent conditions. OTTT only requires constant training memory costs
agnostic to time steps, avoiding the significant memory costs of BPTT for GPU
training. Furthermore, the update rule of OTTT is in the form of three-factor
Hebbian learning, which could pave a path for online on-chip learning. With
OTTT, it is the first time that two mainstream supervised SNN training methods,
BPTT with SG and spike representation-based training, are connected, and
meanwhile in a biologically plausible form. Experiments on CIFAR-10, CIFAR-100,
ImageNet, and CIFAR10-DVS demonstrate the superior performance of our method on
large-scale static and neuromorphic datasets in small time steps.Comment: Accepted by NeurIPS 202
Improved Federated Learning for Handling Long-tail Words
Automatic speech recognition (ASR) machine learning models are deployed on client devices that include speech interfaces. ASR models can benefit from continuous learning and adaptation to large-scale changes, e.g., as new words are added to the vocabulary. While federated learning can be utilized to enable continuous learning for ASR models in a privacy preserving manner, the trained model can perform poorly on rarely occurring, long-tail words if the distribution of data used to train the model is skewed and does not adequately represent long-tail words. This disclosure describes federated learning techniques to improve ASR model quality when interpreting long-tail words given an imbalanced data distribution. Two different approaches - probabilistic sampling and client loss weighting - are described herein. In probabilistic sampling, the federated clients that include fewer long-tail words are less likely to be selected during training. In client loss weighting, incorrect predictions on long-tail words are more heavily penalized than for other words
- …