Chaotic Characteristics of Discrete-time Linear Inclusion Dynamical Systems

In this paper, we study the chaotic behavior of a discrete-time linear inclusion.Comment: 7 pages; submitte

Controllability and Local Accessibility—A Normal Form Approach

Given a system with an uncontrollable linearization at the origin, we study the controllability of the system at equilibria around the origin. If the uncontrollable mode is nonzero, we prove that the system always has other equilibria around the origin. We also prove that these equilibria are linearly controllable provided a coefficient in the normal form is nonzero. Thus, the system is qualitatively changed from being linearly uncontrollable to linearly controllable when the equilibrium point is moved from the origin to a different one. This is called a bifurcation of controllability. As an application of the bifurcation, systems with a positive uncontrollable mode can be stabilized at a nearby equilibrium point. In the last part of this paper, simple sufficient conditions are proved for local accessibility of systems with an uncontrollable mode. Necessary conditions of controllability and local accessibility are also proved for systems with a convergent normal form

Online Training Through Time for Spiking Neural Networks

Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. Particularly, backpropagation through time (BPTT) with surrogate gradients (SG) is popularly used to achieve high performance in a very small number of time steps. However, it is at the cost of large memory consumption for training, lack of theoretical clarity for optimization, and inconsistency with the online property of biological learning and rules on neuromorphic hardware. Other works connect spike representations of SNNs with equivalent artificial neural network formulation and train SNNs by gradients from equivalent mappings to ensure descent directions. But they fail to achieve low latency and are also not online. In this work, we propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning by tracking presynaptic activities and leveraging instantaneous loss and gradients. Meanwhile, we theoretically analyze and prove that gradients of OTTT can provide a similar descent direction for optimization as gradients based on spike representations under both feedforward and recurrent conditions. OTTT only requires constant training memory costs agnostic to time steps, avoiding the significant memory costs of BPTT for GPU training. Furthermore, the update rule of OTTT is in the form of three-factor Hebbian learning, which could pave a path for online on-chip learning. With OTTT, it is the first time that two mainstream supervised SNN training methods, BPTT with SG and spike representation-based training, are connected, and meanwhile in a biologically plausible form. Experiments on CIFAR-10, CIFAR-100, ImageNet, and CIFAR10-DVS demonstrate the superior performance of our method on large-scale static and neuromorphic datasets in small time steps.Comment: Accepted by NeurIPS 202

Improved Federated Learning for Handling Long-tail Words

Automatic speech recognition (ASR) machine learning models are deployed on client devices that include speech interfaces. ASR models can benefit from continuous learning and adaptation to large-scale changes, e.g., as new words are added to the vocabulary. While federated learning can be utilized to enable continuous learning for ASR models in a privacy preserving manner, the trained model can perform poorly on rarely occurring, long-tail words if the distribution of data used to train the model is skewed and does not adequately represent long-tail words. This disclosure describes federated learning techniques to improve ASR model quality when interpreting long-tail words given an imbalanced data distribution. Two different approaches - probabilistic sampling and client loss weighting - are described herein. In probabilistic sampling, the federated clients that include fewer long-tail words are less likely to be selected during training. In client loss weighting, incorrect predictions on long-tail words are more heavily penalized than for other words