537 research outputs found
Bio-inspired learning and hardware acceleration with emerging memories
Machine Learning has permeated many aspects of engineering, ranging from the Internet of Things (IoT) applications to big data analytics. While computing resources available to implement these algorithms have become more powerful, both in terms of the complexity of problems that can be solved and the overall computing speed, the huge energy costs involved remains a significant challenge. The human brain, which has evolved over millions of years, is widely accepted as the most efficient control and cognitive processing platform. Neuro-biological studies have established that information processing in the human brain relies on impulse like signals emitted by neurons called action potentials. Motivated by these facts, the Spiking Neural Networks (SNNs), which are a bio-plausible version of neural networks have been proposed as an alternative computing paradigm where the timing of spikes generated by artificial neurons is central to its learning and inference capabilities. This dissertation demonstrates the computational power of the SNNs using conventional CMOS and emerging nanoscale hardware platforms.
The first half of this dissertation presents an SNN architecture which is trained using a supervised spike-based learning algorithm for the handwritten digit classification problem. This network achieves an accuracy of 98.17% on the MNIST test data-set, with about 4X fewer parameters compared to the state-of-the-art neural networks achieving over 99% accuracy. In addition, a scheme for parallelizing and speeding up the SNN simulation on a GPU platform is presented. The second half of this dissertation presents an optimal hardware design for accelerating SNN inference and training with SRAM (Static Random Access Memory) and nanoscale non-volatile memory (NVM) crossbar arrays. Three prominent NVM devices are studied for realizing hardware accelerators for SNNs: Phase Change Memory (PCM), Spin Transfer Torque RAM (STT-RAM) and Resistive RAM (RRAM). The analysis shows that a spike-based inference engine with crossbar arrays of STT-RAM bit-cells is 2X and 5X more efficient compared to PCM and RRAM memories, respectively. Furthermore, the STT-RAM design has nearly 6X higher throughput per unit Watt per unit area than that of an equivalent SRAM-based (Static Random Access Memory) design. A hardware accelerator with on-chip learning on an STT-RAM memory array is also designed, requiring bits of floating-point synaptic weight precision to reach the baseline SNN algorithmic performance on the MNIST dataset. The complete design with STT-RAM crossbar array achieves nearly 20X higher throughput per unit Watt per unit mm^2 than an equivalent design with SRAM memory.
In summary, this work demonstrates the potential of spike-based neuromorphic computing algorithms and its efficient realization in hardware based on conventional CMOS as well as emerging technologies. The schemes presented here can be further extended to design spike-based systems that can be ubiquitously deployed for energy and memory constrained edge computing applications
Unitary long-time evolution with quantum renormalization groups and artificial neural networks
In this work we combine quantum renormalization group approaches with deep
artificial neural networks for the description of the real-time evolution in
strongly disordered quantum matter. We find that this allows us to accurately
compute the long-time coherent dynamics of large, many-body localized systems
in non-perturbative regimes including the effects of many-body resonances.
Concretely, we use this approach to describe the spatiotemporal buildup of
many-body localized spin glass order in random Ising chains. We observe a
fundamental difference to a non-interacting Anderson insulating Ising chain,
where the order only develops over a finite spatial range. We further apply the
approach to strongly disordered two-dimensional Ising models highlighting that
our method can be used also for the description of the real-time dynamics of
nonergodic quantum matter in a general context.Comment: 4 pages, 3 figure
A Functional Perspective on Learning Symmetric Functions with Neural Networks
Symmetric functions, which take as input an unordered, fixed-size set, are
known to be universally representable by neural networks that enforce
permutation invariance. These architectures only give guarantees for fixed
input sizes, yet in many practical applications, including point clouds and
particle physics, a relevant notion of generalization should include varying
the input size. In this work we treat symmetric functions (of any size) as
functions over probability measures, and study the learning and representation
of neural networks defined on measures. By focusing on shallow architectures,
we establish approximation and generalization bounds under different choices of
regularization (such as RKHS and variation norms), that capture a hierarchy of
functional spaces with increasing degree of non-linear learning. The resulting
models can be learned efficiently and enjoy generalization guarantees that
extend across input sizes, as we verify empirically.Comment: Accepted to ICML 202
Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks
The convergence of GD and SGD when training mildly parameterized neural
networks starting from random initialization is studied. For a broad range of
models and loss functions, including the most commonly used square loss and
cross entropy loss, we prove an ``early stage convergence'' result. We show
that the loss is decreased by a significant amount in the early stage of the
training, and this decrease is fast. Furthurmore, for exponential type loss
functions, and under some assumptions on the training data, we show global
convergence of GD. Instead of relying on extreme over-parameterization, our
study is based on a microscopic analysis of the activation patterns for the
neurons, which helps us derive more powerful lower bounds for the gradient. The
results on activation patterns, which we call ``neuron partition'', help build
intuitions for understanding the behavior of neural networks' training
dynamics, and may be of independent interest
- …