Information Theory and Machine Learning

The recent successes of machine learning, especially regarding systems based on deep neural networks, have encouraged further research activities and raised a new set of challenges in understanding and designing complex machine learning algorithms. New applications require learning algorithms to be distributed, have transferable learning results, use computation resources efficiently, convergence quickly on online settings, have performance guarantees, satisfy fairness or privacy constraints, incorporate domain knowledge on model structures, etc. A new wave of developments in statistical learning theory and information theory has set out to address these challenges. This Special Issue, "Machine Learning and Information Theory", aims to collect recent results in this direction reflecting a diverse spectrum of visions and efforts to extend conventional theories and develop analysis tools for these complex machine learning systems

Optimal coding of a random stimulus by a population of parallel neuron models

Copyright © 2007 SPIE - The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only. Copyright 2007 Society of Photo-Optical Instrumentation Engineers. This paper was published in Noise and Fluctuations in Biological, Biophysical, and Biomedical Systems, edited by Sergey M. Bezrukov, Proc. of SPIE Vol. 6602, 66020R and is made available as an electronic reprint with permission of SPIE. One print or electronic copy may be made for personal use only. Systematic or multiple reproduction, distribution to multiple locations via electronic or other means, duplication of any material in this paper for a fee or for commercial purposes, or modification of the content of the paper are prohibited.We examine the question of how a population of independently noisy sensory neurons should be configured to optimize the encoding of a random stimulus into sequences of neural action potentials. For the case where firing rates are the same in all neurons, we consider the problem of optimizing the noise distribution for a known stimulus distribution, and the converse problem of optimizing the stimulus for a given noise distribution. This work is related to suprathreshold stochastic resonance (SSR). It is shown that, for a large number of neurons, the SSR model is equivalent to a single rate-coding neuron with multiplicative output noise.Mark D. McDonnell, Nigel G. Stocks and Derek Abbot

Hardware Considerations for Signal Processing Systems: A Step Toward the Unconventional.

As we progress into the future, signal processing algorithms are becoming more computationally intensive and power hungry while the desire for mobile products and low power devices is also increasing. An integrated ASIC solution is one of the primary ways chip developers can improve performance and add functionality while keeping the power budget low. This work discusses ASIC hardware for both conventional and unconventional signal processing systems, and how integration, error resilience, emerging devices, and new algorithms can be leveraged by signal processing systems to further improve performance and enable new applications. Specifically this work presents three case studies: 1) a conventional and highly parallel mix signal cross-correlator ASIC for a weather satellite performing real-time synthetic aperture imaging, 2) an unconventional native stochastic computing architecture enabled by memristors, and 3) two unconventional sparse neural network ASICs for feature extraction and object classification. As improvements from technology scaling alone slow down, and the demand for energy efficient mobile electronics increases, such optimization techniques at the device, circuit, and system level will become more critical to advance signal processing capabilities in the future.PhDElectrical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/116685/1/knagphil_1.pd

Time Encoding via Unlimited Sampling: Theory, Algorithms and Hardware Validation

An alternative to conventional uniform sampling is that of time encoding, which converts continuous-time signals into streams of trigger times. This gives rise to Event-Driven Sampling (EDS) models. The data-driven nature of EDS acquisition is advantageous in terms of power consumption and time resolution and is inspired by the information representation in biological nervous systems. If an analog signal is outside a predefined dynamic range, then EDS generates a low density of trigger times, which in turn leads to recovery distortion due to aliasing. In this paper, inspired by the Unlimited Sensing Framework (USF), we propose a new EDS architecture that incorporates a modulo nonlinearity prior to acquisition that we refer to as the modulo EDS or MEDS. In MEDS, the modulo nonlinearity folds high dynamic range inputs into low dynamic range amplitudes, thus avoiding recovery distortion. In particular, we consider the asynchronous sigma-delta modulator (ASDM), previously used for low power analog-to-digital conversion. This novel MEDS based acquisition is enabled by a recent generalization of the modulo nonlinearity called modulo-hysteresis. We design a mathematically guaranteed recovery algorithm for bandlimited inputs based on a sampling rate criterion and provide reconstruction error bounds. We go beyond numerical experiments and also provide a first hardware validation of our approach, thus bridging the gap between theory and practice, while corroborating the conceptual underpinnings of our work.Comment: 27 pgs, 11 figures, IEEE Trans. Sig. Proc., accepted with minor revision

A Neural Signal Processor for Low-Latency Spike Inference

This thesis describes the development of a system that can assign identities to a population of single-units, in multi-electrode recordings, at single-spike resolution with low-latency. The system has two parts. The first is a Field-Programmable Gate Array (FPGA)-based Neural Signal Processor (NSP) that receives raw input and generates labelled spikes as output, a process referred to as real-time spike inference. The second is a piece of software (Spiketag) that runs on a PC, communicates with the NSP, and generates a spike-sorted model to guide the real-time spike inference. The NSP provides clocks and control signals to five 32-channel INTAN RHD2132 chips to manage the acquisition of 160 channels of raw neural data. In parallel, the NSP further filters, detects and extracts extracellular spike waveforms from the raw neural data recorded by tetrodes or silicon probes and assigns single-unit identity to each detected spike. A set of Python application programming interfaces (APIs) was developed in Spiketag to enable the communication between the NSP and the PC. These APIs allow the NSP to obtain a model from the PC, which holds parameters such as reference channels, spike detection thresholds, spike feature transformation matrix and vector quantized clusters generated by spike sorting a short recording session. Using the spike-sorted model, the NSP performs data acquisition and real-time spike inference simultaneously. Algorithmic modules were implemented in the FPGA and pipelined to compute during 40 ms acquisition intervals. At the output end of the FPGA NSP, the real-time assigned single-unit identity (spike-id) is packaged with the timestamp, the electrode group, and the spike features as a spike-id packet. Spike-id packets are asynchronously transmitted through a low-latency Peripheral Component Interconnect Express (PCIe) interface to the PC, producing the real-time spike trains. The real-time spike trains can be used for further processing, such as real-time decoding. Several types of ground-truth data, including intracellular/extracellular paired recordings, synthesized tetrode extracellular waveforms with ground-truth spike timing and high-channel-count silicon probe recordings with ground-truth animal positions during navigation were used to validate the low-latency (1 ms) and high-accuracy (as high as state-of-the-art offline sorting and decoding algorithms) of the NSP’s real-time spike inference and the NSP-based real-time population decoding performance