Direct Feedback Alignment with Sparse Connections for Local Learning

Recent advances in deep neural networks (DNNs) owe their success to training algorithms that use backpropagation and gradient-descent. Backpropagation, while highly effective on von Neumann architectures, becomes inefficient when scaling to large networks. Commonly referred to as the weight transport problem, each neuron's dependence on the weights and errors located deeper in the network require exhaustive data movement which presents a key problem in enhancing the performance and energy-efficiency of machine-learning hardware. In this work, we propose a bio-plausible alternative to backpropagation drawing from advances in feedback alignment algorithms in which the error computation at a single synapse reduces to the product of three scalar values. Using a sparse feedback matrix, we show that a neuron needs only a fraction of the information previously used by the feedback alignment algorithms. Consequently, memory and compute can be partitioned and distributed whichever way produces the most efficient forward pass so long as a single error can be delivered to each neuron. Our results show orders of magnitude improvement in data movement and $2\times$ improvement in multiply-and-accumulate operations over backpropagation. Like previous work, we observe that any variant of feedback alignment suffers significant losses in classification accuracy on deep convolutional neural networks. By transferring trained convolutional layers and training the fully connected layers using direct feedback alignment, we demonstrate that direct feedback alignment can obtain results competitive with backpropagation. Furthermore, we observe that using an extremely sparse feedback matrix, rather than a dense one, results in a small accuracy drop while yielding hardware advantages. All the code and results are available under https://github.com/bcrafton/ssdfa.Comment: 15 pages, 8 figure

Top-down inputs enhance orientation selectivity in neurons of the primary visual cortex during perceptual learning.

Perceptual learning has been used to probe the mechanisms of cortical plasticity in the adult brain. Feedback projections are ubiquitous in the cortex, but little is known about their role in cortical plasticity. Here we explore the hypothesis that learning visual orientation discrimination involves learning-dependent plasticity of top-down feedback inputs from higher cortical areas, serving a different function from plasticity due to changes in recurrent connections within a cortical area. In a Hodgkin-Huxley-based spiking neural network model of visual cortex, we show that modulation of feedback inputs to V1 from higher cortical areas results in shunting inhibition in V1 neurons, which changes the response properties of V1 neurons. The orientation selectivity of V1 neurons is enhanced without changing orientation preference, preserving the topographic organizations in V1. These results provide new insights to the mechanisms of plasticity in the adult brain, reconciling apparently inconsistent experiments and providing a new hypothesis for a functional role of the feedback connections

New Connections Fund Grantseeker Feedback Study

Analysizes and reports on survey results from grantseekers in the James Irvine Foundation's pilot of the New Connections Fund, designed to be an open, competitive process for "unsolicited" grants. Includes feedback from three rounds of funding, including levels of grantee perceptions and satisfaction with the process, as well as suggestions for improvment

Gating Input to Visual Cortex by Feedback to LGN

Anatomical studies have documented massive back-projections from higher to lower visual cortices and to the lateral geniculate nucleus (LGN). The large number of synapses from these sources suggest that they should have a profound influence on the information carried by feed-forward inputs to these cells. However, the functional role of these connections is unclear. In order to explore the role of the feedback connections, we have recorded spike trains from electrodes placed in LGN in the macaque monkey under sufenta anesthesia, and have compared LGN cells' activity with and without suppression by cooling of feedback from primary visual cortex (V1). Normally, magno and parvo LGN cells show a wide range over which their responses are proportional to stimulus contrast. Inactivation of V1 feedback causes LGN cells to become more nonlinear and less sensitive to high contrast than during normal conditions. Responses during V1 inactivation have a similar shape to those of retinal ganglion cells. We have also tested the properties of the so-called extended surround as they relate to cortical activity and to influences on responses to LGN stimulation. A model of this data suggests an interpretation in terms of two fnuctional components of feedback: a contrast-dependent component which dominates at high input contrast, and a constant baseline level of inhibitory feedback. We also show that the influence of the extended surround on the classical center mechanism is more complicated than a simple integration model.National Institutes of Health (EY-05156); Office of Naval Research (N00014-95-1-409

The Effect of the Bulk Sales Article on Existing Commercial Practices

Power control is considered as an important means to combat near-far fading effects and maintain acceptable connections in wireless communications systems. When applying power control in practice, the performance is restricted by a number of fundamental limitations. Here, these are addressed from a control theory perspective. Limited update rate, limited feedback bandwidth, time delays, measurement errors, feedback errors, and filtering effects among other aspects all affect the resulting performance, and are related to radio channnel characteristics. Simulations further illustrate the hampering effects

The Juridical Status of Privileged Combatants Under the Geneva Protocol of 1977 Concerning International Conflicts

Centralized control and coordination of the connections in a wireless network is not possible in practice. To keep the delay from measure-ment instants to actuating the decisions, distributed control is required. This paper focuses on the uplink (from mobiles to base stations) and dis-cusses distributing the decision of when and when not to transmit data (distributed scheduling) to the mobiles. The scheme, uplink transmission timing, utilizes mobile transmitter power control feedback from the base station receiver to determine whether the channel is favorable or not compared to the average channel condition. Thereby, the battery consumption and disturbing power to other connections are reduced. The algorithm can be described as a feedback control system. Some transient behaviors are analyzed using systems theory, and supported by wireless network simulations of a system with a WCDMA (Wideband Code Division Multiple Access) radio interface as in most 3G systems

Laminar fMRI: applications for cognitive neuroscience

The cortex is a massively recurrent network, characterized by feedforward and feedback connections between brain areas as well as lateral connections within an area. Feedforward, horizontal and feedback responses largely activate separate layers of a cortical unit, meaning they can be dissociated by lamina-resolved neurophysiological techniques. Such techniques are invasive and are therefore rarely used in humans. However, recent developments in high spatial resolution fMRI allow for non-invasive, in vivo measurements of brain responses specific to separate cortical layers. This provides an important opportunity to dissociate between feedforward and feedback brain responses, and investigate communication between brain areas at a more fine- grained level than previously possible in the human species. In this review, we highlight recent studies that successfully used laminar fMRI to isolate layer-specific feedback responses in human sensory cortex. In addition, we review several areas of cognitive neuroscience that stand to benefit from this new technological development, highlighting contemporary hypotheses that yield testable predictions for laminar fMRI. We hope to encourage researchers with the opportunity to embrace this development in fMRI research, as we expect that many future advancements in our current understanding of human brain function will be gained from measuring lamina-specific brain responses