58 research outputs found
Dense Associative Memory for Pattern Recognition
A model of associative memory is studied, which stores and reliably retrieves
many more patterns than the number of neurons in the network. We propose a
simple duality between this dense associative memory and neural networks
commonly used in deep learning. On the associative memory side of this duality,
a family of models that smoothly interpolates between two limiting cases can be
constructed. One limit is referred to as the feature-matching mode of pattern
recognition, and the other one as the prototype regime. On the deep learning
side of the duality, this family corresponds to feedforward neural networks
with one hidden layer and various activation functions, which transmit the
activities of the visible neurons to the hidden layer. This family of
activation functions includes logistics, rectified linear units, and rectified
polynomials of higher degrees. The proposed duality makes it possible to apply
energy-based intuition from associative memory to analyze computational
properties of neural networks with unusual activation functions - the higher
rectified polynomials which until now have not been used in deep learning. The
utility of the dense memories is illustrated for two test cases: the logical
gate XOR and the recognition of handwritten digits from the MNIST data set.Comment: Accepted for publication at NIPS 201
Dense Associative Memory is Robust to Adversarial Inputs
Deep neural networks (DNN) trained in a supervised way suffer from two known
problems. First, the minima of the objective function used in learning
correspond to data points (also known as rubbish examples or fooling images)
that lack semantic similarity with the training data. Second, a clean input can
be changed by a small, and often imperceptible for human vision, perturbation,
so that the resulting deformed input is misclassified by the network. These
findings emphasize the differences between the ways DNN and humans classify
patterns, and raise a question of designing learning algorithms that more
accurately mimic human perception compared to the existing methods.
Our paper examines these questions within the framework of Dense Associative
Memory (DAM) models. These models are defined by the energy function, with
higher order (higher than quadratic) interactions between the neurons. We show
that in the limit when the power of the interaction vertex in the energy
function is sufficiently large, these models have the following three
properties. First, the minima of the objective function are free from rubbish
images, so that each minimum is a semantically meaningful pattern. Second,
artificial patterns poised precisely at the decision boundary look ambiguous to
human subjects and share aspects of both classes that are separated by that
decision boundary. Third, adversarial images constructed by models with small
power of the interaction vertex, which are equivalent to DNN with rectified
linear units (ReLU), fail to transfer to and fool the models with higher order
interactions. This opens up a possibility to use higher order models for
detecting and stopping malicious adversarial attacks. The presented results
suggest that DAM with higher order energy functions are closer to human visual
perception than DNN with ReLUs
Earthquake cycles and neural reverberations
Driven systems of interconnected blocks with stick-slip friction capture main features of earthquake processes. The microscopic dynamics closely resemble those of spiking nerve cells. We analyze the differences in the collective behavior and introduce a class of solvable models. We prove that the models exhibit rapid phase locking, a phenomenon of particular interest to both geophysics and neurobiology. We study the dependence upon initial conditions and system parameters, and discuss implications for earthquake modeling and neural computation
Neural network computation by in vitro transcriptional circuits
The structural similarity of neural networks and genetic regulatory networks
to digital circuits, and hence to each other, was noted from the
very beginning of their study [1, 2]. In this work, we propose a simple
biochemical system whose architecture mimics that of genetic regulation
and whose components allow for in vitro implementation of arbitrary
circuits. We use only two enzymes in addition to DNA and RNA
molecules: RNA polymerase (RNAP) and ribonuclease (RNase). We
develop a rate equation for in vitro transcriptional networks, and derive
a correspondence with general neural network rate equations [3].
As proof-of-principle demonstrations, an associative memory task and a
feedforward network computation are shown by simulation. A difference
between the neural network and biochemical models is also highlighted:
global coupling of rate equations through enzyme saturation can lead
to global feedback regulation, thus allowing a simple network without
explicit mutual inhibition to perform the winner-take-all computation.
Thus, the full complexity of the cell is not necessary for biochemical
computation: a wide range of functional behaviors can be achieved with
a small set of biochemical components
Rapid, parallel path planning by propagating wavefronts of spiking neural activity
Efficient path planning and navigation is critical for animals, robotics,
logistics and transportation. We study a model in which spatial navigation
problems can rapidly be solved in the brain by parallel mental exploration of
alternative routes using propagating waves of neural activity. A wave of
spiking activity propagates through a hippocampus-like network, altering the
synaptic connectivity. The resulting vector field of synaptic change then
guides a simulated animal to the appropriate selected target locations. We
demonstrate that the navigation problem can be solved using realistic, local
synaptic plasticity rules during a single passage of a wavefront. Our model can
find optimal solutions for competing possible targets or learn and navigate in
multiple environments. The model provides a hypothesis on the possible
computational mechanisms for optimal path planning in the brain, at the same
time it is useful for neuromorphic implementations, where the parallelism of
information processing proposed here can fully be harnessed in hardware
Nonlinear Network Dynamics on Earthquake Fault Systems
Earthquake faults occur in networks that have dynamical modes not displayed
by single isolated faults. Using simulations of the network of strike-slip
faults in southern California, we find that the physics depends critically on
both the interactions among the faults, which are determined by the geometry of
the fault network, as well as on the stress dissipation properties of the
nonlinear frictional physics, similar to the dynamics of integrate-and-fire
neural networks.Comment: 12 pages, 4 figure
Three-dimensional quantization of the electromagnetic field in dispersive and absorbing inhomogeneous dielectrics
A quantization scheme for the phenomenological Maxwell theory of the full
electromagnetic field in an inhomogeneous three-dimensional, dispersive and
absorbing dielectric medium is developed. The classical Maxwell equations with
spatially varying and Kramers-Kronig consistent permittivity are regarded as
operator-valued field equations, introducing additional current- and
charge-density operator fields in order to take into account the noise
associated with the dissipation in the medium. It is shown that the equal-time
commutation relations between the fundamental electromagnetic fields
and and the potentials and in the Coulomb gauge
can be expressed in terms of the Green tensor of the classical problem. From
the Green tensors for bulk material and an inhomogeneous medium consisting of
two bulk dielectrics with a common planar interface it is explicitly proven
that the well-known equal-time commutation relations of QED are preserved
Aptamer-based multiplexed proteomic technology for biomarker discovery
Interrogation of the human proteome in a highly multiplexed and efficient manner remains a coveted and challenging goal in biology. We present a new aptamer-based proteomic technology for biomarker discovery capable of simultaneously measuring thousands of proteins from small sample volumes (15 [mu]L of serum or plasma). Our current assay allows us to measure ~800 proteins with very low limits of detection (1 pM average), 7 logs of overall dynamic range, and 5% average coefficient of variation. This technology is enabled by a new generation of aptamers that contain chemically modified nucleotides, which greatly expand the physicochemical diversity of the large randomized nucleic acid libraries from which the aptamers are selected. Proteins in complex matrices such as plasma are measured with a process that transforms a signature of protein concentrations into a corresponding DNA aptamer concentration signature, which is then quantified with a DNA microarray. In essence, our assay takes advantage of the dual nature of aptamers as both folded binding entities with defined shapes and unique sequences recognizable by specific hybridization probes. To demonstrate the utility of our proteomics biomarker discovery technology, we applied it to a clinical study of chronic kidney disease (CKD). We identified two well known CKD biomarkers as well as an additional 58 potential CKD biomarkers. These results demonstrate the potential utility of our technology to discover unique protein signatures characteristic of various disease states. More generally, we describe a versatile and powerful tool that allows large-scale comparison of proteome profiles among discrete populations. This unbiased and highly multiplexed search engine will enable the discovery of novel biomarkers in a manner that is unencumbered by our incomplete knowledge of biology, thereby helping to advance the next generation of evidence-based medicine
The Neural Representation of Prospective Choice during Spatial Planning and Decisions
We are remarkably adept at inferring the consequences of our actions, yet the neuronal mechanisms that allow us to plan a sequence of novel choices remain unclear. We used functional magnetic resonance imaging (fMRI) to investigate how the human brain plans the shortest path to a goal in novel mazes with one (shallow maze) or two (deep maze) choice points. We observed two distinct anterior prefrontal responses to demanding choices at the second choice point: one in rostrodorsal medial prefrontal cortex (rd-mPFC)/superior frontal gyrus (SFG) that was also sensitive to (deactivated by) demanding initial choices and another in lateral frontopolar cortex (lFPC), which was only engaged by demanding choices at the second choice point. Furthermore, we identified hippocampal responses during planning that correlated with subsequent choice accuracy and response time, particularly in mazes affording sequential choices. Psychophysiological interaction (PPI) analyses showed that coupling between the hippocampus and rd-mPFC increases during sequential (deep versus shallow) planning and is higher before correct versus incorrect choices. In short, using a naturalistic spatial planning paradigm, we reveal how the human brain represents sequential choices during planning without extensive training. Our data highlight a network centred on the cortical midline and hippocampus that allows us to make prospective choices while maintaining initial choices during planning in novel environments
- …