3 research outputs found
Model Complexity of Program Phases
In resource limited computing systems, sequence prediction models must
operate under tight constraints. Various models are available that cater to
prediction under these conditions that in some way focus on reducing the cost
of implementation. These resource constrained sequence prediction models, in
practice, exhibit a fundamental tradeoff between the cost of implementation and
the quality of its predictions. This fundamental tradeoff seems to be largely
unexplored for models for different tasks. Here we formulate the necessary
theory and an associated empirical procedure to explore this tradeoff space for
a particular family of machine learning models such as deep neural networks. We
anticipate that the knowledge of the behavior of this tradeoff may be
beneficial in understanding the theoretical and practical limits of creation
and deployment of models for resource constrained tasks
Energy-based General Sequential Episodic Memory Networks at the Adiabatic Limit
The General Associative Memory Model (GAMM) has a constant state-dependant
energy surface that leads the output dynamics to fixed points, retrieving
single memories from a collection of memories that can be asynchronously
preloaded. We introduce a new class of General Sequential Episodic Memory
Models (GSEMM) that, in the adiabatic limit, exhibit temporally changing energy
surface, leading to a series of meta-stable states that are sequential episodic
memories. The dynamic energy surface is enabled by newly introduced asymmetric
synapses with signal propagation delays in the network's hidden layer. We study
the theoretical and empirical properties of two memory models from the GSEMM
class, differing in their activation functions. LISEM has non-linearities in
the feature layer, whereas DSEM has non-linearity in the hidden layer. In
principle, DSEM has a storage capacity that grows exponentially with the number
of neurons in the network. We introduce a learning rule for the synapses based
on the energy minimization principle and show it can learn single memories and
their sequential relationships online. This rule is similar to the Hebbian
learning algorithm and Spike-Timing Dependent Plasticity (STDP), which describe
conditions under which synapses between neurons change strength. Thus, GSEMM
combines the static and dynamic properties of episodic memory under a single
theoretical framework and bridges neuroscience, machine learning, and
artificial intelligence