3 research outputs found
Training a Hopfield Variational Autoencoder with Equilibrium Propagation
On dedicated analog hardware, equilibrium propagation is an energy-efficient
alternative to backpropagation. In spite of its theoretical guarantees, its
application in the AI domain remains limited to the discriminative setting.
Meanwhile, despite its high computational demands, generative AI is on the
rise. In this paper, we demonstrate the application of Equilibrium Propagation
in training a variational autoencoder (VAE) for generative modeling. Leveraging
the symmetric nature of Hopfield networks, we propose using a single model to
serve as both the encoder and decoder which could effectively halve the
required chip size for VAE implementations, paving the way for more efficient
analog hardware configurations.Comment: Associative Memory & Hopfield Networks in 2023 (NeurIPS 2023
workshop
Exploring the Temperature-Dependent Phase Transition in Modern Hopfield Networks
The recent discovery of a connection between Transformers and Modern Hopfield
Networks (MHNs) has reignited the study of neural networks from a physical
energy-based perspective. This paper focuses on the pivotal effect of the
inverse temperature hyperparameter on the distribution of energy minima
of the MHN. To achieve this, the distribution of energy minima is tracked in a
simplified MHN in which equidistant normalised patterns are stored. This
network demonstrates a phase transition at a critical temperature
, from a single global attractor towards highly pattern
specific minima as is increased. Importantly, the dynamics are not
solely governed by the hyperparameter but are instead determined by an
effective inverse temperature which also depends on the
distribution and size of the stored patterns. Recognizing the role of
hyperparameters in the MHN could, in the future, aid researchers in the domain
of Transformers to optimise their initial choices, potentially reducing the
necessity for time and energy expensive hyperparameter fine-tuning.Comment: Accepted as poster for Associative Memory and Hopfield Networks
workshop at NeurIPS2