6 research outputs found
Understanding and Improving Optimization in Predictive Coding Networks
Backpropagation (BP), the standard learning algorithm for artificial neural
networks, is often considered biologically implausible. In contrast, the
standard learning algorithm for predictive coding (PC) models in neuroscience,
known as the inference learning algorithm (IL), is a promising, bio-plausible
alternative. However, several challenges and questions hinder IL's application
to real-world problems. For example, IL is computationally demanding, and
without memory-intensive optimizers like Adam, IL may converge to poor local
minima. Moreover, although IL can reduce loss more quickly than BP, the reasons
for these speedups or their robustness remains unclear. In this paper, we
tackle these challenges by 1) altering the standard implementation of PC
circuits to substantially reduce computation, 2) developing a novel optimizer
that improves the convergence of IL without increasing memory usage, and 3)
establishing theoretical results that help elucidate the conditions under which
IL is sensitive to second and higher-order information
Thermodynamic Computing
The hardware and software foundations laid in the first half of the 20th
Century enabled the computing technologies that have transformed the world, but
these foundations are now under siege. The current computing paradigm, which is
the foundation of much of the current standards of living that we now enjoy,
faces fundamental limitations that are evident from several perspectives. In
terms of hardware, devices have become so small that we are struggling to
eliminate the effects of thermodynamic fluctuations, which are unavoidable at
the nanometer scale. In terms of software, our ability to imagine and program
effective computational abstractions and implementations are clearly challenged
in complex domains. In terms of systems, currently five percent of the power
generated in the US is used to run computing systems - this astonishing figure
is neither ecologically sustainable nor economically scalable. Economically,
the cost of building next-generation semiconductor fabrication plants has
soared past $10 billion. All of these difficulties - device scaling, software
complexity, adaptability, energy consumption, and fabrication economics -
indicate that the current computing paradigm has matured and that continued
improvements along this path will be limited. If technological progress is to
continue and corresponding social and economic benefits are to continue to
accrue, computing must become much more capable, energy efficient, and
affordable. We propose that progress in computing can continue under a united,
physically grounded, computational paradigm centered on thermodynamics. Herein
we propose a research agenda to extend these thermodynamic foundations into
complex, non-equilibrium, self-organizing systems and apply them holistically
to future computing systems that will harness nature's innate computational
capacity. We call this type of computing "Thermodynamic Computing" or TC.Comment: A Computing Community Consortium (CCC) workshop report, 36 page
A Framework to Explore Workload-Specific Performance and Lifetime Trade-offs in Neuromorphic Computing
status: publishe
Deep reinforcement learning with modulated Hebbian plus Q-network architecture
In this article, we consider a subclass of partially observable Markov decision process (POMDP) problems which we termed confounding POMDPs. In these types of POMDPs, temporal difference (TD)-based reinforcement learning (RL) algorithms struggle, as TD error cannot be easily derived from observations. We solve these types of problems using a new bio-inspired neural architecture that combines a modulated Hebbian network (MOHN) with deep Q-network (DQN), which we call modulated Hebbian plus Q-network architecture (MOHQA). The key idea is to use a Hebbian network with rarely correlated bio-inspired neural traces to bridge temporal delays between actions and rewards when confounding observations and sparse rewards result in inaccurate TD errors. In MOHQA, DQN learns low-level features and control, while the MOHN contributes to high-level decisions by associating rewards with past states and actions. Thus, the proposed architecture combines two modules with significantly different learning algorithms, a Hebbian associative network and a classical DQN pipeline, exploiting the advantages of both. Simulations on a set of POMDPs and on the Malmo environment show that the proposed algorithm improved DQN's results and even outperformed control tests with advantage-actor critic (A2C), quantile regression DQN with long short-term memory (QRDQN + LSTM), Monte Carlo policy gradient (REINFORCE), and aggregated memory for reinforcement learning (AMRL) algorithms on most difficult POMDPs with confounding stimuli and sparse rewards
Biological underpinnings for lifelong learning machines
Biological organisms learn from interactions with their environment throughout their lifetime. For artificial systems to successfully act and adapt in the real world, it is desirable to similarly be able to learn on a continual basis. This challenge is known as lifelong learning, and remains to a large extent unsolved. In this Perspective article, we identify a set of key capabilities that artificial systems will need to achieve lifelong learning. We describe a number of biological mechanisms, both neuronal and non-neuronal, that help explain how organisms solve these challenges, and present examples of biologically inspired models and biologically plausible mechanisms that have been applied to artificial systems in the quest towards development of lifelong learning machines. We discuss opportunities to further our understanding and advance the state of the art in lifelong learning, aiming to bridge the gap between natural and artificial intelligence