454 research outputs found
Recommended from our members
Synaptic plasticity and memory addressing in biological and artificial neural networks
Biological brains are composed of neurons, interconnected by synapses to create large complex networks. Learning and memory occur, in large part, due to synaptic plasticity -- modifications in the efficacy of information transmission through these synaptic connections. Artificial neural networks model these with neural "units" which communicate through synaptic weights. Models of learning and memory propose synaptic plasticity rules that describe and predict the weight modifications. An equally important but under-evaluated question is the selection of \textit{which} synapses should be updated in response to a memory event. In this work, we attempt to separate the questions of synaptic plasticity from that of memory addressing.
Chapter 1 provides an overview of the problem of memory addressing and a summary of the solutions that have been considered in computational neuroscience and artificial intelligence, as well as those that may exist in biology. Chapter 2 presents in detail a solution to memory addressing and synaptic plasticity in the context of familiarity detection, suggesting strong feedforward weights and anti-Hebbian plasticity as the respective mechanisms. Chapter 3 proposes a model of recall, with storage performed by addressing through local third factors and neo-Hebbian plasticity, and retrieval by content-based addressing. In Chapter 4, we consider the problem of concurrent memory consolidation and memorization. Both storage and retrieval are performed by content-based addressing, but the plasticity rule itself is implemented by gradient descent, modulated according to whether an item should be stored in a distributed manner or memorized verbatim. However, the classical method for computing gradients in recurrent neural networks, backpropagation through time, is generally considered unbiological. In Chapter 5 we suggest a more realistic implementation through an approximation of recurrent backpropagation.
Taken together, these results propose a number of potential mechanisms for memory storage and retrieval, each of which separates the mechanism of synaptic updating -- plasticity -- from that of synapse selection -- addressing. Explicit studies of memory addressing may find applications not only in artificial intelligence but also in biology. In artificial networks, for example, selectively updating memories in large language models can help improve user privacy and security. In biological ones, understanding memory addressing can help with health outcomes and treating memory-based illnesses such as Alzheimers or PTSD
An Intelligent Time and Performance Efficient Algorithm for Aircraft Design Optimization
Die Optimierung des Flugzeugentwurfs erfordert die Beherrschung der komplexen Zusammenhänge mehrerer Disziplinen. Trotz seiner Abhängigkeit von einer Vielzahl unabhängiger Variablen zeichnet sich dieses komplexe Entwurfsproblem durch starke indirekte Verbindungen und eine daraus resultierende geringe Anzahl lokaler Minima aus. Kürzlich entwickelte intelligente Methoden, die auf selbstlernenden Algorithmen basieren, ermutigten die Suche nach einer diesem Bereich zugeordneten neuen Methode. Tatsächlich wird der in dieser Arbeit entwickelte Hybrid-Algorithmus (Cavus) auf zwei Hauptdesignfälle im Luft- und Raumfahrtbereich angewendet: Flugzeugentwurf- und Flugbahnoptimierung. Der implementierte neue Ansatz ist in der Lage, die Anzahl der Versuchspunkte ohne große Kompromisse zu reduzieren. Die Trendanalyse zeigt, dass der Cavus-Algorithmus für die komplexen Designprobleme, mit einer proportionalen Anzahl von Prüfpunkten konservativer ist, um die erfolgreichen Muster zu finden.
Aircraft Design Optimization requires mastering of the complex interrelationships of multiple disciplines. Despite its dependency on a diverse number of independent variables, this complex design problem has favourable nature as having strong indirect links and as a result a low number of local minimums. Recently developed intelligent methods that are based on self-learning algorithms encouraged finding a new method dedicated to this area. Indeed, the hybrid (Cavus) algorithm developed in this thesis is applied two main design cases in aerospace area: aircraft design optimization and trajectory optimization. The implemented new approach is capable of reducing the number of trial points without much compromise. The trend analysis shows that, for the complex design problems the Cavus algorithm is more conservative with a proportional number of trial points in finding the successful patterns
Long Sequence Hopfield Memory
Sequence memory is an essential attribute of natural and artificial
intelligence that enables agents to encode, store, and retrieve complex
sequences of stimuli and actions. Computational models of sequence memory have
been proposed where recurrent Hopfield-like neural networks are trained with
temporally asymmetric Hebbian rules. However, these networks suffer from
limited sequence capacity (maximal length of the stored sequence) due to
interference between the memories. Inspired by recent work on Dense Associative
Memories, we expand the sequence capacity of these models by introducing a
nonlinear interaction term, enhancing separation between the patterns. We
derive novel scaling laws for sequence capacity with respect to network size,
significantly outperforming existing scaling laws for models based on
traditional Hopfield networks, and verify these theoretical results with
numerical simulation. Moreover, we introduce a generalized pseudoinverse rule
to recall sequences of highly correlated patterns. Finally, we extend this
model to store sequences with variable timing between states' transitions and
describe a biologically-plausible implementation, with connections to motor
neuroscience.Comment: NeurIPS 2023 Camera-Ready, 41 page
Analog Photonics Computing for Information Processing, Inference and Optimisation
This review presents an overview of the current state-of-the-art in photonics
computing, which leverages photons, photons coupled with matter, and
optics-related technologies for effective and efficient computational purposes.
It covers the history and development of photonics computing and modern
analogue computing platforms and architectures, focusing on optimization tasks
and neural network implementations. The authors examine special-purpose
optimizers, mathematical descriptions of photonics optimizers, and their
various interconnections. Disparate applications are discussed, including
direct encoding, logistics, finance, phase retrieval, machine learning, neural
networks, probabilistic graphical models, and image processing, among many
others. The main directions of technological advancement and associated
challenges in photonics computing are explored, along with an assessment of its
efficiency. Finally, the paper discusses prospects and the field of optical
quantum computing, providing insights into the potential applications of this
technology.Comment: Invited submission by Journal of Advanced Quantum Technologies;
accepted version 5/06/202
A Semantic Framework for Neural-Symbolic Computing
Two approaches to AI, neural networks and symbolic systems, have been proven
very successful for an array of AI problems. However, neither has been able to
achieve the general reasoning ability required for human-like intelligence. It
has been argued that this is due to inherent weaknesses in each approach.
Luckily, these weaknesses appear to be complementary, with symbolic systems
being adept at the kinds of things neural networks have trouble with and
vice-versa. The field of neural-symbolic AI attempts to exploit this asymmetry
by combining neural networks and symbolic AI into integrated systems. Often
this has been done by encoding symbolic knowledge into neural networks.
Unfortunately, although many different methods for this have been proposed,
there is no common definition of an encoding to compare them. We seek to
rectify this problem by introducing a semantic framework for neural-symbolic
AI, which is then shown to be general enough to account for a large family of
neural-symbolic systems. We provide a number of examples and proofs of the
application of the framework to the neural encoding of various forms of
knowledge representation and neural network. These, at first sight disparate
approaches, are all shown to fall within the framework's formal definition of
what we call semantic encoding for neural-symbolic AI
Sequential Memory with Temporal Predictive Coding
Forming accurate memory of sequential stimuli is a fundamental function of
biological agents. However, the computational mechanism underlying sequential
memory in the brain remains unclear. Inspired by neuroscience theories and
recent successes in applying predictive coding (PC) to \emph{static} memory
tasks, in this work we propose a novel PC-based model for \emph{sequential}
memory, called \emph{temporal predictive coding} (tPC). We show that our tPC
models can memorize and retrieve sequential inputs accurately with a
biologically plausible neural implementation. Importantly, our analytical study
reveals that tPC can be viewed as a classical Asymmetric Hopfield Network (AHN)
with an implicit statistical whitening process, which leads to more stable
performance in sequential memory tasks of structured inputs. Moreover, we find
that tPC exhibits properties consistent with behavioral observations and
theories in neuroscience, thereby strengthening its biological relevance. Our
work establishes a possible computational mechanism underlying sequential
memory in the brain that can also be theoretically interpreted using existing
memory model frameworks.Comment: 37th Conference on Neural Information Processing Systems (NeurIPS
2023
Flexible Phase Dynamics for Bio-Plausible Contrastive Learning
Many learning algorithms used as normative models in neuroscience or as
candidate approaches for learning on neuromorphic chips learn by contrasting
one set of network states with another. These Contrastive Learning (CL)
algorithms are traditionally implemented with rigid, temporally non-local, and
periodic learning dynamics that could limit the range of physical systems
capable of harnessing CL. In this study, we build on recent work exploring how
CL might be implemented by biological or neurmorphic systems and show that this
form of learning can be made temporally local, and can still function even if
many of the dynamical requirements of standard training procedures are relaxed.
Thanks to a set of general theorems corroborated by numerical experiments
across several CL models, our results provide theoretical foundations for the
study and development of CL methods for biological and neuromorphic neural
networks.Comment: 23 pages, 4 figures. Paper accepted to ICML and update includes
changes made based on reviewer feedbac
Behavior quantification as the missing link between fields: Tools for digital psychiatry and their role in the future of neurobiology
The great behavioral heterogeneity observed between individuals with the same
psychiatric disorder and even within one individual over time complicates both
clinical practice and biomedical research. However, modern technologies are an
exciting opportunity to improve behavioral characterization. Existing
psychiatry methods that are qualitative or unscalable, such as patient surveys
or clinical interviews, can now be collected at a greater capacity and analyzed
to produce new quantitative measures. Furthermore, recent capabilities for
continuous collection of passive sensor streams, such as phone GPS or
smartwatch accelerometer, open avenues of novel questioning that were
previously entirely unrealistic. Their temporally dense nature enables a
cohesive study of real-time neural and behavioral signals.
To develop comprehensive neurobiological models of psychiatric disease, it
will be critical to first develop strong methods for behavioral quantification.
There is huge potential in what can theoretically be captured by current
technologies, but this in itself presents a large computational challenge --
one that will necessitate new data processing tools, new machine learning
techniques, and ultimately a shift in how interdisciplinary work is conducted.
In my thesis, I detail research projects that take different perspectives on
digital psychiatry, subsequently tying ideas together with a concluding
discussion on the future of the field. I also provide software infrastructure
where relevant, with extensive documentation.
Major contributions include scientific arguments and proof of concept results
for daily free-form audio journals as an underappreciated psychiatry research
datatype, as well as novel stability theorems and pilot empirical success for a
proposed multi-area recurrent neural network architecture.Comment: PhD thesis cop
Theory of Mind Might Have Spontaneously Emerged in Large Language Models
We explore the intriguing possibility that theory of mind (ToM), or the
uniquely human ability to impute unobservable mental states to others, might
have spontaneously emerged in large language models (LLMs). We designed 40
false-belief tasks, considered a gold standard in testing ToM in humans, and
administered them to several LLMs. Each task included a false-belief scenario,
three closely matched true-belief controls, and the reversed versions of all
four. Smaller and older models solved no tasks; GPT-3-davinci-001 (from May
2020) and GPT-3-davinci-002 (from January 2022) solved 10%; and
GPT-3-davinci-003 (from November 2022) and ChatGPT-3.5-turbo (from March 2023)
solved 35% of the tasks, mirroring the performance of three-year-old children.
ChatGPT-4 (from June 2023) solved 90% of the tasks, matching the performance of
seven-year-old children. These findings suggest the intriguing possibility that
ToM, previously considered exclusive to humans, may have spontaneously emerged
as a byproduct of LLMs' improving language skills.Comment: TRY RUNNING ToM EXPERIMENTS ON YOUR OWN: The code and tasks used in
this study are available at Colab
(https://colab.research.google.com/drive/1ZRtmw87CdA4xp24DNS_Ik_uA2ypaRnoU).
Don't worry if you are not an expert coder, you should be able to run this
code with no-to-minimum Python skills. Or copy-paste the tasks to ChatGPT's
web interfac
- …