42,739 research outputs found
Topological exploration of artificial neuronal network dynamics
One of the paramount challenges in neuroscience is to understand the dynamics
of individual neurons and how they give rise to network dynamics when
interconnected. Historically, researchers have resorted to graph theory,
statistics, and statistical mechanics to describe the spatiotemporal structure
of such network dynamics. Our novel approach employs tools from algebraic
topology to characterize the global properties of network structure and
dynamics.
We propose a method based on persistent homology to automatically classify
network dynamics using topological features of spaces built from various
spike-train distances. We investigate the efficacy of our method by simulating
activity in three small artificial neural networks with different sets of
parameters, giving rise to dynamics that can be classified into four regimes.
We then compute three measures of spike train similarity and use persistent
homology to extract topological features that are fundamentally different from
those used in traditional methods. Our results show that a machine learning
classifier trained on these features can accurately predict the regime of the
network it was trained on and also generalize to other networks that were not
presented during training. Moreover, we demonstrate that using features
extracted from multiple spike-train distances systematically improves the
performance of our method
Dynamical Synapses Enhance Neural Information Processing: Gracefulness, Accuracy and Mobility
Experimental data have revealed that neuronal connection efficacy exhibits
two forms of short-term plasticity, namely, short-term depression (STD) and
short-term facilitation (STF). They have time constants residing between fast
neural signaling and rapid learning, and may serve as substrates for neural
systems manipulating temporal information on relevant time scales. The present
study investigates the impact of STD and STF on the dynamics of continuous
attractor neural networks (CANNs) and their potential roles in neural
information processing. We find that STD endows the network with slow-decaying
plateau behaviors-the network that is initially being stimulated to an active
state decays to a silent state very slowly on the time scale of STD rather than
on the time scale of neural signaling. This provides a mechanism for neural
systems to hold sensory memory easily and shut off persistent activities
gracefully. With STF, we find that the network can hold a memory trace of
external inputs in the facilitated neuronal interactions, which provides a way
to stabilize the network response to noisy inputs, leading to improved accuracy
in population decoding. Furthermore, we find that STD increases the mobility of
the network states. The increased mobility enhances the tracking performance of
the network in response to time-varying stimuli, leading to anticipative neural
responses. In general, we find that STD and STP tend to have opposite effects
on network dynamics and complementary computational advantages, suggesting that
the brain may employ a strategy of weighting them differentially depending on
the computational purpose.Comment: 40 pages, 17 figure
Deep neural networks architectures from the perspective of manifold learning
Despite significant advances in the field of deep learning in ap-plications
to various areas, an explanation of the learning pro-cess of neural network
models remains an important open ques-tion. The purpose of this paper is a
comprehensive comparison and description of neural network architectures in
terms of ge-ometry and topology. We focus on the internal representation of
neural networks and on the dynamics of changes in the topology and geometry of
a data manifold on different layers. In this paper, we use the concepts of
topological data analysis (TDA) and persistent homological fractal dimension.
We present a wide range of experiments with various datasets and configurations
of convolutional neural network (CNNs) architectures and Transformers in CV and
NLP tasks. Our work is a contribution to the development of the important field
of explainable and interpretable AI within the framework of geometrical deep
learning.Comment: 11 pages, 12 figures, PRAI2023. arXiv admin note: substantial text
overlap with arXiv:2204.0862
Learning Generative ConvNets via Multi-grid Modeling and Sampling
This paper proposes a multi-grid method for learning energy-based generative
ConvNet models of images. For each grid, we learn an energy-based probabilistic
model where the energy function is defined by a bottom-up convolutional neural
network (ConvNet or CNN). Learning such a model requires generating synthesized
examples from the model. Within each iteration of our learning algorithm, for
each observed training image, we generate synthesized images at multiple grids
by initializing the finite-step MCMC sampling from a minimal 1 x 1 version of
the training image. The synthesized image at each subsequent grid is obtained
by a finite-step MCMC initialized from the synthesized image generated at the
previous coarser grid. After obtaining the synthesized examples, the parameters
of the models at multiple grids are updated separately and simultaneously based
on the differences between synthesized and observed examples. We show that this
multi-grid method can learn realistic energy-based generative ConvNet models,
and it outperforms the original contrastive divergence (CD) and persistent CD.Comment: CVPR 201
On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models
This study investigates the effects of Markov chain Monte Carlo (MCMC)
sampling in unsupervised Maximum Likelihood (ML) learning. Our attention is
restricted to the family of unnormalized probability densities for which the
negative log density (or energy function) is a ConvNet. We find that many of
the techniques used to stabilize training in previous studies are not
necessary. ML learning with a ConvNet potential requires only a few
hyper-parameters and no regularization. Using this minimal framework, we
identify a variety of ML learning outcomes that depend solely on the
implementation of MCMC sampling.
On one hand, we show that it is easy to train an energy-based model which can
sample realistic images with short-run Langevin. ML can be effective and stable
even when MCMC samples have much higher energy than true steady-state samples
throughout training. Based on this insight, we introduce an ML method with
purely noise-initialized MCMC, high-quality short-run synthesis, and the same
budget as ML with informative MCMC initialization such as CD or PCD. Unlike
previous models, our energy model can obtain realistic high-diversity samples
from a noise signal after training.
On the other hand, ConvNet potentials learned with non-convergent MCMC do not
have a valid steady-state and cannot be considered approximate unnormalized
densities of the training data because long-run MCMC samples differ greatly
from observed images. We show that it is much harder to train a ConvNet
potential to learn a steady-state over realistic images. To our knowledge,
long-run MCMC samples of all previous models lose the realism of short-run
samples. With correct tuning of Langevin noise, we train the first ConvNet
potentials for which long-run and steady-state MCMC samples are realistic
images.Comment: Code available at: https://github.com/point0bar1/ebm-anatom
Short-Term Memory Through Persistent Activity: Evolution of Self-Stopping and Self-Sustaining Activity in Spiking Neural Networks
Memories in the brain are separated in two categories: short-term and
long-term memories. Long-term memories remain for a lifetime, while short-term
ones exist from a few milliseconds to a few minutes. Within short-term memory
studies, there is debate about what neural structure could implement it.
Indeed, mechanisms responsible for long-term memories appear inadequate for the
task. Instead, it has been proposed that short-term memories could be sustained
by the persistent activity of a group of neurons. In this work, we explore what
topology could sustain short-term memories, not by designing a model from
specific hypotheses, but through Darwinian evolution in order to obtain new
insights into its implementation. We evolved 10 networks capable of retaining
information for a fixed duration between 2 and 11s. Our main finding has been
that the evolution naturally created two functional modules in the network: one
which sustains the information containing primarily excitatory neurons, while
the other, which is responsible for forgetting, was composed mainly of
inhibitory neurons. This demonstrates how the balance between inhibition and
excitation plays an important role in cognition.Comment: 28 page
Associative memory of phase-coded spatiotemporal patterns in leaky Integrate and Fire networks
We study the collective dynamics of a Leaky Integrate and Fire network in
which precise relative phase relationship of spikes among neurons are stored,
as attractors of the dynamics, and selectively replayed at differentctime
scales. Using an STDP-based learning process, we store in the connectivity
several phase-coded spike patterns, and we find that, depending on the
excitability of the network, different working regimes are possible, with
transient or persistent replay activity induced by a brief signal. We introduce
an order parameter to evaluate the similarity between stored and recalled
phase-coded pattern, and measure the storage capacity. Modulation of spiking
thresholds during replay changes the frequency of the collective oscillation or
the number of spikes per cycle, keeping preserved the phases relationship. This
allows a coding scheme in which phase, rate and frequency are dissociable.
Robustness with respect to noise and heterogeneity of neurons parameters is
studied, showing that, since dynamics is a retrieval process, neurons preserve
stablecprecise phase relationship among units, keeping a unique frequency of
oscillation, even in noisy conditions and with heterogeneity of internal
parameters of the units
Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening
This work introduces a number of algebraic topology approaches, such as
multicomponent persistent homology, multi-level persistent homology and
electrostatic persistence for the representation, characterization, and
description of small molecules and biomolecular complexes. Multicomponent
persistent homology retains critical chemical and biological information during
the topological simplification of biomolecular geometric complexity.
Multi-level persistent homology enables a tailored topological description of
inter- and/or intra-molecular interactions of interest. Electrostatic
persistence incorporates partial charge information into topological
invariants. These topological methods are paired with Wasserstein distance to
characterize similarities between molecules and are further integrated with a
variety of machine learning algorithms, including k-nearest neighbors, ensemble
of trees, and deep convolutional neural networks, to manifest their descriptive
and predictive powers for chemical and biological problems. Extensive numerical
experiments involving more than 4,000 protein-ligand complexes from the PDBBind
database and near 100,000 ligands and decoys in the DUD database are performed
to test respectively the scoring power and the virtual screening power of the
proposed topological approaches. It is demonstrated that the present approaches
outperform the modern machine learning based methods in protein-ligand binding
affinity predictions and ligand-decoy discrimination
- …