133 research outputs found
A Multi-In and Multi-Out Dendritic Neuron Model and its Optimization
Artificial neural networks (ANNs), inspired by the interconnection of real
neurons, have achieved unprecedented success in various fields such as computer
vision and natural language processing. Recently, a novel mathematical ANN
model, known as the dendritic neuron model (DNM), has been proposed to address
nonlinear problems by more accurately reflecting the structure of real neurons.
However, the single-output design limits its capability to handle multi-output
tasks, significantly lowering its applications. In this paper, we propose a
novel multi-in and multi-out dendritic neuron model (MODN) to tackle
multi-output tasks. Our core idea is to introduce a filtering matrix to the
soma layer to adaptively select the desired dendrites to regress each output.
Because such a matrix is designed to be learnable, MODN can explore the
relationship between each dendrite and output to provide a better solution to
downstream tasks. We also model a telodendron layer into MODN to simulate
better the real neuron behavior. Importantly, MODN is a more general and
unified framework that can be naturally specialized as the DNM by customizing
the filtering matrix. To explore the optimization of MODN, we investigate both
heuristic and gradient-based optimizers and introduce a 2-step training method
for MODN. Extensive experimental results performed on 11 datasets on both
binary and multi-class classification tasks demonstrate the effectiveness of
MODN, with respect to accuracy, convergence, and generality
Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networks
Throughout history, the development of artificial intelligence, particularly
artificial neural networks, has been open to and constantly inspired by the
increasingly deepened understanding of the brain, such as the inspiration of
neocognitron, which is the pioneering work of convolutional neural networks.
Per the motives of the emerging field: NeuroAI, a great amount of neuroscience
knowledge can help catalyze the next generation of AI by endowing a network
with more powerful capabilities. As we know, the human brain has numerous
morphologically and functionally different neurons, while artificial neural
networks are almost exclusively built on a single neuron type. In the human
brain, neuronal diversity is an enabling factor for all kinds of biological
intelligent behaviors. Since an artificial network is a miniature of the human
brain, introducing neuronal diversity should be valuable in terms of addressing
those essential problems of artificial networks such as efficiency,
interpretability, and memory. In this Primer, we first discuss the
preliminaries of biological neuronal diversity and the characteristics of
information transmission and processing in a biological neuron. Then, we review
studies of designing new neurons for artificial networks. Next, we discuss what
gains can neuronal diversity bring into artificial networks and exemplary
applications in several important fields. Lastly, we discuss the challenges and
future directions of neuronal diversity to explore the potential of NeuroAI
Single Biological Neurons as Temporally Precise Spatio-Temporal Pattern Recognizers
This PhD thesis is focused on the central idea that single neurons in the
brain should be regarded as temporally precise and highly complex
spatio-temporal pattern recognizers. This is opposed to the prevalent view of
biological neurons as simple and mainly spatial pattern recognizers by most
neuroscientists today. In this thesis, I will attempt to demonstrate that this
is an important distinction, predominantly because the above-mentioned
computational properties of single neurons have far-reaching implications with
respect to the various brain circuits that neurons compose, and on how
information is encoded by neuronal activity in the brain. Namely, that these
particular "low-level" details at the single neuron level have substantial
system-wide ramifications. In the introduction we will highlight the main
components that comprise a neural microcircuit that can perform useful
computations and illustrate the inter-dependence of these components from a
system perspective. In chapter 1 we discuss the great complexity of the
spatio-temporal input-output relationship of cortical neurons that are the
result of morphological structure and biophysical properties of the neuron. In
chapter 2 we demonstrate that single neurons can generate temporally precise
output patterns in response to specific spatio-temporal input patterns with a
very simple biologically plausible learning rule. In chapter 3, we use the
differentiable deep network analog of a realistic cortical neuron as a tool to
approximate the gradient of the output of the neuron with respect to its input
and use this capability in an attempt to teach the neuron to perform nonlinear
XOR operation. In chapter 4 we expand chapter 3 to describe extension of our
ideas to neuronal networks composed of many realistic biological spiking
neurons that represent either small microcircuits or entire brain regions
Compensação digital de distorções da fibra em sistemas de comunicação óticos de longa distância
The continuous increase of traffic demand in long-haul communications motivated
the network operators to look for receiver side techniques to mitigate the nonlinear
effects, resulting from signal-signal and signal-noise interaction, thus pushing the
current Capacity boundaries. Machine learning techniques are a very hot-topic
with given proofs in the most diverse applications. This dissertation aims to study
nonlinear impairments in long-haul coherent optical links and the current state of
the art in DSP techniques for impairment mitigation as well as the integration of
machine learning strategies in optical networks. Starting with a simplified fiber
model only impaired by ASE noise, we studied how to integrate an ANN-based
symbol estimator into the signal pipeline, enabling to validate the implementation
by matching the theoretical performance. We then moved to nonlinear proof of
concept with the incorporation of NLPN in the fiber link. Finally, we evaluated
the performance of the estimator under realistic simulations of Single and Multi-
Channel links in both SSFM and NZDSF fibers. The obtained results indicate
that even though it may be hard to find the best architecture, Nonlinear Symbol
Estimator networks have the potential to surpass more conventional DSP strategies.O aumento contínuo de tráfego nas comunicações de longo-alcance motivou os
operadores de rede a procurar técnicas do lado do receptor para atenuar os efeitos
não lineares resultantes da interacção sinal-sinal e sinal-ruído, alargando assim os
limites da capacidade do sistema. As técnicas de aprendizagem-máquina são um
tópico em ascenção com provas dadas nas mais diversas aplicações e setores. Esta
dissertação visa estudar as principais deficiências nas ligações de longo curso e o
actual estado da arte em técnicas de DSP para mitigação das mesmas, bem como
a integração de estratégias de aprendizagem-máquina em redes ópticas. Começando
com um modelo simplificado de fibra apenas perturbado pelo ruído ASE,
estudámos como integrar um estimador de símbolos baseado em ANN na cadeia
do prodessamento de sinal, conseguindo igualar o desempenho teórico. Procedemos
com uma prova de conceito perante não linearidades com a incorporação do
ruído de fase não linear na propagação. Finalmente, avaliamos o desempenho do
estimador com simulações realistas de links Single e Multi canal tanto em fibras
SSFM como NZDSF. Os resultados obtidos indicam que apesar da dificuldade de
encontrar a melhor arquitectura, a estimação não linear baseada em redes neuronais
têm o potencial para ultrapassar estratégias DSP mais convencionais.Mestrado em Engenharia Eletrónica e Telecomunicaçõe
NMDA-driven dendritic modulation enables multitask representation learning in hierarchical sensory processing pathways.
While sensory representations in the brain depend on context, it remains unclear how such modulations are implemented at the biophysical level, and how processing layers further in the hierarchy can extract useful features for each possible contextual state. Here, we demonstrate that dendritic N-Methyl-D-Aspartate spikes can, within physiological constraints, implement contextual modulation of feedforward processing. Such neuron-specific modulations exploit prior knowledge, encoded in stable feedforward weights, to achieve transfer learning across contexts. In a network of biophysically realistic neuron models with context-independent feedforward weights, we show that modulatory inputs to dendritic branches can solve linearly nonseparable learning problems with a Hebbian, error-modulated learning rule. We also demonstrate that local prediction of whether representations originate either from different inputs, or from different contextual modulations of the same input, results in representation learning of hierarchical feedforward weights across processing layers that accommodate a multitude of contexts
Infomorphic networks: Locally learning neural networks derived from partial information decomposition
Understanding the intricate cooperation among individual neurons in
performing complex tasks remains a challenge to this date. In this paper, we
propose a novel type of model neuron that emulates the functional
characteristics of biological neurons by optimizing an abstract local
information processing goal. We have previously formulated such a goal function
based on principles from partial information decomposition (PID). Here, we
present a corresponding parametric local learning rule which serves as the
foundation of "infomorphic networks" as a novel concrete model of neural
networks. We demonstrate the versatility of these networks to perform tasks
from supervised, unsupervised and memory learning. By leveraging the
explanatory power and interpretable nature of the PID framework, these
infomorphic networks represent a valuable tool to advance our understanding of
cortical function.Comment: 31 pages, 5 figure
Brain-Inspired Computational Intelligence via Predictive Coding
Artificial intelligence (AI) is rapidly becoming one of the key technologies
of this century. The majority of results in AI thus far have been achieved
using deep neural networks trained with the error backpropagation learning
algorithm. However, the ubiquitous adoption of this approach has highlighted
some important limitations such as substantial computational cost, difficulty
in quantifying uncertainty, lack of robustness, unreliability, and biological
implausibility. It is possible that addressing these limitations may require
schemes that are inspired and guided by neuroscience theories. One such theory,
called predictive coding (PC), has shown promising performance in machine
intelligence tasks, exhibiting exciting properties that make it potentially
valuable for the machine learning community: PC can model information
processing in different brain areas, can be used in cognitive control and
robotics, and has a solid mathematical grounding in variational inference,
offering a powerful inversion scheme for a specific class of continuous-state
generative models. With the hope of foregrounding research in this direction,
we survey the literature that has contributed to this perspective, highlighting
the many ways that PC might play a role in the future of machine learning and
computational intelligence at large.Comment: 37 Pages, 9 Figure
- …