Neural Networks. A General Framework for Non-Linear Function Approximation

The focus of this paper is on the neural network modelling approach that has gained increasing recognition in GIScience in recent years. The novelty about neural networks lies in their ability to model non-linear processes with few, if any, a priori assumptions about the nature of the data-generating process. The paper discusses some important issues that are central for successful application development. The scope is limited to feedforward neural networks, the leading example of neural networks. It is argued that failures in applications can usually be attributed to inadequate learning and/or inadequate complexity of the network model. Parameter estimation and a suitably chosen number of hidden units are, thus, of crucial importance for the success of real world neural network applications. The paper views network learning as an optimization problem, reviews two alternative approaches to network learning, and provides insights into current best practice to optimize complexity so to perform well on generalization tasks

The curse of dimensionality in operator learning

Neural operator architectures employ neural networks to approximate operators mapping between Banach spaces of functions; they may be used to accelerate model evaluations via emulation, or to discover models from data. Consequently, the methodology has received increasing attention over recent years, giving rise to the rapidly growing field of operator learning. The first contribution of this paper is to prove that for general classes of operators which are characterized only by their $C^r$- or Lipschitz-regularity, operator learning suffers from a curse of dimensionality, defined precisely here in terms of representations of the infinite-dimensional input and output function spaces. The result is applicable to a wide variety of existing neural operators, including PCA-Net, DeepONet and the FNO. The second contribution of the paper is to prove that the general curse of dimensionality can be overcome for solution operators defined by the Hamilton-Jacobi equation; this is achieved by leveraging additional structure in the underlying solution operator, going beyond regularity. To this end, a novel neural operator architecture is introduced, termed HJ-Net, which explicitly takes into account characteristic information of the underlying Hamiltonian system. Error and complexity estimates are derived for HJ-Net which show that this architecture can provably beat the curse of dimensionality related to the infinite-dimensional input and output function spaces

Innovative Techniques of Neuromodulation and Neuromodeling Based on Focal Non-Invasive Transcranial Magnetic Stimulation for Neurological Disorders

This dissertation aims to develop alternative technology that improves the current range of application of transcranial magnetic stimulation (TMS), on a scale that would permit defining specific non-invasive treatments for Parkinson’s disease and other neurological disorders. This is accomplished through three specific objectives. 1) The design of a neurostimulation system that increases the focality in TMS to regions of narrow target areas and variable depths in the brain cortex. 2) The assessment of the feasibility of novel high-frequency neuromodulation techniques that would allow increasing the focality in deeper areas beyond the cortical surface. 3) The development of a computational model of the motor pathway that allows studying the underlying mechanisms that originate PD symptoms, and the effects of TMS for the development of new treatments. The results successfully demonstrated the feasibility of using the novel high-frequency neuromodulation technique as an effective manner to reduce the necessary current in TMS coils. This reduction, which reached an order of magnitude of 100 times compared to commercial TMS technology, made it possible to reduce the coil sizes, making them more focal to targets (in the order of a few millimeters square). Finally, our innovative oscillatory model of the motor pathway allowed us to conclude that an internal regulatory mechanism that we believe neurons activate in advanced PD stages seems to be the pathological response of some neural subpopulations to dopamine depletion, trying to compensate for the downstream effects in the system. We also found that such a mechanism seems to the burstiness in PD

Stochastic neural network dynamics: synchronisation and control

Biological brains exhibit many interesting and complex behaviours. Understanding of the mechanisms behind brain behaviours is critical for continuing advancement in fields of research such as artificial intelligence and medicine. In particular, synchronisation of neuronal firing is associated with both improvements to and degeneration of the brain’s performance; increased synchronisation can lead to enhanced information-processing or neurological disorders such as epilepsy and Parkinson’s disease. As a result, it is desirable to research under which conditions synchronisation arises in neural networks and the possibility of controlling its prevalence. Stochastic ensembles of FitzHugh-Nagumo elements are used to model neural networks for numerical simulations and bifurcation analysis. The FitzHugh-Nagumo model is employed because of its realistic representation of the flow of sodium and potassium ions in addition to its advantageous property of allowing phase plane dynamics to be observed. Network characteristics such as connectivity, configuration and size are explored to determine their influences on global synchronisation generation in their respective systems. Oscillations in the mean-field are used to detect the presence of synchronisation over a range of coupling strength values. To ensure simulation efficiency, coupling strengths between neurons that are identical and fixed with time are investigated initially. Such networks where the interaction strengths are fixed are referred to as homogeneously coupled. The capacity of controlling and altering behaviours produced by homogeneously coupled networks is assessed through the application of weak and strong delayed feedback independently with various time delays. To imitate learning, the coupling strengths later deviate from one another and evolve with time in networks that are referred to as heterogeneously coupled. The intensity of coupling strength fluctuations and the rate at which coupling strengths converge to a desired mean value are studied to determine their impact upon synchronisation performance. The stochastic delay differential equations governing the numerically simulated networks are then converted into a finite set of deterministic cumulant equations by virtue of the Gaussian approximation method. Cumulant equations for maximal and sub-maximal connectivity are used to generate two-parameter bifurcation diagrams on the noise intensity and coupling strength plane, which provides qualitative agreement with numerical simulations. Analysis of artificial brain networks, in respect to biological brain networks, are discussed in light of recent research in sleep theor

MCMC-driven learning

This paper is intended to appear as a chapter for the Handbook of Markov Chain Monte Carlo. The goal of this chapter is to unify various problems at the intersection of Markov chain Monte Carlo (MCMC) and machine learning\unicode{x2014}which includes black-box variational inference, adaptive MCMC, normalizing flow construction and transport-assisted MCMC, surrogate-likelihood MCMC, coreset construction for MCMC with big data, Markov chain gradient descent, Markovian score climbing, and more\unicode{x2014}within one common framework. By doing so, the theory and methods developed for each may be translated and generalized

Heterogeneous neural networks: theory and applications

Aquest treball presenta una classe de funcions que serveixen de models neuronals generalitzats per ser usats en xarxes neuronals artificials. Es defineixen com una mesura de similitud que actúa com una definició flexible de neurona vista com un reconeixedor de patrons. La similitud proporciona una marc conceptual i serveix de cobertura unificadora de molts models neuronals de la literatura i d'exploració de noves instàncies de models de neurona. La visió basada en similitud porta amb naturalitat a integrar informació heterogènia, com ara quantitats contínues i discretes (nominals i ordinals), i difuses ó imprecises. Els valors perduts es tracten de manera explícita. Una neurona d'aquesta classe s'anomena neurona heterogènia i qualsevol arquitectura neuronal que en faci ús serà una Xarxa Neuronal Heterogènia.En aquest treball ens concentrem en xarxes neuronals endavant, com focus inicial d'estudi. Els algorismes d'aprenentatge són basats en algorisms evolutius, especialment extesos per treballar amb informació heterogènia. En aquesta tesi es descriu com una certa classe de neurones heterogènies porten a xarxes neuronals que mostren un rendiment molt satisfactori, comparable o superior al de xarxes neuronals tradicionals (com el perceptró multicapa ó la xarxa de base radial), molt especialment en presència d'informació heterogènia, usual en les bases de dades actuals.This work presents a class of functions serving as generalized neuron models to be used in artificial neural networks. They are cast into the common framework of computing a similarity function, a flexible definition of a neuron as a pattern recognizer. The similarity endows the model with a clear conceptual view and serves as a unification cover for many of the existing neural models, including those classically used for the MultiLayer Perceptron (MLP) and most of those used in Radial Basis Function Networks (RBF). These families of models are conceptually unified and their relation is clarified. The possibilities of deriving new instances are explored and several neuron models --representative of their families-- are proposed. The similarity view naturally leads to further extensions of the models to handle heterogeneous information, that is to say, information coming from sources radically different in character, including continuous and discrete (ordinal) numerical quantities, nominal (categorical) quantities, and fuzzy quantities. Missing data are also explicitly considered. A neuron of this class is called an heterogeneous neuron and any neural structure making use of them is an Heterogeneous Neural Network (HNN), regardless of the specific architecture or learning algorithm. Among them, in this work we concentrate on feed-forward networks, as the initial focus of study. The learning procedures may include a great variety of techniques, basically divided in derivative-based methods (such as the conjugate gradient)and evolutionary ones (such as variants of genetic algorithms).In this Thesis we also explore a number of directions towards the construction of better neuron models --within an integrant envelope-- more adapted to the problems they are meant to solve.It is described how a certain generic class of heterogeneous models leads to a satisfactory performance, comparable, and often better, to that of classical neural models, especially in the presence of heterogeneous information, imprecise or incomplete data, in a wide range of domains, most of them corresponding to real-world problems.Postprint (published version