Variational Bayesian Methods for Inferring Spatial Statistics and Nonlinear Dynamics

This thesis discusses four novel statistical methods and approximate inference techniques for analyzing structured neural and molecular sequence data. The main contributions are new algorithms for approximate inference and learning in Bayesian latent variable models involving spatial statistics and nonlinear dynamics. First, we propose an amortized variational inference method to separate a set of overlapping signals into spatially localized source functions without knowledge of the original signals or the mixing process. In the second part of this dissertation, we discuss two approaches for uncovering nonlinear, smooth latent dynamics from sequential data. Both algorithms construct variational families on extensions of nonlinear state space models where the underlying systems are described by hidden stochastic differential equations. The first method proposes a structured approximate posterior describing spatially-dependent linear dynamics, as well as an algorithm that relies on the fixed-point iteration method to achieve convergence. The second method proposes a variational backward simulation technique from an unbiased estimate of the marginal likelihood defined through a subsampling process. In the final chapter, we develop connections between discrete and continuous variational sequential search for Bayesian phylogenetic inference. We propose a technique that uses sequential search to construct a variational objective defined on the composite space of non-clock phylogenetic trees. Each of these techniques are motivated by real problems within computational biology and applied to provide insights into the underlying structure of complex data

Statistical Machine Learning Methods for High-dimensional Neural Population Data Analysis

Advances in techniques have been producing increasingly complex neural recordings, posing significant challenges for data analysis. This thesis discusses novel statistical methods for analyzing high-dimensional neural data. Part one discusses two extensions of state space models tailored to neural data analysis. First, we propose using a flexible count data distribution family in the observation model to faithfully capture over-dispersion and under-dispersion of the neural observations. Second, we incorporate nonlinear observation models into state space models to improve the flexibility of the model and get a more concise representation of the data. For both extensions, novel variational inference techniques are developed for model fitting, and simulated and real experiments show the advantages of our extensions. Part two discusses a fast region of interest (ROI) detection method for large-scale calcium imaging data based on structured matrix factorization. Part three discusses a method for sampling from a maximum entropy distribution with complicated constraints, which is useful for hypothesis testing for neural data analysis and many other applications related to maximum entropy formulation. We conclude the thesis with discussions and future works

Macroscopic Models and Phase Resetting of Coupled Biological Oscillators

This thesis concerns the derivation and analysis of macroscopic mathematical models for coupled biological oscillators. Circadian rhythms, heart beats, and brain waves are all examples of biological rhythms formed through the aggregation of the rhythmic contributions of thousands of cellular oscillations. These systems evolve in an extremely high-dimensional phase space having at least as many degrees of freedom as the number of oscillators. This high-dimensionality often contrasts with the low-dimensional behavior observed on the collective or macroscopic scale. Moreover, the macroscopic dynamics are often of greater interest in biological applications. Therefore, it is imperative that mathematical techniques are developed to extract low-dimensional models for the macroscopic behavior of these systems. One such mathematical technique is the Ott-Antonsen ansatz. The Ott-Antonsen ansatz may be applied to high-dimensional systems of heterogeneous coupled oscillators to derive an exact low-dimensional description of the system in terms of macroscopic variables. We apply the Ott-Antonsen technique to determine the sensitivity of collective oscillations to perturbations with applications to neuroscience. The power of the Ott-Antonsen technique comes at the expense of several limitations which could limit its applicability to biological systems. To address this we compare the Ott-Antonsen ansatz with experimental measurements of circadian rhythms and numerical simulations of several other biological systems. This analysis reveals that a key assumption of the Ott-Antonsen approach is violated in these systems. However, we discover a low-dimensional structure in these data sets and characterize its emergence through a simple argument depending only on general phase-locking behavior in coupled oscillator systems. We further demonstrate the structure's emergence in networks of noisy heterogeneous oscillators with complex network connectivity. We show how this structure may be applied as an ansatz to derive low-dimensional macroscopic models for oscillator population activity. This approach allows for the incorporation of cellular-level experimental data into the macroscopic model whose parameters and variables can then be directly associated with tissue- or organism-level properties, thereby elucidating the core properties driving the collective behavior of the system. We first apply our ansatz to study the impact of light on the mammalian circadian system. To begin we derive a low-dimensional macroscopic model for the core circadian clock in mammals. Significantly, the variables and parameters in our model have physiological interpretations and may be compared with experimental results. We focus on the effect of four key factors which help shape the mammalian phase response to light: heterogeneity in the population of oscillators, the structure of the typical light phase response curve, the fraction of oscillators which receive direct light input and changes in the coupling strengths associated with seasonal day-lengths. We find these factors can explain several experimental results and provide insight into the processing of light information in the mammalian circadian system. In a second application of our ansatz we derive a pair of low-dimensional models for human circadian rhythms. We fit the model parameters to measurements of light sensitivity in human subjects, and validate these parameter fits with three additional data sets. We compare our model predictions with those made by previous phenomenological models for human circadian rhythms. We find our models make new predictions concerning the amplitude dynamics of the human circadian clock and the light entrainment properties of the clock. These results could have applications to the development of light-based therapies for circadian disorders.PHDApplied and Interdisciplinary MathematicsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/138766/1/khannay_1.pd

Data assimilation for conductance-based neuronal models

This dissertation illustrates the use of data assimilation algorithms to estimate unobserved variables and unknown parameters of conductance-based neuronal models. Modern data assimilation (DA) techniques are widely used in climate science and weather prediction, but have only recently begun to be applied in neuroscience. The two main classes of DA techniques are sequential methods and variational methods. Throughout this work, twin experiments, where the data is synthetically generated from output of the model, are used to validate use of these techniques for conductance-based models observing only the voltage trace. In Chapter 1, these techniques are described in detail and the estimation problem for conductance-based neuron models is derived. In Chapter 2, these techniques are applied to a minimal conductance-based model, the Morris-Lecar model. This model exhibits qualitatively different types of neuronal excitability due to changes in the underlying bifurcation structure and it is shown that the DA methods can identify parameter sets that produce the correct bifurcation structure even with initial parameter guesses that correspond to a different excitability regime. This demonstrates the ability of DA techniques to perform nonlinear state and parameter estimation, and introduces the geometric structure of inferred models as a novel qualitative measure of estimation success. Chapter 3 extends the ideas of variational data assimilation to include a control term to relax the problem further in a process that is referred to as nudging from the geoscience community. The nudged 4D-Var is applied to twin experiments from a more complex, Hodgkin-Huxley-type two-compartment model for various time-sampling strategies. This controlled 4D-Var with nonuniform time-samplings is then applied to voltage traces from current-clamp recordings of suprachiasmatic nucleus neurons in diurnal rodents to improve upon our understanding of the driving forces in circadian (~24) rhythms of electrical activity. In Chapter 4 the complementary strengths of 4D-Var and UKF are leveraged to create a two-stage algorithm that uses 4D-Var to estimate fast timescale parameters and UKF for slow timescale parameters. This coupled approach is applied to data from a conductance-based model of neuronal bursting with distinctive slow and fast time-scales present in the dynamics. In Chapter 5, the ideas of identifiability and sensitivity are introduced. The Morris-Lecar model and a subset of its parameters are shown to be identifiable through the use of numerical techniques. Chapter 6 frames the selection of stimulus waveforms to inject into neurons during patch-clamp recordings as an optimal experimental design problem. Results on the optimal stimulus waveforms for improving the identifiability of parameters for a Hodgkin-Huxley-type model are presented. Chapter 7 shows the preliminary application of data assimilation for voltage-clamp, rather than current-clamp, data and expands on voltage-clamp principles to formulate a reduced assimilation problem driven by the observed voltage. Concluding thoughts are given in Chapter 8

Bayesian inference for stable differential equation models with applications in computational neuroscience

Inference for mechanistic models is challenging because of nonlinear interactions between model parameters and a lack of identifiability. Here we focus on a specific class of mechanistic models, which we term stable differential equations. The dynamics in these models are approximately linear around a stable fixed point of the system. We exploit this property to develop fast approximate methods for posterior inference. We first illustrate our approach using simulated EEG data on the Liley et al model, a mechanistic neural population model. Then we apply our methods to experimental EEG data from rats to estimate how parameters in the Liley et al model vary with level of isoflurane anaesthesia. More generally, stable differential equation models and the corresponding inference methods are useful for analysis of stationary time-series data. Compared to the existing state-of-the art, our methods are several orders of magnitude faster, and are particularly suited to analysis of long time-series (>10,000 time-points) and models of moderate dimension (10-50 state variables and 10-50 parameters.

Building theories of neural circuits with machine learning

As theoretical neuroscience has grown as a field, machine learning techniques have played an increasingly important role in the development and evaluation of theories of neural computation. Today, machine learning is used in a variety of neuroscientific contexts from statistical inference to neural network training to normative modeling. This dissertation introduces machine learning techniques for use across the various domains of theoretical neuroscience, and the application of these techniques to build theories of neural circuits. First, we introduce a variety of optimization techniques for normative modeling of neural activity, which were used to evaluate theories of primary motor cortex (M1) and supplementary motor area (SMA). Specifically, neural responses during a cycling task performed by monkeys displayed distinctive dynamical geometries, which motivated hypotheses of how these geometries conferred computational properties necessary for the robust production of cyclic movements. By using normative optimization techniques to predict neural responses encoding muscle activity while ascribing to an “untangled” geometry, we found that minimal tangling was an accurate model of M1. Analyses with trajectory constrained RNNs showed that such an organization of M1 neural activity confers noise robustness, and that minimally “divergent” trajectories in SMA enable the tracking of contextual factors. In the remainder of the dissertation, we focus on the introduction and application of deep generative modeling techniques for theoretical neuroscience. Specifically, both techniques employ recent advancements in approaches to deep generative modeling -- normalizing flows -- to capture complex parametric structure in neural models. The first technique, which is designed for statistical generative models, enables look-up inference in intractable exponential family models. The efficiency of this technique is demonstrated by inferring neural firing rates in a log-gaussian poisson model of spiking responses to drift gratings in primary visual cortex. The second technique is designed for statistical inference in mechanistic models, where the inferred parameter distribution is constrained to produce emergent properties of computation. Once fit, the deep generative model confers analytic tools for quantifying the parametric structure giving rise to emergent properties. This technique was used for novel scientific insight into the nature of neuron-type variability in primary visual cortex and of distinct connectivity regimes of rapid task switching in superior colliculus

Synaptic Learning for Neuromorphic Vision - Processing Address Events with Spiking Neural Networks

Das Gehirn übertrifft herkömmliche Computerarchitekturen in Bezug auf Energieeffizienz, Robustheit und Anpassungsfähigkeit. Diese Aspekte sind auch für neue Technologien wichtig. Es lohnt sich daher, zu untersuchen, welche biologischen Prozesse das Gehirn zu Berechnungen befähigen und wie sie in Silizium umgesetzt werden können. Um sich davon inspirieren zu lassen, wie das Gehirn Berechnungen durchführt, ist ein Paradigmenwechsel im Vergleich zu herkömmlichen Computerarchitekturen erforderlich. Tatsächlich besteht das Gehirn aus Nervenzellen, Neuronen genannt, die über Synapsen miteinander verbunden sind und selbstorganisierte Netzwerke bilden. Neuronen und Synapsen sind komplexe dynamische Systeme, die durch biochemische und elektrische Reaktionen gesteuert werden. Infolgedessen können sie ihre Berechnungen nur auf lokale Informationen stützen. Zusätzlich kommunizieren Neuronen untereinander mit kurzen elektrischen Impulsen, den so genannten Spikes, die sich über Synapsen bewegen. Computational Neuroscientists versuchen, diese Berechnungen mit spikenden neuronalen Netzen zu modellieren. Wenn sie auf dedizierter neuromorpher Hardware implementiert werden, können spikende neuronale Netze wie das Gehirn schnelle, energieeffiziente Berechnungen durchführen. Bis vor kurzem waren die Vorteile dieser Technologie aufgrund des Mangels an funktionellen Methoden zur Programmierung von spikenden neuronalen Netzen begrenzt. Lernen ist ein Paradigma für die Programmierung von spikenden neuronalen Netzen, bei dem sich Neuronen selbst zu funktionalen Netzen organisieren. Wie im Gehirn basiert das Lernen in neuromorpher Hardware auf synaptischer Plastizität. Synaptische Plastizitätsregeln charakterisieren Gewichtsaktualisierungen im Hinblick auf Informationen, die lokal an der Synapse anliegen. Das Lernen geschieht also kontinuierlich und online, während sensorischer Input in das Netzwerk gestreamt wird. Herkömmliche tiefe neuronale Netze werden üblicherweise durch Gradientenabstieg trainiert. Die durch die biologische Lerndynamik auferlegten Einschränkungen verhindern jedoch die Verwendung der konventionellen Backpropagation zur Berechnung der Gradienten. Beispielsweise behindern kontinuierliche Aktualisierungen den synchronen Wechsel zwischen Vorwärts- und Rückwärtsphasen. Darüber hinaus verhindern Gedächtnisbeschränkungen, dass die Geschichte der neuronalen Aktivität im Neuron gespeichert wird, so dass Verfahren wie Backpropagation-Through-Time nicht möglich sind. Neuartige Lösungen für diese Probleme wurden von Computational Neuroscientists innerhalb des Zeitrahmens dieser Arbeit vorgeschlagen. In dieser Arbeit werden spikende neuronaler Netzwerke entwickelt, um Aufgaben der visuomotorischen Neurorobotik zu lösen. In der Tat entwickelten sich biologische neuronale Netze ursprünglich zur Steuerung des Körpers. Die Robotik stellt also den künstlichen Körper für das künstliche Gehirn zur Verfügung. Auf der einen Seite trägt diese Arbeit zu den gegenwärtigen Bemühungen um das Verständnis des Gehirns bei, indem sie schwierige Closed-Loop-Benchmarks liefert, ähnlich dem, was dem biologischen Gehirn widerfährt. Auf der anderen Seite werden neue Wege zur Lösung traditioneller Robotik Probleme vorgestellt, die auf vom Gehirn inspirierten Paradigmen basieren. Die Forschung wird in zwei Schritten durchgeführt. Zunächst werden vielversprechende synaptische Plastizitätsregeln identifiziert und mit ereignisbasierten Vision-Benchmarks aus der realen Welt verglichen. Zweitens werden neuartige Methoden zur Abbildung visueller Repräsentationen auf motorische Befehle vorgestellt. Neuromorphe visuelle Sensoren stellen einen wichtigen Schritt auf dem Weg zu hirninspirierten Paradigmen dar. Im Gegensatz zu herkömmlichen Kameras senden diese Sensoren Adressereignisse aus, die lokalen Änderungen der Lichtintensität entsprechen. Das ereignisbasierte Paradigma ermöglicht eine energieeffiziente und schnelle Bildverarbeitung, erfordert aber die Ableitung neuer asynchroner Algorithmen. Spikende neuronale Netze stellen eine Untergruppe von asynchronen Algorithmen dar, die vom Gehirn inspiriert und für neuromorphe Hardwaretechnologie geeignet sind. In enger Zusammenarbeit mit Computational Neuroscientists werden erfolgreiche Methoden zum Erlernen räumlich-zeitlicher Abstraktionen aus der Adressereignisdarstellung berichtet. Es wird gezeigt, dass Top-Down-Regeln der synaptischen Plastizität, die zur Optimierung einer objektiven Funktion abgeleitet wurden, die Bottom-Up-Regeln übertreffen, die allein auf Beobachtungen im Gehirn basieren. Mit dieser Einsicht wird eine neue synaptische Plastizitätsregel namens "Deep Continuous Local Learning" eingeführt, die derzeit den neuesten Stand der Technik bei ereignisbasierten Vision-Benchmarks erreicht. Diese Regel wurde während eines Aufenthalts an der Universität von Kalifornien, Irvine, gemeinsam abgeleitet, implementiert und evaluiert. Im zweiten Teil dieser Arbeit wird der visuomotorische Kreis geschlossen, indem die gelernten visuellen Repräsentationen auf motorische Befehle abgebildet werden. Drei Ansätze werden diskutiert, um ein visuomotorisches Mapping zu erhalten: manuelle Kopplung, Belohnungs-Kopplung und Minimierung des Vorhersagefehlers. Es wird gezeigt, wie diese Ansätze, welche als synaptische Plastizitätsregeln implementiert sind, verwendet werden können, um einfache Strategien und Bewegungen zu lernen. Diese Arbeit ebnet den Weg zur Integration von hirninspirierten Berechnungsparadigmen in das Gebiet der Robotik. Es wird sogar prognostiziert, dass Fortschritte in den neuromorphen Technologien und bei den Plastizitätsregeln die Entwicklung von Hochleistungs-Lernrobotern mit geringem Energieverbrauch ermöglicht

Simulation Intelligence: Towards a New Generation of Scientific Methods

The original "Seven Motifs" set forth a roadmap of essential methods for the field of scientific computing, where a motif is an algorithmic method that captures a pattern of computation and data movement. We present the "Nine Motifs of Simulation Intelligence", a roadmap for the development and integration of the essential algorithms necessary for a merger of scientific computing, scientific simulation, and artificial intelligence. We call this merger simulation intelligence (SI), for short. We argue the motifs of simulation intelligence are interconnected and interdependent, much like the components within the layers of an operating system. Using this metaphor, we explore the nature of each layer of the simulation intelligence operating system stack (SI-stack) and the motifs therein: (1) Multi-physics and multi-scale modeling; (2) Surrogate modeling and emulation; (3) Simulation-based inference; (4) Causal modeling and inference; (5) Agent-based modeling; (6) Probabilistic programming; (7) Differentiable programming; (8) Open-ended optimization; (9) Machine programming. We believe coordinated efforts between motifs offers immense opportunity to accelerate scientific discovery, from solving inverse problems in synthetic biology and climate science, to directing nuclear energy experiments and predicting emergent behavior in socioeconomic settings. We elaborate on each layer of the SI-stack, detailing the state-of-art methods, presenting examples to highlight challenges and opportunities, and advocating for specific ways to advance the motifs and the synergies from their combinations. Advancing and integrating these technologies can enable a robust and efficient hypothesis-simulation-analysis type of scientific method, which we introduce with several use-cases for human-machine teaming and automated science

25th Annual Computational Neuroscience Meeting: CNS-2016

Abstracts of the 25th Annual Computational Neuroscience Meeting: CNS-2016 Seogwipo City, Jeju-do, South Korea. 2–7 July 201