Analysis of Video Feature Learning in Two-Stream CNNs on the Example of Zebrafish Swim Bout Classification

Semmelhack et al. (2014) have achieved high classification accuracy in distinguishing swim bouts of zebrafish using a Support Vector Machine (SVM). Convolutional Neural Networks (CNNs) have reached superior performance in various image recognition tasks over SVMs, but these powerful networks remain a black box. Reaching better transparency helps to build trust in their classifications and makes learned features interpretable to experts. Using a recently developed technique called Deep Taylor Decomposition, we generated heatmaps to highlight input regions of high relevance for predictions. We find that our CNN makes predictions by analyzing the steadiness of the tail's trunk, which markedly differs from the manually extracted features used by Semmelhack et al. (2014). We further uncovered that the network paid attention to experimental artifacts. Removing these artifacts ensured the validity of predictions. After correction, our best CNN beats the SVM by 6.12%, achieving a classification accuracy of 96.32%. Our work thus demonstrates the utility of AI explainability for CNNs.Comment: 18 pages incl. references and appendix, 16 figures, ICLR 2020 Conferenc

Concurrent Analysis of Neural Activity at Multiple Scales Using Mixed Vine Copulas

concurrent measures of neural signals with different recordings modalities. However, statistical methods that take full advantage of the great differences in multi-modal statistics and their intricate dependencies are currently lacking. We developed a framework based on vine copulas with mixed margins to model multivariate data that are partly discrete such as neural spike counts and partly continuous such as local field potentials. The vine copula approach allowed us to derive efficient methods for likelihood calculation, inference and sampling with quadratic complexity in the number of modeled elements. We combined these methods by means of Monte-Carlo integration to obtain unbiased estimates of entropy and mutual information. To test our methods, we generated artificial data from parametric multivariate models. We also generated artificial spike counts and local field potentials from biologically realistic network models using the VERTEX simulator. We applied our methods to these data and show that our new approach provides a model fit that is significantly better than that of corresponding independent models. Moreover, we demonstrate that mutual information estimates of fully continuous and mixed independent models can strongly differ from our proposed model which is faithful to the statistics of the margins and their dependencies. Our framework presents the prospect of an improved analysis of neural data recorded simultaneously at different scales and from different modalities. Our models can also be used to construct Bayes-optimal decoders in brain-machine interfaces that benefit from concurrent recordings of various modalities

A Maximum Entropy Test for Evaluating Higher-Order Correlations in Spike Counts

Evaluating the importance of higher-order correlations of neural spike counts has been notoriously hard. A large number of samples are typically required in order to estimate higher-order correlations and resulting information theoretic quantities. In typical electrophysiology data sets with many experimental conditions, however, the number of samples in each condition is rather small. Here we describe a method that allows to quantify evidence for higher-order correlations in exactly these cases. We construct a family of reference distributions: maximum entropy distributions, which are constrained only by marginals and by linear correlations as quantified by the Pearson correlation coefficient. We devise a Monte Carlo goodness-of-fit test, which tests - for a given divergence measure of interest - whether the experimental data lead to the rejection of the null hypothesis that it was generated by one of the reference distributions. Applying our test to artificial data shows that the effects of higher-order correlations on these divergence measures can be detected even when the number of samples is small. Subsequently, we apply our method to spike count data which were recorded with multielectrode arrays from the primary visual cortex of anesthetized cat during an adaptation experiment. Using mutual information as a divergence measure we find that there are spike count bin sizes at which the maximum entropy hypothesis can be rejected for a substantial number of neuronal pairs. These results demonstrate that higher-order correlations can matter when estimating information theoretic quantities in V1. They also show that our test is able to detect their presence in typical in-vivo data sets, where the number of samples is too small to estimate higher-order correlations directly

Mixed vine copula flows for flexible modeling of neural dependencies

Recordings of complex neural population responses provide a unique opportunity for advancing our understanding of neural information processing at multiple scales and improving performance of brain computer interfaces. However, most existing analytical techniques fall short of capturing the complexity of interactions within the concerted population activity. Vine copula-based approaches have shown to be successful at addressing complex high-order dependencies within the population, disentangled from the single-neuron statistics. However, most applications have focused on parametric copulas which bear the risk of misspecifying dependence structures. In order to avoid this risk, we adopted a fully non-parametric approach for the single-neuron margins and copulas by using Neural Spline Flows (NSF). We validated the NSF framework on simulated data of continuous and discrete types with various forms of dependency structures and with different dimensionality. Overall, NSFs performed similarly to existing non-parametric estimators, while allowing for considerably faster and more flexible sampling which also enables faster Monte Carlo estimation of copula entropy. Moreover, our framework was able to capture low and higher order heavy tail dependencies in neuronal responses recorded in the mouse primary visual cortex during a visual learning task while the animal was navigating a virtual reality environment. These findings highlight an often ignored aspect of complexity in coordinated neuronal activity which can be important for understanding and deciphering collective neural dynamics for neurotechnological applications

State-dependent brainstem ensemble dynamics and their interactions with hippocampus across sleep states

The brainstem plays a crucial role in sleep-wake regulation. However, the ensemble dynamics underlying sleep regulation remain poorly understood. Here, we show slow, state-predictive brainstem ensemble dynamics and state-dependent interactions between the brainstem and the hippocampus in mice. On a timescale of seconds to minutes, brainstem populations can predict pupil dilation and vigilance states and exhibit longer prediction power than hippocampal CA1 neurons. On a timescale of sub-seconds, pontine waves (P-waves) are accompanied by synchronous firing of brainstem neurons during both rapid eye movement (REM) and non-REM (NREM) sleep. Crucially, P-waves functionally interact with CA1 activity in a state-dependent manner: during NREM sleep, hippocampal sharp wave-ripples (SWRs) precede P-waves. On the other hand, P-waves during REM sleep are phase-locked with ongoing theta oscillations and are followed by burst firing of CA1 neurons. This state-dependent global coordination between the brainstem and hippocampus implicates distinct functional roles of sleep

Mixed vine copulas as joint models of spike counts and local field potentials

Concurrent measurements of neural activity at multiple scales, sometimes performed with multimodal techniques, become increasingly important for studying brain function. However, statistical methods for their concurrent analysis are currently lacking. Here we introduce such techniques in a framework based on vine copulas with mixed margins to construct multivariate stochastic models. These models can describe detailed mixed interactions between discrete variables such as neural spike counts, and continuous variables such as local field potentials. We propose efficient methods for likelihood calculation, inference, sampling and mutual information estimation within this framework. We test our methods on simulated data and demonstrate applicability on mixed data generated by a biologically realistic neural network. Our methods hold the promise to considerably improve statistical analysis of neural data recorded simultaneously at different scales

CalciumGAN: A Generative Adversarial Network Model for Synthesising Realistic Calcium Imaging Data of Neuronal Populations

Calcium imaging has become a powerful and popular technique to monitor the activity of large populations of neurons in vivo. However, for ethical considerations and despite recent technical developments, recordings are still constrained to a limited number of trials and animals. This limits the amount of data available from individual experiments and hinders the development of analysis techniques and models for more realistic size of neuronal populations. The ability to artificially synthesize realistic neuronal calcium signals could greatly alleviate this problem by scaling up the number of trials. Here we propose a Generative Adversarial Network (GAN) model to generate realistic calcium signals as seen in neuronal somata with calcium imaging. To this end, we adapt the WaveGAN architecture and train it with the Wasserstein distance. We test the model on artificial data with known ground-truth and show that the distribution of the generated signals closely resembles the underlying data distribution. Then, we train the model on real calcium signals recorded from the primary visual cortex of behaving mice and confirm that the deconvolved spike trains match the statistics of the recorded data. Together, these results demonstrate that our model can successfully generate realistic calcium imaging data, thereby providing the means to augment existing datasets of neuronal activity for enhanced data exploration and modeling