A Spiking Neural Network Model of Model-Free Reinforcement Learning with High-Dimensional Sensory Input and Perceptual Ambiguity

<div><p>A theoretical framework of reinforcement learning plays an important role in understanding action selection in animals. Spiking neural networks provide a theoretically grounded means to test computational hypotheses on neurally plausible algorithms of reinforcement learning through numerical simulation. However, most of these models cannot handle observations which are noisy, or occurred in the past, even though these are inevitable and constraining features of learning in real environments. This class of problem is formally known as partially observable reinforcement learning (PORL) problems. It provides a generalization of reinforcement learning to partially observable domains. In addition, observations in the real world tend to be rich and high-dimensional. In this work, we use a spiking neural network model to approximate the free energy of a restricted Boltzmann machine and apply it to the solution of PORL problems with high-dimensional observations. Our spiking network model solves maze tasks with perceptually ambiguous high-dimensional observations without knowledge of the true environment. An extended model with working memory also solves history-dependent tasks. The way spiking neural networks handle PORL problems may provide a glimpse into the underlying laws of neural information processing which can only be discovered through such a top-down approach.</p></div

The structures of the spiking neural networks.

<p>State neurons are used for the MDP task. Observation neurons are used for the PORL tasks instead of state neurons. Memory architecture (bounded by dashed line in the figure) is introduced only for the history-dependent PORL task.</p

The number of neurons in each task.

<p>The number of neurons in each task.</p

Digit center reaching task.

<p>(A) A set of digits used in the training. (B) The cumulative reward obtained with test dataset. (C, D) The activation of hidden neurons projected on the first two principal components in different reward settings. (C) The reward setting is the same as in the simple task. (D) The agent always gets reward of 2000 for any states and actions. Each point shows the hidden activation for each state using test digit dataset.</p

Comparison between the SNN and the original RBM.

<p>Red colored symbols and lines indicate rightward actions, blue colored symbols and lines indicate leftward actions. (A) Spike counts of action neurons in the SNN (left) and the negative free-energy in the original RBM (right) for each state. Differences of spike counts (SNN) and negative free-energies (RBM) for action selection (middle). (B) Negative iFE of the SNN and negative free-energy of the equivalent RBM for certain state-action pairs. (C) Correlations between hidden neuron spike counts and the posterior over hidden nodes for each action (left) and when the weights are scaled (right, solid lines) and randomized (right, dotted lines). (D) Spike counts of hidden neurons in the SNN (top panel) and the posterior of the original RBM (middle and bottom panels).</p

MOESM1 of Empirical Bayesian significance measure of neuronal spike response

Additional file 1. Supplementary document

Multiple co-clustering based on nonparametric mixture models with heterogeneous marginal distributions

<div><p>We propose a novel method for multiple clustering, which is useful for analysis of high-dimensional data containing heterogeneous types of features. Our method is based on nonparametric Bayesian mixture models in which features are automatically partitioned (into views) for each clustering solution. This feature partition works as feature selection for a particular clustering solution, which screens out irrelevant features. To make our method applicable to high-dimensional data, a co-clustering structure is newly introduced for each view. Further, the outstanding novelty of our method is that we simultaneously model different distribution families, such as Gaussian, Poisson, and multinomial distributions in each cluster block, which widens areas of application to real data. We apply the proposed method to synthetic and real data, and show that our method outperforms other multiple clustering methods both in recovering true cluster structures and in computation time. Finally, we apply our method to a depression dataset with no true cluster structure available, from which useful inferences are drawn about possible clustering structures of the data.</p></div

Prediction of clinical depression scores and detection of changes in whole-brain using resting-state functional MRI data with partial least squares regression

<div><p>In diagnostic applications of statistical machine learning methods to brain imaging data, common problems include data high-dimensionality and co-linearity, which often cause over-fitting and instability. To overcome these problems, we applied partial least squares (PLS) regression to resting-state functional magnetic resonance imaging (rs-fMRI) data, creating a low-dimensional representation that relates symptoms to brain activity and that predicts clinical measures. Our experimental results, based upon data from clinically depressed patients and healthy controls, demonstrated that PLS and its kernel variants provided significantly better prediction of clinical measures than ordinary linear regression. Subsequent classification using predicted clinical scores distinguished depressed patients from healthy controls with 80% accuracy. Moreover, loading vectors for latent variables enabled us to identify brain regions relevant to depression, including the default mode network, <i>the right superior frontal gyrus</i>, and <i>the superior motor area</i>.</p></div

Samples from image datasets for person ‘an2i’ and ‘at33’.

<p>Pixels surrounded by color boxes are selected features that yielded relevant sample clustering to useid in data2. Image configurations are (‘an2i’, non sunglass, straight), (‘at33’, non sunglass, straight), ‘an2i’, sunglass, left), and (‘at33’, sunglass, left), respectively. Expression is neutral for all samples. In these examples, the multiple clustering method correctly identified these persons.</p

Results for data 2 of the facial image data.

<p>Contingency table of the true labels (useid) and yielded clustering of the multiple co-clustering (Mul), COALA, decorrelated <i>K</i>-means (DecK), and restricted multiple (rMul) method from (a) to (d). T1 and T2 are true classifications (an2i, at33); C1, C2, C3 and C4 are yielded clusters.</p