Performance of estimator <i>C</i>_{sparse+latent} expressed as validation loss (eq. 10) relative to the other estimators: <i>C</i>_sample, <i>C</i>_diag, <i>C</i>_factor, and <i>C</i>_sparse.

<p>Covariance estimators <i>C</i>_sample, <i>C</i>_diag, <i>C</i>_factor, and <i>C</i>_sparse produced consistently greater validation losses than <i>C</i>_{sparse+latent} (<i>p</i> < 0.01 in each comparison, Wilcoxon signed rank test, <i>n</i> = 27 sites in 14 mice). The box plots indicate the 25^<i>th</i>, 50^<i>th</i>, and 75^<i>th</i> percentiles with the whiskers extending to the minimum and maximum values after excluding the outliers marked with ‘+’.</p

Performance of covariance estimators on samples drawn from Ising models.

<p><b>A–D</b> Validation losses of covariance matrix estimators relative to the estimator whose structure matches the ground truth. The calculation is performed identically to <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004083#pcbi.1004083.g001" target="_blank">Fig. 1</a> Row 6 except Ising models are used as ground truth.</p

Acquisition of neural signals for the estimation of noise correlations.

<p>Visual stimuli comprising full-field drifting gratings interleaved with blank screens (<b>A</b>) presented during two-photon recordings of somatic calcium signals using fast 3D random-access microscopy (<b>B</b>). <b>C–F</b>. Calcium activity data from an example site. <b>C</b>. Representative calcium signals of seven cells, downsampled to 20 Hz, out of the 292 total recorded cells. Spiking activity inferred by nonnegative deconvolution is shown by red ticks below the trace. <b>D</b>. The spatial arrangement and orientation tuning of the 292 cells from the imaged site. The cells’ colors indicate their orientation preferences. The gray cells were not significantly tuned. <b>E</b>. The sample noise correlation matrix of the activity of the neural population. <b>F</b>. Histogram of noise correlation coefficients in one site. The red line indicates the mean correlation coefficient of 0.020.</p

Properties of <i>C</i>_{sparse+latent} estimates from all imaged sites.

<div><p>Each point represents an imaged site with its color indicating the population size as shown in panels A and B. The example site from Figs. <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004083#pcbi.1004083.g003" target="_blank">3</a> and <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004083#pcbi.1004083.g005" target="_blank">5</a> is circled in blue.</p> <p><b>A.</b> The number of inferred latent units <i>vs</i>. population size. <b>B.</b> The connectivity of the sparse component of partial correlations as a function of population size. <b>C.</b> The average sample correlations <i>vs</i>. the average partial correlations (<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004083#pcbi.1004083.e007" target="_blank">Eq. 4</a>) of the <i>C</i>_{sparse+latent} estimate. <b>D.</b> The percentage of negative interactions vs. connectivity in the <i>C</i>_{sparse+latent} estimates.</p></div

Regularized estimators whose structure matches the true structure in the data are more efficient.

<div><p><b>Row 1.</b> Graphical models of the target estimates of the four respective regularized covariance matrix estimators. Recorded neurons are represented by the green spheres and latent units by the lightly shaded spheres. Edges represent conditional dependencies, <i>i.e.</i> ‘interactions’. <b>Row 1, A</b>. For estimator <i>C</i>_diag, the target estimate is a diagonal matrix, which describes systems that lack linear dependencies. <b>Row 1, B.</b> For estimator <i>C</i>_factor, the target estimate is a factor model (low-rank matrix plus a diagonal matrix), representing systems in which correlations arise due to common input from latent units. <b>Row 1, C</b>. For estimator <i>C</i>_sparse, the covariance matrix is approximated as the inverse of a sparse matrix. This approximation describes systems in which correlations arise from a sparse set of linear associations between the observed units. <b>Row 1, D</b>. For estimator <i>C</i>_{sparse+latent}, the covariance matrix is approximated as the inverse of the sum of a sparse matrix and a low-rank matrix. This approximation describes a model wherein correlations arise due to sparse associations between the recorded cells <i>and</i> due to several latent units.</p> <p><b>Row 2:</b> Examples of 50 × 50 correlation matrices corresponding to each structure: <b>A.</b> the diagonal correlation matrix, <b>B.</b> a factor model with four latent units, <b>C.</b> a correlation matrix with 67% off-diagonal zeros in its inverse, and <b>D.</b> a correlation matrix whose inverse is the sum of a rank-3 matrix (<i>i.e.</i> three latent units) and a sparse matrix with 76% off-diagonal zeros.</p> <p><b>Row 3:</b> Sample correlation matrices calculated from samples of size <i>n</i> = 500 drawn from simulated random processes with respective correlation matrices shown in Row 2. The structure of the sample correlation matrix is difficult to discern by eye.</p> <p><b>Row 4:</b> Estimates computed from the same data as in Row 3 using structured estimators of the correct type, optimized by cross-validation. The regularized estimates are closer to the truth than the sample correlation matrices.</p> <p><b>Row 5:</b> True loss (<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004083#pcbi.1004083.e013" target="_blank">Eq. 9</a>) for the five estimators as a function of sample size. The error bars indicate the standard deviation of the mean. Estimators with structure that matches the true model converged to zero faster than the other estimators.</p> <p><b>Row 6:</b> Validation loss (<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004083#pcbi.1004083.e019" target="_blank">Eq. 10</a>) for the five estimators relative to the matching estimators for each type of ground truth. Error bars indicate the standard deviation of the mean. Differences in validation loss approximate differences in true loss.</p></div

Dependence of sample correlations, regularized partial correlations, and connectivity inferred by <i>C</i>_{sparse+latent} on the differences in preferred orientations, Δori, and physical distances: horizontal Δ<i>x</i> and depth Δ<i>z</i>.

<div><p>Five sites with highest connectivity (see <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004083#pcbi.1004083.g006" target="_blank">Fig. 6 B</a>) were selected for this analysis.</p> <p><b>A</b>–<b>C.</b> Mean sample correlations in relation to Δori, Δ<i>x</i> and Δ<i>z</i>, respectively. For Δ<i>x</i> averages, only horizontally aligned cell pairs with Δ<i>z</i> < 30 <i>μm</i> were considered. Similarly, for Δ<i>z</i> averages, only vertically aligned cell pairs with Δ<i>x</i> < 30 <i>μm</i> were considered.</p> <p><b>D</b>–<b>F.</b> Mean partial correlations regularized by the <i>C</i>_{sparse+latent} estimator binned the same way as the sample correlations above. The partial correlations exhibit stronger dependence on Δori, Δ<i>x</i>, and Δ<i>z</i> than sample correlations.</p> <p><b>G</b>–<b>I.</b> Positive connectivity (green) and negative connectivity (red) inferred by the <i>C</i>_{sparse+latent} estimator. Positive and negative connectivities refer to the fractions of the positive and negative partial correlations computed from the sparse component <i>S</i> of <i>C</i>_{sparse+latent}. Positive connectivity decreases with Δori, Δ<i>x</i>, and Δ<i>z</i>. Negative connectivity does not decrease with Δori, Δ<i>x</i> within the examined range, and with Δ<i>z</i> for small values of Δ<i>z</i> < 60 <i>μm</i>.</p></div

Top algorithms make highly correlated predictions.

<p><b>A.-B.</b> Example cells from the test set for dataset 1 (OGB-1) and dataset 3 (GCaMP6s) show highly similar predictions between most algorithms. <b>C.</b> Average correlation coefficients between predictions of different algorithms across all cells in the test set at 25 Hz (40 ms bins).</p

Overview over datasets with training and test data used in the competition.

<p>Overview over datasets with training and test data used in the competition.</p

Summary of algorithm performance.

<p>Δ correlation is computed as the mean difference in correlation coefficient compared to the STM algorithm. Δ var. exp. in % is computed as the mean relative improvement variance explained (<i>r</i>²). Note that since variance explained is a nonlinear function of correlation, algorithms can be ranked differently according to the two measures. All means are taken over <i>N</i> = 32 recordings in the test set, except for training correlation, which is computed over <i>N</i> = 60 recordings in the training set.</p

Different spike inference metrics reach similar conclusions.

<p><b>A.</b> Area under the curve (AUC) of the inferred spike rate used as a binary predictor for the presence of spikes (evaluated at 25 Hz, 50 ms bins) on the test set. Colors indicate different datasets. Black dots are mean correlation coefficients across all <i>N</i> = 32 cells in the test set. Colored dots are jittered for better visibility. STM: Spike-triggered mixture model [<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1006157#pcbi.1006157.ref015" target="_blank">15</a>]; f-oopsi: fast non-negative deconvolution [<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1006157#pcbi.1006157.ref009" target="_blank">9</a>] <b>B.</b> Information gain of the inferred spike rate about the true spike rate on the test set (evaluated at 25 Hz, 40 ms bins).</p