Synaptic weight distributions.

<p>Comparing the distributions of the synaptic weights at critical capacity for three different values of robustness obtained from simulation. The distribution of weights approaches a Dirac-delta distribution at zero plus a truncated Gaussian. As the patterns become more robust, the center of the partial Gaussian shifts towards the left, and the number of silent synapses increases.</p

Capacity as a function of the robustness parameter <i>ϵ</i> at sparseness <i>f</i> = 0.2.

<p>The theoretical calculations is compared with the simulations for <i>f</i> = 0.2. Note that the capacity in the sparse regime is higher than in the dense regime.</p

Critical capacity as a function of the basin size and the robustness parameter.

<p><b>A.</b> The red plot shows the critical capacity as a function of the size of the basins of attraction (<i>N</i> = 1001 neurons in the dense regime <i>f</i> = 0.5) when the strength of the external field is large (<i>γ</i> = 6) such that the ON and OFF neuronal populations are well separated. The points indicate 0.5 probability of successful storage at a given basin size, optimized over the robustness parameter <i>ϵ</i>. The error bars show the [0.95,0.05] probability interval for successful storage. The blue plot shows the performance of the Hopfield model with <i>N</i> = 1001 neurons. The maximal capacity at zero basin size (the Gardner bound) is equal to 2. <b>B.</b> To compare the result of simulation of our model with the analytical results, we plotted the critical capacity as a function of the robustness parameter <i>ϵ</i>. The dark red curve is the critical capacity versus <i>ϵ</i> for our model obtained form analytical calculations (see <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004439#sec011" target="_blank">Materials and Methods</a>), the cyan line shows the result of simulations of our model, and the dark blue shows the Gardner bound for a network with no constraints on synaptic weights. The difference between the two theoretical curves is due to the constraints on the weights in our network.</p

The three-threshold learning rule (3TLR), and its relationship with the standard perceptron learning rule (PLR).

<p>The perceptron learning rule modifies the synaptic weights by comparing the desired output with the actual output to obtain an error signal, subsequently changing the weights in the opposite direction of the error signal (see the table in the left panel). For a pattern which is uncorrelated with the current synaptic weights, the distribution is Gaussian (in the limit of large <i>N</i>), due to the central limit theorem. <i>H</i>₀ is set such that, on average, a fraction <i>f</i> of the local fields are above the neuronal threshold <i>θ</i>; in the case of <i>f</i> = 0.5, this means that the Gaussian is centered on <i>θ</i> (left panel). In our model (<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004439#pcbi.1004439.g001" target="_blank">Fig 1B</a>), the desired output is given as a strong external input, whose distribution across the population is bimodal (with two delta functions on <i>x</i>_<i>i</i> = 0 and <i>x</i>_<i>i</i> = <i>X</i>); therefore, the distribution of the local fields during stimulus presentation becomes bimodal as well (right panel). The left and right bumps of this distribution correspond to cases where the desired outputs are zero and one, respectively. Note that, since the external input also elicits an inhibitory response, the neurons in the network which are not directly affected by the external input (i.e. those with desired output equal to zero) are effectively hyperpolarized. If <i>X</i> is sufficiently large, the two distributions do not overlap, and the four cases of the PLR can be mapped to the four regions determined from the three thresholds, indicated by vertical dashed lines (see text).</p

Distribution of local fields before and after learning for <i>f</i> = 0.5 and non-zero robustness.

<p><b>A.</b> Before learning begins, the distribution of local field of neurons is a Gaussian distribution (due to central limit theorem) centered around neuronal threshold <i>θ</i> both for neurons with the desired output zero (OFF neurons) and with the desired output one (ON neurons). The goal is to have the local field distribution of ON neurons (red curve) to be above the threshold <i>θ</i>, and that of OFF neurons to be below <i>θ</i>. <b>B.</b> Once any of the to-be-stored patterns are presented as strong external fields, right before the learning process starts, the local field distribution of the OFF neuron shifts toward the left-side centered around </p><p></p><p></p><p></p><p><mi>θ</mi><mn>0</mn></p><mo>+</mo><mi>f</mi><mi>ϵ</mi><p><mi>N</mi></p><p></p><p></p><p></p>, whereas the distribution of the ON neurons moves toward the right-side, centered around <p></p><p></p><p></p><p><mi>θ</mi><mn>1</mn></p><mo>−</mo><mi>f</mi><mi>ϵ</mi><p><mi>N</mi></p><p></p><p></p><p></p>, with a negligible overlap between the two curves if the external field is strong enough. Thanks to the strong external fields and global inhibition, the local fields of the ON and OFF neurons are well separated. <b>C.</b> Due to the learning process, the local fields within the depression region [i.e. (<i>θ</i>₀, <i>θ</i>)] get pushed to the left-side, below <i>θ</i>₀, whereas those within the potentiation region get pushed further to the right-side, above <i>θ</i>₁. If the learning process is successful, it will result in a region (<i>θ</i>₀, <i>θ</i>₁) which no longer contain local fields, with two sharp peaks on <i>θ</i>₀ and <i>θ</i>₁. <b>D.</b> After successful learning, once the external fields are removed, the blue and red curves come closer, with a gap equal to <p></p><p></p><p><mn>2</mn><mi>f</mi><mi>ϵ</mi></p><p><mi>N</mi></p><p></p><p></p><p></p>. The larger the robustness parameter <i>ϵ</i>, the more the gap between the left- and right-side of the distribution. Notice that now the red curve is fully above <i>θ</i> which means those neurons remain stably ON, while the the blue curve is fully below <i>θ</i>, which means those neurons are stably OFF. Therefore the corresponding pattern is successfully stored by the network.<p></p

Direct comparions of the 3TLR and the PLR.

<p>Success probability for the 3TLR at <i>γ</i> = 6 (blue curve, left axis) and the PLR (red curve) at f=0.5 and ϵ=3; the results for the 3TLR at <i>γ</i> = 12 are identical to those of the PLR (red curve). The orange points show the absolute difference of weights between the final values of the weights for the PLR at <i>γ</i> = 6 and the PLR (right axis): the points show the median of the distribution, while the error bars span the 5th-95th percentiles, showing that, while the distribution is concentrated at near-zero values, outliers appear at the critical capacity of the 3TLR algorithm. (Note that the average value of the weights is in all cases approximately 1.08; also compare the discrepancies with the overall distribution of the weights, <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004439#pcbi.1004439.g008" target="_blank">Fig 8</a>).</p

Capacity as a function of the robustness parameter <i>ϵ</i> at sparseness <i>f</i> = 0.2.

<p>The theoretical calculations is compared with the simulations for <i>f</i> = 0.2. Note that the capacity in the sparse regime is higher than in the dense regime.</p

Dependence of the critical capacity on the strength of the external input.

<p>We varied the strength of the external field (<i>γ</i>) in order to quantify its effect on the learning process. The critical capacity is plotted as a function of <i>γ</i> at a fixed robustness <i>ϵ</i> = 0.3 in the dense regime <i>f</i> = 0.5. The simulations show that there is a very sharp drop in the maximum <i>α</i> when <i>γ</i> goes below ≈ 2.4.</p

A sketch of the network and the neuron model.

<p><b>A.</b> Structure of the network. The fully-connected network consists of <i>N</i> binary (<i>s</i>_<i>i</i> ∈ {0,1}) neurons and an aggregated inhibitory unit. The global inhibition is a function of the state of the network and the external fields, i.e. </p><p></p><p><mi>I</mi><mo stretchy="false">(</mo></p><p><mi>x</mi><mo>→</mo></p><mo>,</mo><p><mi>s</mi><mo>→</mo></p><mo stretchy="false">)</mo><p></p><p></p>. A memory pattern <p></p><p></p><p><mi>ξ</mi><mo>→</mo></p><p></p><p></p> is encoded as strong external fields, i.e. <p></p><p></p><p><mi>x</mi><mo>→</mo></p><mo>=</mo><mi>X</mi><p><mi>ξ</mi><mo>→</mo></p><p></p><p></p> and presented to the network during the learning phase. <b>B.</b> Each neuron receives excitatory recurrent inputs (thin black arrows) from the other neurons, a global inhibitory input (red connections), and a strong binary external field (<i>x</i>_<i>i</i> ∈ {0, <i>X</i>}; thick black arrows). All these inputs are summed to obtain the total field, which is then compared to a neuronal threshold <i>θ</i>; the output of the neuron is a step function of the result.<p></p

Capacity as a function of correlations in the input patterns, for <i>f</i> = 0.2 at <i>ϵ</i> = 3.0.

<p>Patterns are organized in categories, with a correlation <i>c</i> with the prototype of the corresponding category (see text).</p