Overlapping between shape-based categories and D-MST neuronal-based clusters.

<p>The table reports the overlap (fifth column) between each shape-based category (first column) and the D-MST neuronal-based cluster (second column) containing the best matching sub-tree of contiguous objects. Same table structure and symbols as in <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi-1003167-t001" target="_blank">Table 1</a>.</p

Similarity matrix, hierarchical clustering and PCA of IT population responses to visual objects.

<p>(A) Each pixel in the matrix color-codes the correlation (i.e., similarity) between the neuronal population vectors representing a pair of visual objects. The order of the objects along the axes is defined by the dendrogram produced by hierarchical clustering of the population vectors (to avoid crowding, one every three objects is shown; the complete object set is shown in <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi-1003167-g002" target="_blank">Fig. 2</a>). The first two branches of the dendrogram (shown at the top) are colored in cyan and magenta. (B) The fraction of animate and inanimate objects is not significantly different in the first two branches of the dendrogram (NS, <i>p</i>>0.1, <i>χ</i>² test). (C) The proportion of large and small objects is significantly different in the first two branches of the dendrogram (**, <i>p</i><0.001, <i>χ</i>² test), (D) Layout of visual objects in the two-dimensional space defined by the first two principal components of the IT population responses (to avoid crowding, only some of the objects are shown). (E) Object area and object ranking along the first principal component are linearly related (<i>r</i> = −0.69, <i>p</i><0.001, <i>t</i>-test).</p

Overlapping between low-level categories and D-MST neuronal-based clusters.

<p>The table reports the overlap (fifth column) between each low-level category (first column) and the D-MST neuronal-based cluster (second column) containing the best matching sub-tree of contiguous objects. Same table structure and symbols as in <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi-1003167-t001" target="_blank">Table 1</a>.</p

Overlapping between semantic categories and D-MST neuronal-based clusters.

<p>The table reports the overlap (fifth column) between each semantic category (first column) and the D-MST neuronal-based cluster (second column) containing the best matching sub-tree of contiguous objects, according to a score defined as the ratio between the intersection of the sub-tree with the category and their union (fifth column). Significance of the overlap was computed by permuting (1,000,000 times) either sets of twin objects (forth- and third-to-last columns) or individual objects (second-to-last and last columns) across the categories of a given clustering hypotheses: Holm-Bonferroni corrected <i>p</i><0.01 (**) and <i>p</i><0.05 (* and *+); and uncorrected <i>p</i><0.01 (++ and *+) and <i>p</i><0.05 (+). For comparison with <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi.1003167-Kiani1" target="_blank">[14]</a>, two other overlap metrics (Ratio 1 = the fraction of objects in the category overlapping with the cluster; and Ratio 2 = the fraction of objects in the cluster overlapping with the category) are also reported.</p

Overlap between <i>k</i>-means clusters in the IT neuronal space and object categories of the clustering hypotheses.

<p>(A) Fifteen object clusters obtained by a typical run of the <i>k</i>-means algorithm over the IT neuronal representation space. The clusters' arrangement was determined by applying a hierarchical clustering algorithm to their centroids (see the dendrogram on the top; the same approach was used to arrange the shape-based categories shown in C, which resulted from the <i>k</i>-means object clustering in the output layer of an object recognition model <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi.1003167-Mutch1" target="_blank">[44]</a>). (B–D) The semantic (B), shape-based (C) and low-level (D) categories that significantly overlapped with some of the neuronal-based clusters shown in A. Overlapping neuronal-based clusters and categories are indicated by matching names (e.g., <i>faces</i>) in A and B–D, with the objects in common between a cluster and a category enclosed by either a yellow (semantic), a red (shape-based) or a cyan (low-level) frame. (E) Average number of significant overlaps between neuronal-based clusters and semantic (first bar), shape-based (second bar) and low-level (third bar) categories across 1,000 runs of the <i>k</i>-means algorithm over both the neuronal representation space and the model representation space. The yellow, red and cyan striped portion of the first bar indicates the number of neuronal-based clusters that significantly overlapped with both a semantic category and either a shape-based or a low-level category.</p

Low-level categories significantly represented in IT according to the D-MST and the FLD analyses.

<p>The second and third columns report what low-level categories were found to be significantly represented in IT according, respectively, to the D-MST analysis (when significance was computed by permuting twins' sets; i.e., same data as in <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi-1003167-g005" target="_blank">Fig. 5</a> and in the third-to-last column of <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi-1003167-t003" target="_blank">Table 3</a>) and to the FLD analysis (when classifiers were applied to the pruned object categories; i.e., same data as in <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi-1003167-g006" target="_blank">Fig. 6C</a>). Same significance level symbols as in <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi-1003167-t001" target="_blank">Table 1</a>. The last column shows what low-level categories were found to be significantly represented in IT according to both the D-MST and the FLD analyses.</p

Fisher Linear Discriminant (FLD) analysis of IT population activity.

<p>(A) Each gray bar reports the average performance of a binary FLD at correctly classifying members of a given object category (e.g., faces) from all other objects in the set. For each binary classification task, the standard deviation of the performance (error bars), and the mean and standard deviation of the null distribution (gray circles and their error bars), against which significant deviation of performance from chance was assessed (same significance level symbols as in <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi-1003167-t001" target="_blank">Table 1</a>), are also reported (see <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#s4" target="_blank">Materials and Methods</a> for a description of the cross-validation and permutation procedures yielding these summary statistics). (B) Examples of “pruned” semantic, shape-based and low-level categories that were obtained by subsampling the original object categories (shown in <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi.1003167.s001" target="_blank">Fig. S1</a>), so as to minimize the overlap between semantic and visual information (see <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#s4" target="_blank">Materials and Methods</a> for details). (C) Performance of the FLDs at correctly classifying members of the pruned categories (same symbols as in A).</p

Recording locations.

<p>The blue dots show the projections of the recording chamber grid-point locations from the top of the skull to the ventral bank of the superior temporal sulcus (STS) and the ventral surface lateral to the anterior middle temporal sulcus (AMTS). The projections are shown over a sequence of MRI images (spanning a 13–17 anteroposterior range; Horsley-Clarke coordinates) that were collected, for one of the monkeys, before the chamber implant surgery. Only the grid locations in which the electrode was inserted at least once are shown. The red-shaded areas highlight the estimated cortical span that was likely sampled during recording, given that: 1) each electrode penetration usually spanned the whole depth of the targeted cortical bank (either STS or AMTS); and 2) the upper bound of the variability of each recording location along the mediolateral axis (due to bending of the electrode during insertion) can be estimated as ±2 mm <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi.1003167-Cox1" target="_blank">[80]</a>. The figure also shows the range of possible locations of the three anterior face patches (AL, AF and AM) according to <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi.1003167-Tsao3" target="_blank">[33]</a>, so as to highlight their potential overlap with the recording locations.</p

The stimulus set.

<p>The full set of 213 objects used in our study. The set consists of: i) 188 images of real-world objects belonging to 94 different categories (e.g., two hats, two accordions, two monkey faces, etc.); ii) 5 cars, 5 human faces, and 5 abstract silhouettes; iii) 5 patches of texture (e.g., random dots and oriented bars); iv) a blank frame; v) 4 low contrast (10%, 3%, 2% and 1.5%) images of one of the objects (a camera).</p

Shape-based categories significantly represented in IT according to the D-MST and the FLD analyses.

<p>The second and third columns report what shape-based categories were found to be significantly represented in IT according, respectively, to the D-MST analysis (when significance was computed by permuting twins' sets; i.e., same data as in <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi-1003167-g005" target="_blank">Fig. 5</a> and in the third-to-last column of <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi-1003167-t002" target="_blank">Table 2</a>) and to the FLD analysis (when classifiers were applied to the pruned object categories; i.e., same data as in <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi-1003167-g006" target="_blank">Fig. 6C</a>). Same significance level symbols as in <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003167#pcbi-1003167-t001" target="_blank">Table 1</a>. The last column shows what shape-based categories were found to be significantly represented in IT according to both the D-MST and the FLD analyses.</p