Deep Neural Networks as a Computational Model for Human Shape Sensitivity - Fig 4

<p>(a) Examples of geons [<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004896#pcbi.1004896.ref024" target="_blank">24</a>]. In order to measure model’s sensitivity to changes in non-accidental properties, model’s output is computed for a particular stimulus (middle column) and compared to the output when another variant of the same kind of stimulus is presented (right column) and when a non-accidental change in the stimulus is introduced (left column) that is physically (in the metric space) just as far from the base as the metric variant. We used 22 such triplets in total. (b) Model performance on discriminating between stimuli. For each triplet, model’s output is counted as accurate if the non-accidental variant is more dissimilar from the base stimulus than the metric variant is from the base. Chance level (50%) is indicated by a dashed line. (c) HMAX and convnet performance on the task at different layers. Non-accidental stimuli appear to be closer to the base in the early layers, which is consistent with a conservative design of the stimuli [<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004896#pcbi.1004896.ref024" target="_blank">24</a>]. Best performance is observed at the upper layers of convnets with a slight dip at the output layer. Vertical dotted lines indicate where fully-connected layers start. In both plots, error bars (or bands) depict 95% bootstrapped confidence intervals.</p

Model preference for shape.

<p>(a) Stimulus set with physical and perceived shape dimensions manipulated orthogonally [<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004896#pcbi.1004896.ref005" target="_blank">5</a>]. (b) Multidimensional scaling plots for the dissimilarity matrices of physical form, perceived shape, and the GoogLeNet outputs (at the top layer). The separation between shapes based on their perceived rather than physical similarity is evident in the GoogLeNet outputs (for visualization purposes, indicated by the lines separating the three clusters). (c) A correlation between model outputs and the physical form similarity of stimuli. Most shallow models are capturing physical similarity reasonably well, whereas HMAX and deep models are largely less representative of the physical similarity. (d) A correlation between model outputs and the perceived shape similarity of stimuli. Here, in contrast, deep models show a tendency of capturing perceived shape better than shallow and HMAX models. Gray band indicates estimated ceiling correlation based on human performance. (e) Correlation with physical (green) and perceived (orange) shape similarity across the layers of HMAX models and convnets. A preference for the perceived shape emerges in the upper layers. Vertical dotted lines indicate where fully-connected layers start. In all plots, error bars (or bands) indicate the 95% bootstrapped confidence intervals.</p

Categorical representations in convnets.

<p>(a) Stimulus set from [<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004896#pcbi.1004896.ref037" target="_blank">37</a>] with shape and category information largely orthogonal. (Adapted with permission from [<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004896#pcbi.1004896.ref037" target="_blank">37</a>].) (b) Multidimensional scaling plot representing object dissimilarity in the output layer of GoogLeNet. The black line indicates a clear separation between natural (brown, orange, yellow) and man-made (blue, gray, pink) objects. (c-d) A correlation between model representations and human shape (c) and category (d) judgments. Gray band indicates estimated ceiling correlation based on human performance. (e) A correlation with human shape (orange) and category (blue) judgments across the layers of HMAX models and convnets. Vertical dotted lines indicate where fully-connected layers start. In all plots error bars (or bands) represent 95% bootstrapped confidence intervals.</p

Object categorization.

<p>(a) Examples of stimuli from the modified Snodgrass and Vanderwart stimulus set [<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004896#pcbi.1004896.ref021" target="_blank">21</a>]. Stimulus images courtesy of Michael J. Tarr, Center for the Neural Basis of Cognition and Department of Psychology, Carnegie Mellon University, <a href="http://www.tarrlab.org/" target="_blank">http://www.tarrlab.org/</a>. (b) Human (<i>n</i> = 10 for each variant of the stimulus set) and convnet (CaffeNet, VGG-19, GoogLeNet) accuracy in naming objects. For each stimulus set variant, mean human performance is indicated by a gray horizontal line, with the gray surrounding band depicting 95% bootstrapped confidence intervals. Error bars on model performance also depict 95% bootstrapped confidence intervals. (c) A consistency between human and convnet naming of objects. A consistency of .5 means that about half of responses (whether correct or not) we consistent between a model and an average of humans. Error bars indicate 95% bootstrapped confidence intervals. Gray bands indicate estimated ceiling performance based on between-human consistency. (d) Correlation with human performance on the silhouette stimulus set. The x-axis depicts an average human accuracy for a particular silhouette [<a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1004896#pcbi.1004896.ref015" target="_blank">15</a>] and the y-axis depicts GoogLeNet performance on the same silhouette (either correct (value 1.0) or incorrect (value 0.0)). Model’s performance is jittered on the x- and y-axis for better visibility. Dark gray bubbles indicate average model’s performance for 11 bins of human performance (i.e., 0–5%, 5–15%, 15–25%, etc.) with the size of each bubble reflecting the number of data points per bin. The orange line shows the logistic regression fit with a 95% bootstrapped confidence interval (light orange shaded). The slope of the logistic regression is reliably different from zero.</p

Illustration of the experimental conditions and the anatomical position of the regions of interest (ROIs).

<p>(A), Illustration of the position of the four stimulus categories (faces, objects, scenes, and scrambled images) left and right of the fixation spot. (B) Illustration of the 5 ROIs for one subject onto a flattened brain. Indicated sulci: CS: calcarine sulcus; ITS: inferior temporal sulcus. Indicated anatomical directions: D: dorsal; V: ventral; P: posterior; A: anterior.</p

Responses to contralateral and ipsilateral stimuli in the regions of interest.

<p>(A) The response (percent signal change relative to the fixation condition) in each ROI is shown for each stimulus condition (F: faces, O: objects, Se: scenes, Sa: scrambled images). (B) Preference index in each ROI averaged across all stimulus conditions. (C) Preference index in each ROI for the stimulus condition that elicited the strongest responses. Error bars represent the standard error of the mean across subjects.</p

Visual Space and Object Space in the Cerebral Cortex of Retinal Disease Patients

<div><p>The lower areas of the hierarchically organized visual cortex are strongly retinotopically organized, with strong responses to specific retinotopic stimuli, and no response to other stimuli outside these preferred regions. Higher areas in the ventral occipitotemporal cortex show a weak eccentricity bias, and are mainly sensitive for object category (e.g., faces versus buildings). This study investigated how the mapping of eccentricity and category sensitivity using functional magnetic resonance imaging is affected by a retinal lesion in two very different low vision patients: a patient with a large central scotoma, affecting central input to the retina (juvenile macular degeneration), and a patient where input to the peripheral retina is lost (retinitis pigmentosa). From the retinal degeneration, we can predict specific losses of retinotopic activation. These predictions were confirmed when comparing stimulus activations with a no-stimulus fixation baseline. At the same time, however, seemingly contradictory patterns of activation, unexpected given the retinal degeneration, were observed when different stimulus conditions were directly compared. These unexpected activations were due to position-specific deactivations, indicating the importance of investigating absolute activation (relative to a no-stimulus baseline) rather than relative activation (comparing different stimulus conditions). Data from two controls, with simulated scotomas that matched the lesions in the two patients also showed that retinotopic mapping results could be explained by a combination of activations at the stimulated locations and deactivations at unstimulated locations. Category sensitivity was preserved in the two patients. In sum, when we take into account the full pattern of activations and deactivations elicited in retinotopic cortex and throughout the ventral object vision pathway in low vision patients, the pattern of (de)activation is consistent with the retinal loss.</p></div

Preference and activity patterns for different eccentricities in lower visual areas for the RP patient.

<p>(A) The medial view of the posterior part of right and left hemisphere is shown on an inflated cortical surface. The approximate location of the calcarine sulcus is marked with a dotted line. The color legend is shown above (orange-red for central stimuli, green for paracentral stimuli, blue-purple for peripheral stimuli) and reflects the relative preference to the different eccentricities. In black two regions are marked which are further characterized for illustration purposes. The data of one region (red arrow/box) are mostly dominated by a positive response, and for the other region (blue arrow/box) mostly by a negative response compared to a no-stimulus baseline (B) Activity patterns in both hemispheres compared to a fixation baseline, at p<0.05 uncorrected for one of three conditions: central (8 most central stimuli, contrasted against baseline), paracentral (8 paracentral stimuli, contrasted against baseline) and peripheral (8 most eccentric stimuli, contrasted against baseline). The selected ROIs now show the underlying positive and negative responses. (C) Time course averaged across runs and across stimulus sequences to represent the response in a selected ROI to different eccentricities. The red dotted lines represent the 95% confidence intervals (calculated using the variation across runs). (C, left panel) A positive response to the most central stimuli, with a strong drop in activation to a near zero response when more eccentric stimuli are presented (C, right panel) Strong deactivations for the central and paracentral stimuli, and a response close to zero for the peripheral stimuli (D) Average beta values in each selected ROI. (D, left panel) Positive responses in the central and paracentral conditions, and a near zero response in the peripheral condition. (D, right panel) Negative responses (beta values) for the (para)central conditions and a near zero response to the peripheral condition.</p

Preference and activity patterns for different eccentricities in lower visual areas for the JMD controls.

<p>(A) The medial view of the posterior part of right and left hemisphere is shown on an inflated cortical surface for the two controls (control 1: upper figure, control 2: lower figure). They were tested with the stimulus set simulating the JMD scotoma. The approximate location of the calcarine sulcus is marked with a dotted line. The color legend is shown above (orange-red for central stimuli, green for paracentral stimuli, blue-purple for peripheral stimuli) and reflects the relative preference to the different eccentricities. In black two regions are marked which are further characterized for illustration purposes. The data of one region (red arrow/box) are mostly dominated by a positive response, and for the other region (blue arrow/box) mostly by a negative response compared to a no-stimulus baseline. (B) Average beta values in each selected ROI. The red arrows and box indicate activity of a ROI that shows a positive response to the (visible) peripheral stimuli, while the blue arrows and box show a ROI where negative responses to unstimulated central parts of the visual field cause a phase preference in the absence of activation in the other conditions.</p

Preference and activity patterns for different eccentricities in the ventral cortex with central (simulated) scotoma.

<p>(Left) Relative preference in the eccentricity mapping paradigm for the JMD patient (A), control 1 (C) and control 2 (E), shown on an inflated hemisphere. The color legend is shown above (orange-red for central stimuli, green for paracentral stimuli, blue-purple for peripheral stimuli). The black lines mark the face-sensitive areas (FA), the red lines mark the house (place)-sensitive areas (PA) defined by the blocked localizer design. (Right) average beta values of three conditions, when the eccentricity data are analyzed as a block design and compared to a fixation baseline, in both the FA and PA region for the JMD patient (B), control 1 (D) and control 2 (F).</p