The processing of biologically plausible and implausible forms in American Sign Language: evidence for perceptual tuning

<p>The human auditory system distinguishes speech-like information from general auditory signals in a remarkably fast and efficient way. Combining psychophysics and neurophysiology (MEG), we demonstrate a similar result for the processing of visual information used for language communication in users of sign languages. We demonstrate that the earliest visual cortical responses in deaf signers viewing American Sign Language signs show specific modulations to violations of anatomic constraints that would make the sign either possible or impossible to articulate. These neural data are accompanied with a significantly increased perceptual sensitivity to the anatomical incongruity. The differential effects in the early visual evoked potentials arguably reflect an expectation-driven assessment of somatic representational integrity, suggesting that language experience and/or auditory deprivation may shape the neuronal mechanisms underlying the analysis of complex human form. The data demonstrate that the perceptual tuning that underlies the discrimination of language and non-language information is not limited to spoken languages but extends to languages expressed in the visual modality.</p

Efficiency for identifying letters and words as a function of their complexity.

<p>Efficiency is nearly inversely proportional to complexity over a nearly hundred-fold range. The horizontal scale is the perimetric complexity (perimeter squared over ink area) of the letter or word. Each<b>+</b>is efficiency for identifying one of 26 words of a given length (1 to 17) in Courier <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Pelli1" target="_blank">[24]</a>. Each Courier letter has a complexity of 100 (averaging a-z), and the complexity of a word is proportional to its length. Each △ is efficiency for identifying one letter of one of 14 traditional fonts and alphabets by native or highly trained readers, in order of increasing complexity <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Pelli2" target="_blank">[25]</a>: <i>Braille, bold Helvetica, bold Bookman, Sloan, Helvetica, Hebrew, Devanagari, Courier, Armenian, Bookman, Arabic, uppercase Bookman, Chinese, Künstler</i>. The outlying □ is efficiency for a letter in an untraditional alphabet: <i>4×4 random checkerboards</i>, after extended training <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Pelli2" target="_blank">[25]</a>. The outlying ○ is efficiency for identifying the location of a disk. (See ‘Experiment 5. Identifying disks’ at the end of <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#s3" target="_blank">Materials and Methods</a>.) A disk has the lowest possible perimetric complexity <i>K</i> = 4<i>π</i> = 12.6. A linear regression of log efficiency vs. log complexity for the traditional letters (13 fonts and alphabets) and words (13 lengths), excluding the untraditional alphabet and disk, has a slope of −0.92 and <i>R</i>² = 0.99. The regression line and its equation are shown.</p

Assessing efficiency for combining the parts of a word: energy threshold as a function of word length.

<p>The summation index <i>k</i> is 1 minus the slope. Ideal thresholds, not shown, are independent of word length, with slope zero. (<b>A</b>) For a written word <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Pelli1" target="_blank">[24]</a>, the summation index is <i>k = </i>0.1. (<b>B</b>) For a spoken word <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Hirsh1" target="_blank">[48]</a>, <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Rubenstein1" target="_blank">[49]</a>, the summation index is <i>k</i> = 0.5 or 0.7. See <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#s3" target="_blank">Methods</a> for details.</p

Predictions and Results.

<p>(<b>A</b>) Audio-visual summation is summarized by the summation index <i>k</i> of a smooth curve (Eq. 8) fitted to the threshold energies. The horizontal and vertical scales represent the normalized visual and audio energy components <i>v</i> = <i>V</i>/<i>V</i>_uni and <i>a</i> = <i>A</i>/<i>A</i>_uni of the bimodal signal at threshold. Each audio:visual ratio – including the two unimodal conditions (<i>V</i>_uni, 0) and (0, <i>A</i>_uni) – is a condition. All conditions are randomly interleaved, trial by trial (with one exception, described at the end of this caption). The noise is always present in both streams. For a given audio:visual ratio <i>A</i>/<i>V</i>, we measure the threshold (<i>V</i>, <i>A</i>) radially, along a line from the origin (0, 0). The curves represent degrees of summation ranging from none (<i>k</i> = 0) to complete (<i>k</i> = 1). The special case of <i>k</i> = 0 is to be understood as the limit as <i>k</i> approaches 0, which is max(<i>v</i>,<i>a</i>) = 1. (<b>B</b>) Averaging <i>k</i> over our ten observers, we find the same summation for reporting either a single word (<i>k = </i>0.75, red, Experiment 1) or a sentence (<i>k</i> = 0.76, blue, Experiment 2). The error bars indicate mean ± standard error. The curves obtained for each individual observer are shown in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803.s002" target="_blank">Figure S2</a>. The virtue of randomly interleaving conditions (<i>a:v</i> ratios) is that the observer begins every trial in the same state, which enhances the comparability of the conditions plotted above. However, one might wonder how much better the observer would perform when the whole block is devoted to one condition. Random interleaving produces uncertainty; blocking each condition does not. Testing one observer (MD) on three conditions (audio, visual, and audiovisual signal; noise always present in both streams) we find insignificant difference in thresholds measured with and without uncertainty (i.e. interleaved vs. blocked conditions). Furthermore, ideal observer thresholds for the same conditions are negligibly different with and without uncertainty. This indicates that the results presented in this figure, found with uncertainty, also apply to performance without uncertainty.</p

Efficiency for identifying a disk “letter”.

<p>Thresholds and efficiencies are reported as mean ± se.</p

Materials and Methods.

<p>(<b>A</b>) A sentence (or a word) is presented as two concurrent streams: text and speech in visual and audio white noise. The observer identifies the words. In Experiments 1, 3, and 4, the visual stream includes only one word. In Experiment 2, the visual stream is a rapid serial visual presentation <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Potter1" target="_blank">[35]</a> of a sentence, presented one word at a time. The audio stream presents the same words as the visual stream. (<b>B & C</b>) The critical difference between models B and C is whether the two streams converge before or after detecting the signal. This dichotomy has been called “pre- and post-labelling” in speech recognition <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Braida1" target="_blank">[36]</a>. A neural receptive field computes a weighted average of the stimulus, i.e. the cross correlation of the stimulus and the receptive field’s weighting function <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Barlow1" target="_blank">[37]</a>. In fact, if the noise is white, taking the weighting function to be a known signal, the receptive field is computing the log likelihood ratio of the presence of that signal in the stimulus, relative to zero signal. When the possible signals are equally probable, the best performance is attained by the maximum likelihood choice. (<b>B</b>) In probability summation, there is a receptive field for each possible signal. Detection occurs independently in each stream and the detections are combined logically to yield the overt decision. This is practically optimal when there is uncertainty among the known signals <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Pelli3" target="_blank">[38]</a>. (<b>C</b>) In linear summation there is just one receptive field. The signals are linearly combined by a single audio-visual receptive field, followed by a single detector, which emits the final decision <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Green1" target="_blank">[22]</a>. This is optimal for a known audiovisual signal.</p

A survey of summation.

<p>Taken from 23 papers, these are <i>summation</i> experiments: the observer’s thresholds for a compound stimulus are compared with the thresholds for the components of the compound. [31, Sec. 1.11.2]. The index of summation, <i>k</i>, ranges from none (<i>k = </i>0) to optimal (<i>k = </i>1). This table shows that, whether using one or several senses, whether the task is high- or low-level, the summation is strong (<i>k</i> near 1) only if the two components are both perceived as aspects of the “same” thing. When cues are “different” things, summation is weak 0≤ <i>k</i> ≤0.58 with a mean 0.37; when cues are “same”, summation is strong 0.68≤ <i>k</i> ≤1 with a mean 0.89. This table is based on 23 papers, including all the perceptual summation efficiencies for adults that we found (or could calculate from published results) in a quick survey of the literature. It includes only summation of cues that are consistent and informative. It omits cue conflict studies, in which the cues provide conflicting information about the quality to be reported. It also omits the facilitation paradigm, in which one of the cues provides no information. At an <i>n</i>-component threshold that equates the energy of the components, if the summation index is <i>k</i> then the summation efficiency is_251658240. Conservation of energy, i.e. the optimal algorithm, has <i>k</i> = 1. Independence of successes, i.e. probability summation, has <i>k</i> ≈ 0.57. In most cases the papers do not report whether their observers perceived the two components as the “same” object, so we have guessed, based on our experience and a close reading of what the papers do say. Another two papers turned up by our search could not be included in the Table because we were unable to classify their stimuli, with any confidence, as “same” or “different” <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Alais2" target="_blank">[45]</a>, <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803-Landy1" target="_blank">[65]</a>. Obviously our guesses must yield to better assessments. <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone.0064803.s003" target="_blank">Table S1</a> explains how <i>k</i> was estimated for each paper. Note that the predicted value for the summation index of roughly 0.6 for probability summation is for detection, whereas most of the experiments in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone-0064803-t001" target="_blank">Table 1</a> are identification.</p

Histogram of the values of the summation index <i>k</i> reported in Table 1.

<p>Histogram of the values of the summation index <i>k</i> reported in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0064803#pone-0064803-t001" target="_blank">Table 1</a>.</p

Mean ± SEM of reaction time (in seconds) during equal-interval trials.

<p>Regardless of interval-difference, flute-tone intervals elicited shorter reaction times from all subjects, compared to intervals with pure and voice tones (shown with <b>*</b>, <i>p</i><0.05).</p

Processed behavioral and neurophysiological data 2

Processed behavioral and neurophysiological data