Syntactic surprisal affects spoken word duration in conversational contexts

Abstract We present results of a novel experiment to investigate speech production in conversational data that links speech rate to information density. We provide the first evidence for an association between syntactic surprisal and word duration in recorded speech. Using the AMI corpus which contains transcriptions of focus group meetings with precise word durations, we show that word durations correlate with syntactic surprisal estimated from the incremental Roark parser over and above simpler measures, such as word duration estimated from a state-of-the-art text-to-speech system and word frequencies, and that the syntactic surprisal estimates are better predictors of word durations than a simpler version of surprisal based on trigram probabilities. This result supports the uniform information density (UID) hypothesis and points a way to more realistic artificial speech generation

Dynamic Formant Trajectories in German Read Speech: Impact of Predictability and Prominence

Phonetic structures expand temporally and spectrally when they are difficult to predict from their context. To some extent, effects of predictability are modulated by prosodic structure. So far, studies on the impact of contextual predictability and prosody on phonetic structures have neglected the dynamic nature of the speech signal. This study investigates the impact of predictability and prominence on the dynamic structure of the first and second formants of German vowels. We expect to find differences in the formant movements between vowels standing in different predictability contexts and a modulation of this effect by prominence. First and second formant values are extracted from a large German corpus. Formant trajectories of peripheral vowels are modeled using generalized additive mixed models, which estimate nonlinear regressions between a dependent variable and predictors. Contextual predictability is measured as biphone and triphone surprisal based on a statistical German language model. We test for the effects of the information-theoretic measures surprisal and word frequency, as well as prominence, on formant movement, while controlling for vowel phonemes and duration. Primary lexical stress and vowel phonemes are significant predictors of first and second formant trajectory shape. We replicate previous findings that vowels are more dispersed in stressed syllables than in unstressed syllables. The interaction of stress and surprisal explains formant movement: unstressed vowels show more variability in their formant trajectory shape at different surprisal levels than stressed vowels. This work shows that effects of contextual predictability on fine phonetic detail can be observed not only in pointwise measures but also in dynamic features of phonetic segments

Cognitive control of eye movements in reading and visual search: Evidence from frequency-based effects

Experiments in this dissertation investigate the role of cognition in eye-movement behavior during scanning and reading. Shorter and fewer fixations on the more frequent and predictable words have been observed during reading, but not when scanning text for a target word, e.g., zebra (e.g., Rayner & Raney, 1996). Past research has employed these effects of word frequency and predictability to argue that cognition drives eye movements during reading, but not during scanning. Similarly, the present studies use effects of stimulus frequency and predictability to index cognitive control of eye-movement behavior. Experiments 1 and 2 focus on the frequency effects for non-word and word stimuli, respectively. Experiment 1 employed clusters of Landolt Cs to examine how the gap size of and frequency of exposure to clusters affected eye movements during a scanning task. The findings demonstrated that, in parallel to word frequency effects observed in reading, more frequent clusters elicited fewer and shorter fixations. Experiment 2 compared eye-movement behavior on fully-crossed high- and low-frequency adjective-noun pairs embedded in paragraphs when participants were reading vs. scanning for a target word with an asterisk (e.g., “h*rse”), a word containing the letter "q" (e.g., “quilt”), or a word rhyming with "blue" (e.g., “shoe”). The results demonstrated that eye-movement measures are affected by frequency in the tasks requiring in-depth processing, such as reading and rhyme-judgment, but not in shallow-processing tasks like asterisk-detection. Experiments 3 and 4 focus on the frequency-based predictability effects for non-word and word stimuli, respectively. Experiment 3 employed similar materials to Experiment 1 and, in addition, manipulated frequency for pairs of clusters. The more predictable clusters in the repeating pairs elicited fewer fixations, providing tentative evidence of transitional predictability effects during scanning. Experiment 4 examined the effect of transitional probability in reading by increasing the frequency of co-occurrence for pairs of words (e.g., tulip’s blossoms) in paragraphs of text. The more predictable words elicited shorter first fixation durations, suggesting that co-occurrence frequency may result in forming short-term predictions during reading. Cumulatively, the findings demonstrate that cognitive effects are not unique to reading, and afford a more sophisticated characterization of the cognitive-oculomotor coordination

The influence of visual information on word predictability and processing effort

A word’s predictability or surprisal in linguistic context, as determined by cloze probabilities or languagemodels (e.g.,Frank,2013a) is related to processing effort, in that less expected words take more effort to process (e.g., Hale, 2001). This shows how, in purely linguistic contexts, rational approaches have been proven valid to predict and formalise results from language processing studies. However, the surprisal (or predictability) of a word may also be influenced by extra-linguistic factors, such as visual context information, as given in situated language processing. While, in the case of linguistic contexts, it is known that the incrementally processed information affects the mental model (e.g.,Zwaan and Radvansky, 1998) at each word in a probabilistic way, no such observations have been made so far in the case of visual context information. Although it has been shown that in the visual world paradigm (VWP), anticipatory eye movements suggest that listeners exploit the scene to predict what will be mentioned next (Altmann and Kamide,1999), it is so far unclear how visual information actually affects expectations for and processing effort of target words. If visual context effects on word processing effort can be observed, we hypothesise that rational concepts can be extended in order to formalise these effects, hereby making them statistically accessible for language models. In a line of experiments, I hence observe how visual information – which is inherently different from linguistic context, for instance in its non-incremental-at once-accessibility– affects target words. Our findings are a clear and robust demonstration that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort as assessed by two different on-line measures of effort (a pupillary and an EEG one). Finally, I use surprisal to formalise the measured results and propose an extended formula to take visual information into account.Deutsche Forschungsgemeinschaft (DFG), Grant SFB 110