A Neurocognitive Approach

One of the most important hallmarks of human language is the use of symbolic signs in that words just “symbolically” represent the objects they refer to. That is, neither the written word ‘tree’ nor the spoken sequence of sounds /triː/ has any inherent similarity or analogy to a real tree. This is a core assumption of classic linguistics, which states that words are paired with objects, mental images, or concepts in an arbitrary fashion. Thus, the sound of a word per se has no inherent semantic content, nor does it play any contributing role in shaping the meaning of words. The goal of this dissertation is to contemplate, to rethink, and to examine the aforementioned statement, which has dominated language research throughout the last century. Indeed, there are a vast number of counterexamples showing how meaningful single phonemes, or their combination—as in nonwords—can be. Consider the standalone role of sound in poetry, or the use of single syllables or phonemes in various sacred rituals, or the prevalence of onomatopoetic words, i.e., words that sound similar to what they mean (e.g., click, zigzag), across all of the languages in the world, or the tendency of language users toward using harsh-sounding words as swear words, or cross-linguistic phenomena such as a preference to match the nonword BOUBA with a curvy round shape and KIKI with a spiky angular shape: These all constitute excellent examples that could potentially falsify the radical assumption of the sound of words being per se meaningless. The present dissertation now wants to shed new empirical light on this old debate, which dates back to Greek antiquity, yet faces a number of unanswered questions regarding the cognitive and neuropsychological mechanisms underlying the potential effect of sound on the processes of meaning making. More specifically, my focus is on the existence of sound-meaning relationships in the affective domain, termed affective iconicity, and on the investigation of different aspects of this phenomenon in both everyday language and poetry. By taking an interdisciplinary approach, the present work combines a variety of methods and techniques, such as behavioral and neuroimaging experiments, phonological and acoustic analysis of a large-scale lexicon, computational modelling, and corpus analysis, in order to provide comprehensive answers for this multi-faceted phenomenon. The theoretical part of the dissertation explores and integrates linguistic perspectives on the iconic mappings, explaining how a linguistic sign can acquire meaning based on similarities between its form and the object it refers to. A surprising neglect of the role of emotion in empirical models of language in general, and in previous investigations of iconicity in particular, is discussed. Specific hypotheses are formulated via the predictions made by the recently proposed Neurocognitive Poetics Model (NCPM), and through reviewing the previous works on the topic. Accordingly, three main questions are formulated that the present dissertation aims to address: i) Does the sound of words evoke affective responses observable at the behavioral and neural level? ii) Does the sound of words influence the processes of meaning making in the affective domain? iii) Does the sound of words in a poem contribute to its global affective meaning as perceived by readers? Six empirical studies attempt to address these questions which are subdivided into six more precise research questions. Results of the empirical part provide a comprehensive picture of the interplay between sound and meaning at different levels of processing (i.e., rating, semantic decision, and passive listening) for different presentation modalities (i.e., visual, and auditory) and for different textual levels (i.e., single word, and entire text). In short, results of Study 1 and Study 2 indicated a high similarity between the affective potential of the sound of words and other types of affective sounds (e.g., nonverbal emotional vocalization and affective prosody) at both the level of psychological perception (Study 1) and the level of neural correlates and substrates (Study 2). Furthermore, when giving their affective judgments (valence and arousal) about the meaning of words, participants, as shown in Study 1, were implicitly influenced by the sound of words even when words were presented visually and read silently. These results were extended in Study 3 in which iconic words, as operationalized by congruence between affective sound and affective meaning, were evaluated more quickly and more accurately than their non-iconic counterparts, suggesting that a similarity between the form and meaning of a word may help language users to more readily access its meaning through direct form-meaning mappings. Study 4 investigated the neural mechanisms underlying the facilitative effect observed in Study 3. Results showed an enhanced fMRI signal in the left amygdala, known for its role in multimodal emotion integration, for both a comparison between iconic and non-iconic words, as well as functional connectivity between two seed regions representing the sound (superior temporal gyrus) and meaning (inferior frontal gyrus) of words modulated by iconic condition. Lastly, results of Study 5 and Study 6 emphasize the role of foregrounded phonological units in the affective and aesthetic processes of literary reading. This clearly supports the initial hypotheses that iconicity is a feasible indicator of the affective qualities of a literary text as evoked by particular phonemic structures. The presented method for measuring the basic affective tone of the poems investigated could account for a considerable part of the variance in the ratings of their general affective meaning. In summary, this dissertation provides strong psychological and neuroimaging evidence for a device that has long been deployed in poetry and the arts, i.e., evoking affective (and aesthetic) responses by the use of certain words with specific sound patterns. The results were used to upgrade the standard models of visual word processing by conceiving corresponding modules responsible for the evaluation of affective sound and its interaction with the evaluation of the affective meaning of words. Lastly, at the more complex level of the whole text, the findings of this dissertation confirm the central assumption of the NCPM regarding the role of foregrounded elements in enhancing affective perception, although the Panksepp-Jakopson hypothesis might need to be extended to human-specific brain regions which originally evolved for other, more simple, tasks. Also, the literary model of reading may need to be updated by adding feedback loops from resulting reading behavior (e.g., fluent reading) to the perceived emotions (e.g., lust and play) based on the findings concerning the facilitative role of iconicity in language processing

On the Relation between the General Affective Meaning and the Basic Sublexical, Lexical, and Inter-lexical Features of Poetic Texts - A Case Study Using 57 Poems of H. M. Enzensberger

The literary genre of poetry is inherently related to the expression and elicitation of emotion via both content and form. To explore the nature of this affective impact at an extremely basic textual level, we collected ratings on eight different general affective meaning scales—valence, arousal, friendliness, sadness, spitefulness, poeticity, onomatopoeia, and liking—for 57 German poems (“die verteidigung der wölfe”) which the contemporary author H. M. Enzensberger had labeled as either “friendly,” “sad,” or “spiteful.” Following Jakobson's (1960) view on the vivid interplay of hierarchical text levels, we used multiple regression analyses to explore the specific influences of affective features from three different text levels (sublexical, lexical, and inter-lexical) on the perceived general affective meaning of the poems using three types of predictors: (1) Lexical predictor variables capturing the mean valence and arousal potential of words; (2) Inter-lexical predictors quantifying peaks, ranges, and dynamic changes within the lexical affective content; (3) Sublexical measures of basic affective tone according to sound-meaning correspondences at the sublexical level (see Aryani et al., 2016). We find the lexical predictors to account for a major amount of up to 50% of the variance in affective ratings. Moreover, inter-lexical and sublexical predictors account for a large portion of additional variance in the perceived general affective meaning. Together, the affective properties of all used textual features account for 43–70% of the variance in the affective ratings and still for 23–48% of the variance in the more abstract aesthetic ratings. In sum, our approach represents a novel method that successfully relates a prominent part of variance in perceived general affective meaning in this corpus of German poems to quantitative estimates of affective properties of textual components at the sublexical, lexical, and inter-lexical level

Why 'piss' is ruder than 'pee'? The role of sound in affective meaning making

Most language users agree that some words sound harsh (e.g. grotesque) whereas others sound soft and pleasing (e.g. lagoon). While this prominent feature of human language has always been creatively deployed in art and poetry, it is still largely unknown whether the sound of a word in itself makes any contribution to the word’s meaning as perceived and interpreted by the listener. In a large-scale lexicon analysis, we focused on the affective substrates of words’ meaning (i.e. affective meaning) and words’ sound (i.e. affective sound); both being measured on a two-dimensional space of valence (ranging from pleasant to unpleasant) and arousal (ranging from calm to excited). We tested the hypothesis that the sound of a word possesses affective iconic characteristics that can implicitly influence listeners when evaluating the affective meaning of that word. The results show that a significant portion of the variance in affective meaning ratings of printed words depends on a number of spectral and temporal acoustic features extracted from these words after converting them to their spoken form (study1). In order to test the affective nature of this effect, we independently assessed the affective sound of these words using two different methods: through direct rating (study2a), and through acoustic models that we implemented based on pseudoword materials (study2b). In line with our hypothesis, the estimated contribution of words’ sound to ratings of words’ affective meaning was indeed associated with the affective sound of these words; with a stronger effect for arousal than for valence. Further analyses revealed crucial phonetic features potentially causing the effect of sound on meaning: For instance, words with short vowels, voiceless consonants, and hissing sibilants (as in ‘piss’) feel more arousing and negative. Our findings suggest that the process of meaning making is not solely determined by arbitrary mappings between formal aspects of words and concepts they refer to. Rather, even in silent reading, words’ acoustic profiles provide affective perceptual cues that language users may implicitly use to construct words’ overall meaning

Making sense of social interaction: Emotional coherence drives semantic integration as assessed by event-related potentials

Available online 18 January 2019.We compared event-related potentials during sentence reading, using impression formation equations of a model of affective coherence, to investigate the role of affective content processing during meaning making. The model of Affect Control Theory (ACT; Heise, 1979, 2007) predicts and quantifies the degree to which social interactions deflect from prevailing social norms and values – based on the affective meanings of involved concepts. We tested whether this model can predict the amplitude of brain waves traditionally associated with semantic processing. To this end, we visually presented sentences describing basic subject-verb-object social interactions and measured event-related potentials for final words of sentences from three different conditions of affective deflection (low, medium, high) as computed by a variant of the ACT model (Schröder, 2011). Sentence stimuli were closely controlled across conditions for alternate semantic dimensions such as contextual constraints, cloze probabilities, co-occurrences of subject-object and verb-object relations. Personality characteristics (schizotypy, Big Five) were assessed to account for individual differences, assumed to influence emotion-language interactions in information processing. Affective deflection provoked increased negativity of ERP waves during the P2/N2 and N400 components. Our data suggest that affective incoherence is perceived as conflicting information interfering with early semantic processing and that increased respective processing demands – in particular in the case of medium violations of social norms – linger on until the N400 time window classically associated with the integration of concepts into embedding context. We conclude from these results that affective meanings influence basic stages of meaning making.This research was funded by a grant from the Cluster of Excellence “Languages of Emotion” at Freie Universität Berlin to C.v.S, T.S. and M.C. (project #411) as part of the Excellence Initiative (research grant EXC 302) of the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG)

Affective Congruence between Sound and Meaning of Words Facilitates Semantic Decision

A similarity between the form and meaning of a word (i.e., iconicity) may help language users to more readily access its meaning through direct form-meaning mapping. Previous work has supported this view by providing empirical evidence for this facilitatory effect in sign language, as well as for onomatopoetic words (e.g., cuckoo) and ideophones (e.g., zigzag). Thus, it remains largely unknown whether the beneficial role of iconicity in making semantic decisions can be considered a general feature in spoken language applying also to “ordinary” words in the lexicon. By capitalizing on the affective domain, and in particular arousal, we organized words in two distinctive groups of iconic vs. non-iconic based on the congruence vs. incongruence of their lexical (meaning) and sublexical (sound) arousal. In a two-alternative forced choice task, we asked participants to evaluate the arousal of printed words that were lexically either high or low arousing. In line with our hypothesis, iconic words were evaluated more quickly and more accurately than their non-iconic counterparts. These results indicate a processing advantage for iconic words, suggesting that language users are sensitive to sound-meaning mappings even when words are presented visually and read silently

Interplay of bigram frequency and orthographic neighborhood statistics in language membership decision

Language-specific orthography (i.e., letters or bigrams that exist in only one language) is known to facilitate language membership recognition. Yet the contribution of continuous sublexical and lexical statistics to language membership decisions during visual word processing is unknown. Here, we used pseudo-words to investigate whether continuous sublexical and lexical statistics bias explicit language decisions (Experiment 1) and language attribution during naming (Experiment 2). We also asked whether continuous statistics would have an effect in the presence of orthographic markers. Language attribution in both experiments was influenced by lexical neighborhood size differences between languages, even in presence of orthographic markers. Sublexical frequencies of occurrence affected reaction times only for unmarked pseudo-words in both experiments, with greater effects in naming. Our results indicate that bilinguals rely on continuous language-specific statistics at sublexical and lexical levels to infer language membership. Implications are discussed with respect to models of bilingual visual word recognition.Peer Reviewe

The Sound of Words Evokes Affective Brain Responses

The long history of poetry and the arts, as well as recent empirical results suggest that the way a word sounds (e.g., soft vs. harsh) can convey affective information related to emotional responses (e.g., pleasantness vs. harshness). However, the neural correlates of the affective potential of the sound of words remain unknown. In an fMRI study involving passive listening, we focused on the affective dimension of arousal and presented words organized in two discrete groups of sublexical (i.e., sound) arousal (high vs. low), while controlling for lexical (i.e., semantic) arousal. Words sounding high arousing, compared to their low arousing counterparts, resulted in an enhanced BOLD signal in bilateral posterior insula, the right auditory and premotor cortex, and the right supramarginal gyrus. This finding provides first evidence on the neural correlates of affectivity in the sound of words. Given the similarity of this neural network to that of nonverbal emotional expressions and affective prosody, our results support a unifying view that suggests a core neural network underlying any type of affective sound processing

Phonological Iconicity electrifies: An ERP study on affective sound-to-meaning correspondences in German

While linguistic theory posits an arbitrary relation between signifiers and the signified (de Saussure, 1916), our analysis of a large-scale German database containing affective ratings of words revealed that certain phoneme clusters occur more often in words denoting concepts with negative and arousing meaning. Here, we investigate how such phoneme clusters that potentially serve as sublexical markers of affect can influence language processing. We registered the EEG signal during a lexical decision task with a novel manipulation of the words’ putative sublexical affective potential: the means of valence and arousal values for single phoneme clusters, each computed as a function of respective values of words from the database these phoneme clusters occur in. Our experimental manipulations also investigate potential contributions of formal salience to the sublexical affective potential: Typically, negative high-arousing phonological segments—based on our calculations—tend to be less frequent and more structurally complex than neutral ones. We thus constructed two experimental sets, one involving this natural confound, while controlling for it in the other. A negative high-arousing sublexical affective potential in the strictly controlled stimulus set yielded an early posterior negativity (EPN), in similar ways as an independent manipulation of lexical affective content. When other potentially salient formal features at the sublexical level were not controlled for, the effect of the sublexical affective potential was strengthened and prolonged (250-650 ms), presumably because formal salience helps making specific phoneme clusters efficient sublexical markers of negative high-arousing affective meaning. These neurophysiological data support the assumption that the organization of a language´s vocabulary involves systematic sound-to-meaning correspondences at the phonemic level that influence the way we process language

Phonological Iconicity Electrifies:An ERP Study on Affective Sound-to-Meaning Correspondences in German

While linguistic theory posits an arbitrary relation between signifiers and the signified (de Saussure, 1916), our analysis of a large-scale German database containing affective ratings of words revealed that certain phoneme clusters occur more often in words denoting concepts with negative and arousing meaning. Here, we investigate how such phoneme clusters that potentially serve as sublexical markers of affect can influence language processing. We registered the EEG signal during a lexical decision task with a novel manipulation of the words' putative sublexical affective potential: the means of valence and arousal values for single phoneme clusters, each computed as a function of respective values of words from the database these phoneme clusters occur in. Our experimental manipulations also investigate potential contributions of formal salience to the sublexical affective potential: Typically, negative high-arousing phonological segments—based on our calculations—tend to be less frequent and more structurally complex than neutral ones. We thus constructed two experimental sets, one involving this natural confound, while controlling for it in the other. A negative high-arousing sublexical affective potential in the strictly controlled stimulus set yielded an early posterior negativity (EPN), in similar ways as an independent manipulation of lexical affective content did. When other potentially salient formal features at the sublexical level were not controlled for, the effect of the sublexical affective potential was strengthened and prolonged (250–650 ms), presumably because formal salience helps making specific phoneme clusters efficient sublexical markers of negative high-arousing affective meaning. These neurophysiological data support the assumption that the organization of a language's vocabulary involves systematic sound-to-meaning correspondences at the phonemic level that influence the way we process language