Neural pathways for visual speech perception

This paper examines the questions, what levels of speech can be perceived visually, and how is visual speech represented by the brain? Review of the literature leads to the conclusions that every level of psycholinguistic speech structure (i.e., phonetic features, phonemes, syllables, words, and prosody) can be perceived visually, although individuals differ in their abilities to do so; and that there are visual modality-specific representations of speech qua speech in higher-level vision brain areas. That is, the visual system represents the modal patterns of visual speech. The suggestion that the auditory speech pathway receives and represents visual speech is examined in light of neuroimaging evidence on the auditory speech pathways. We outline the generally agreed-upon organization of the visual ventral and dorsal pathways and examine several types of visual processing that might be related to speech through those pathways, specifically, face and body, orthography, and sign language processing. In this context, we examine the visual speech processing literature, which reveals widespread diverse patterns activity in posterior temporal cortices in response to visual speech stimuli. We outline a model of the visual and auditory speech pathways and make several suggestions: (1) The visual perception of speech relies on visual pathway representations of speech qua speech. (2) A proposed site of these representations, the temporal visual speech area (TVSA) has been demonstrated in posterior temporal cortex, ventral and posterior to multisensory posterior superior temporal sulcus (pSTS). (3) Given that visual speech has dynamic and configural features, its representations in feedforward visual pathways are expected to integrate these features, possibly in TVSA

Doctor of Philosophy

dissertationThe primate auditory system is responsible for analyzing complex patterns of pressure differences and then synthesizing this information into a behaviorally relevant representation of the external world. How the auditory cortex accomplishes this complex task is unknown. This thesis examines the neural mechanisms underlying auditory perception in the primate auditory cortex, focusing on the neural representation of communication sounds. This thesis is composed of three studies of auditory cortical processing in the macaque and human. The first examines coding in primary and tertiary auditory cortex as it relates to the possibility for developing a stimulating auditory neural prosthesis. The second study applies an information theoretic approach to understanding information transfer between primary and tertiary auditory cortex. The final study examines visual influences on human tertiary auditory cortical processing during illusory audiovisual speech perception. Together, these studies provide insight into the cortical physiology underlying sound perception and insight into the creation of a stimulating cortical neural prosthesis for the deaf

Similarity in the generalization of implicitly learned sound patterns

A core property of language is the ability to generalize beyond observed examples. In two experiments, we explore how listeners generalize implicitly learned sound patterns to new nonwords and to new sounds, with the goal of shedding light on how similarity affects treatment of potential generalization targets. During the exposure phase, listeners heard nonwords whose onset consonant was restricted to a subset of a natural class (e.g., /d g v z Z/). During the test phase, listeners were presented with new nonwords and asked to judge how frequently they had been presented before; some of the test items began with a consonant from the exposure set (e.g., /d/), and some began with novel consonants with varying relations to the exposure set (e.g., /b/, which is highly similar to all onsets in the training set; /t/, which is highly similar to one of the training onsets; and /p/, which is less similar than the other two). The exposure onset was rated most frequent, indicating that participants encoded onset attestation in the exposure set, and generalized it to new nonwords. Participants also rated novel consonants as somewhat frequent, indicating generalization to onsets that did not occur in the exposure phase. While generalization could be accounted for in terms of featural distance, it was insensitive to natural class structure. Generalization to new sounds was predicted better by models requiring prior linguistic knowledge (either traditional distinctive features or articulatory phonetic information) than by a model based on a linguistically naïve measure of acoustic similarity

Inner Speech: Philosophical and Psychological Investigations

This project investigates philosophical and psychological aspects of “that little voice in our head”, or inner speech. Research on inner speech has been guided by the assumption that it is essentially a speech phenomenon. This dissertation argues against this assumption from three independent angles. In Chapter 2, I consider the idea that the content of inner speech is speech-specific. After arguing against this position, I go on to claim that the content of inner speech is vocalic. In Chapter 3, I examine models that treat inner speech as a prediction. I show that these models are problematic on both empirical and theoretical grounds. In their place, I argue for a model on which inner speech is treated as a goal state. In Chapter 4, I consider the popular idea that breakdowns in inner speech processing explain auditory verbal hallucination. I argue that inner speech-based explanations of AVH belong to a problematic class of psychological explanations, which confuse phenomenological and scientific understanding. My discussion of inner speech has implications for a wide range of topics, including metacognition, imagery, and the explanation of pathological mental states

An automated lexical stress classification tool for assessing dysprosody in childhood apraxia of speech

Childhood apraxia of speech (CAS) commonly affects the production of lexical stress contrast in polysyllabic words. Automated classification tools have the potential to increase reliability and efficiency in measuring lexical stress. Here, factors affecting the accuracy of a custom-built deep neural network (DNN)-based classification tool are evaluated. Sixteen children with typical development (TD) and 26 with CAS produced 50 polysyllabic words. Words with strong–weak (SW, e.g., dinosaur) or WS (e.g., banana) stress were fed to the classification tool, and the accuracy measured (a) against expert judgment, (b) for speaker group, and (c) with/without prior knowledge of phonemic errors in the sample. The influence of segmental features and participant factors on tool accuracy was analysed. Linear mixed modelling showed significant interaction between group and stress type, surviving adjustment for age and CAS severity. For TD, agreement for SW and WS words was >80%, but CAS speech was higher for SW (>80%) than WS (~60%). Prior knowledge of segmental errors conferred no clear advantage. Automatic lexical stress classification shows promise for identifying errors in children’s speech at diagnosis or with treatment-related change, but accuracy for WS words in apraxic speech needs improvement. Further training of algorithms using larger sets of labelled data containing impaired speech and WS words may increase accuracy

Hearing in dementia: defining deficits and assessing impact

The association between hearing impairment and dementia has emerged as a major public health challenge, with significant opportunities for earlier diagnosis, treatment and prevention. However, the nature of this association has not been defined. We hear with our brains, particularly within the complex soundscapes of everyday life: neurodegenerative pathologies target the auditory brain and are therefore predicted to damage hearing function early and profoundly. Here I present evidence for this proposition, based on structural and functional features of auditory brain organisation that confer vulnerability to neurodegeneration, the extensive, reciprocal interplay between ‘peripheral’ and ‘central’ hearing dysfunction, and recently characterised auditory signatures of canonical neurodegenerative dementias (Alzheimer’s disease and frontotemporal dementia). In chapter 3, I examine pure tone audiometric thresholds in AD and FTD syndromes and explore the functional interplay between the auditory brain and auditory periphery by assessing the contribution of auditory cognitive factors on pure tone detection. In chapter 4, I develop this further by examining the processing of degraded speech signals, leveraging the increased importance of top-down integrative and predictive mechanisms on resolving impoverished bottom-up sensory encoding. In chapter 5, I use a more discrete test of phonological processing to focus in on a specific brain region that is an early target in logopenic aphasia, to explore the potential of auditory cognitive tests as disease specific functional biomarkers. Finally, in chapter 6, I use auditory symptom questionnaires to capture real-world hearing in daily life amongst patients with dementia as well as their carers and measure how this correlates with audiometric performance and degraded speech processing. I call for a clinical assessment of real-world hearing in these diseases that moves beyond pure tone perception to the development of novel auditory ‘cognitive stress tests’ and proximity markers for the early diagnosis of dementia and management strategies that harness retained auditory plasticity

Nonword repetition depends on the frequency of sublexical representations at different grain sizes: evidence from a multi-factorial analysis

The nonword repetition task (NWR) has been widely used in basic cognitive and clinical research, as well as in clinical assessment, and has been proposed as a clinical marker for Specific Language Impairment (SLI). Yet the mechanisms underlying performance on this task are not clear. This study offers insights into these mechanisms through a comprehensive examination of item-related variables identified in previous research as possibly contributing to NWR scores and through testing the predictive power of each in relation to the others. A unique feature of the study is that all factors are considered simultaneously. Fifty-seven typically developing children were tested with a NWR task containing 150 nonwords differing in length, phonotactic probability, lexical neighbourhood and phonological complexity. The results indicate that phonological processing of novel words draws on sublexical representations at all grain sizes and that these representations are phonological, unstructured and insensitive to morphemehood. We propose a novel index – mean ngram frequency of all phonemes – that best captures the extent to which a nonword draws on sublexical representations. The study demonstrates the primacy of sublexical representations in NWR performance with implications for the nature of the deficit in SLI

Phoneme learning is influenced by the taxonomic organization of the semantic referents

International audienceWord learning relies on the ability to master the sound contrasts that are phonemic (i.e., signal meaning difference) in a given language. Though the timeline of phoneme development has been studied extensively over the past few decades, the mechanism of this development is poorly understood. Previous work has shown that human learners rely on referential information to differentiate similar sounds, but largely ignored the problem of taxonomic ambiguity at the semantic level (two different objects may be described by one or two words depending on how abstract the meaning intended by the speaker is). In this study, we varied the taxonomic distance of pairs of objects and tested how adult learners judged the phonemic status of the sound contrast associated with each of these pairs. We found that judgments were sensitive to gradients in the taxonomic structure, suggesting that learners use probabilistic information at the semantic level to optimize the accuracy of their judgements at the phonological level. The findings provide evidence for an interaction between phonological learning and meaning generalization, raising important questions about how these two important processes of language acquisition are related