Neural Mechanisms for Combinatorial Semantics in Language and Vision: Evidence From FMRI, Patients, and Brain Stimulation

Throughout our daily experience, humans make nearly constant use of semantic knowledge. Over the last 20-30 years, the majority of work on the neural basis of semantic memory has examined the representation of semantic categories (e.g., animate versus inanimate). However, a defining aspect of human cognition is the ability to integrate this stored semantic information to form complex combinations of concepts. For example, humans can comprehend “plaid” and “jacket” as separate concepts, but can also effortlessly integrate this information to create the idea of a “plaid jacket.” This process is essential to human cognition, but little work has examined the neural regions that underlie conceptual combination. Many models of semantic memory have proposed that convergence zones, or neural hubs, help to integrate the semantic features of word meaning to form coherent representations from stored semantic knowledge. However, few studies have specifically examined the integrative semantic functions that these high-level hub regions carry out. This thesis presents three experiments that examine lexical-semantic combinatorial processing (as in the “plaid jacket” example above): 1) a study in healthy adults using fMRI, 2) a study in healthy adults using brain stimulation, and 3) a study examining impairments of lexical-semantic integration in patients with neurodegenerative disease. The fourth and final experiment of this thesis examines semantic aspects of combinatorial codes for visual-object representation. This study identifies neural regions that encode the feature combinations that define an object’s meaning. The findings from these four experiments elucidate specific cortical hubs for semantic-feature integration during language comprehension and visual-object processing, and they advance our understanding of the role of heteromodal brain regions in semantic memory

Single-Neuron Correlates of Visual Object Representation in the Human Brain: Effects of Attention, Memory, and Choice

Neurons in the medial temporal lobe (amygdala and hippocampus) are known to respond selectively to specific object categories, such as faces. This dissertation investigates two novel extensions of this work: (1) how are such neuronal responses influenced by where we attend; (2) how is category information used by the brain to make decisions. The first question evaluated the representation of faces in the primate amygdala during naturalistic conditions, by recording from both human and macaque amygdala neurons during free viewing of arrays of images with concurrent eye tracking. We found that category-selective responses were very strongly modulated by where people, or monkeys, fixated (overt attention). Subsequent experiments in humans only further demonstrated that this effect holds even when people allocate visual attention while keeping central fixation (covert attention). In both monkeys and humans, the majority of face-selective neurons preferred faces of conspecifics, a bias also seen behaviorally in first fixation preferences. Response latencies, relative to fixation onset, were shortest for conspecific-selective neurons. Response latencies were also notably shorter in monkeys than in humans. To answer the second question, we investigated how visual representations in the medial temporal lobe are subsequently used to make two types of decisions: a recognition memory choice ("have you seen this image before?"), and a stimulus categorization choice ("Is this a face?"). We show that (i) there are distinct populations of cells in the medial frontal cortex (including anterior cingulate cortex, and supplementary motor cortex) encoding recognition memory or categorization-based choices; (ii) category-selective cells in the medial temporal lobe are insensitive to such task conditions; and (iii) spike-field coherence between field potentials in the medial temporal lobe and action potentials in the medial frontal cortex is enhanced during recognition memory choices. This suggests that inter-areal communication between these two brain regions may be facilitated selectively in tasks that rely on recognition memory-based information. Taken together these two components of this dissertation provide novel insights into how visual object representations in the human brain are gated by attention, and how they are used in decisions. This work thus for the first time provides a comprehensive characterization of how single neurons in the human brain participate in the cycle from perception to action.</p