Modality Switching in a Property Verification Task: An ERP Study of What Happens When Candles Flicker after High Heels Click

The perceptual modalities associated with property words, such as flicker or click, have previously been demonstrated to affect subsequent property verification judgments (Pecher et al., 2003). Known as the conceptual modality switch effect, this finding supports the claim that brain systems for perception and action help subserve the representation of concepts. The present study addressed the cognitive and neural substrate of this effect by recording event-related potentials (ERPs) as participants performed a property verification task with visual or auditory properties in key trials. We found that for visual property verifications, modality switching was associated with an increased amplitude N400. For auditory verifications, switching led to a larger late positive complex. Observed ERP effects of modality switching suggest property words access perceptual brain systems. Moreover, the timing and pattern of the effects suggest perceptual systems impact the decision-making stage in the verification of auditory properties, and the semantic stage in the verification of visual properties

Embodied Conceptual Combination

Conceptual combination research investigates the processes involved in creating new meaning from old referents. It is therefore essential that embodied theories of cognition are able to explain this constructive ability and predict the resultant behavior. However, by failing to take an embodied or grounded view of the conceptual system, existing theories of conceptual combination cannot account for the role of perceptual, motor, and affective information in conceptual combination. In the present paper, we propose the embodied conceptual combination (ECCo) model to address this oversight. In ECCo, conceptual combination is the result of the interaction of the linguistic and simulation systems, such that linguistic distributional information guides or facilitates the combination process, but the new concept is fundamentally a situated, simulated entity. So, for example, a cactus beetle is represented as a multimodal simulation that includes visual (e.g., the shiny appearance of a beetle) and haptic (e.g., the prickliness of the cactus) information, all situated in the broader location of a desert environment under a hot sun, and with (at least for some people) an element of creepy-crawly revulsion. The ECCo theory differentiates interpretations according to whether the constituent concepts are destructively, or non-destructively, combined in the situated simulation. We compare ECCo to other theories of conceptual combination, and discuss how it accounts for classic effects in the literature

Exploring modality switching effects in negated sentences: further evidence for grounded representations

Theories of embodied cognition (e.g., Perceptual Symbol Systems Theory; Barsalou, 1999, 2009) suggest that modality specific simulations underlie the representation of concepts. Supporting evidence comes from modality switch costs: participants are slower to verify a property in one modality (e.g., auditory, BLENDER-loud) after verifying a property in a different modality (e.g., gustatory, CRANBERRIES-tart) compared to the same modality (e.g., LEAVES-rustling, Pecher et al., 2003). Similarly, modality switching costs lead to a modulation of the N400 effect in event-related potentials (ERPs; Collins et al., 2011; Hald et al., 2011). This effect of modality switching has also been shown to interact with the veracity of the sentence (Hald et al., 2011). The current ERP study further explores the role of modality match/mismatch on the processing of veracity as well as negation (sentences containing “not”). Our results indicate a modulation in the ERP based on modality and veracity, plus an interaction. The evidence supports the idea that modality specific simulations occur during language processing, and furthermore suggest that these simulations alter the processing of negation

The Role of Motor Affordances in Visual Working Memory

Motor affordances are important for object knowledge. Semantic tasks on visual objects often show interactions with motor actions. Prior neuro-imaging studies suggested that motor affordances also play a role in visual working memory for objects. When participants remembered manipulable objects (e.g., hammer) greater premotor cortex activation was observed than when they remembered non-manipulable objects (e.g., polar bear). In the present study participants held object pictures in working memory while performing concurrent tasks such as articulation of nonsense syllables and performing hand movements. Although concurrent tasks did interfere with working memory performance, in none of the experiments did we find any evidence that concurrent motor tasks affected memory differently for manipulable and non-manipulable objects. I conclude that motor affordances are not used for visual working memory

Switching modalities in a sentence verification task: ERP evidence for embodied language processing

In an event related potential (ERP) experiment using written language materials only, we investigated a potential modulation of the N400 by the modality switch effect. The modality switch effect occurs when a first sentence, describing a fact grounded in one modality, is followed by a second sentence describing a second fact grounded in a different modality. For example, "A cellar is dark" (visual), was preceded by either another visual property "Ham is pink" or by a tactile property "A mitten is soft." We also investigated whether the modality switch effect occurs for false sentences ("A cellar is light"). We found that, for true sentences, the ERP at the critical word "dark" elicited a significantly greater frontal, early N400-like effect (270370 ms) when there was a modality mismatch than when there was a modality-match. This pattern was not found for the critical word "light" in false sentences. Results similar to the frontal negativity were obtained in a late time window (500700 ms). The obtained ERP effect is similar to one previously obtained for pictures. We conclude that in this paradigm we obtained fast access to conceptual properties for modality-matched pairs, which leads to embodiment effects similar to those previously obtained with pictorial stimuli

The basal ganglia: A vertebrate solution to the selection problem?

A selection problem arises whenever two or more competing systems seek simultaneous access to a restricted resource. Consideration of several selection architectures suggests there are significant advantages for systems which incorporate a central switching mechanism. We propose that the vertebrate basal ganglia have evolved as a centralized selection device, specialized to resolve conflicts over access to limited motor and cognitive resources. Analysis of basal ganglia functional architecture and its position within a wider anatomical framework suggests it can satisfy many of the requirements expected of an efficient selection mechanism

Hierarchical control over effortful behavior by rodent medial frontal cortex : a computational model

The anterior cingulate cortex (ACC) has been the focus of intense research interest in recent years. Although separate theories relate ACC function variously to conflict monitoring, reward processing, action selection, decision making, and more, damage to the ACC mostly spares performance on tasks that exercise these functions, indicating that they are not in fact unique to the ACC. Further, most theories do not address the most salient consequence of ACC damage: impoverished action generation in the presence of normal motor ability. In this study we develop a computational model of the rodent medial prefrontal cortex that accounts for the behavioral sequelae of ACC damage, unifies many of the cognitive functions attributed to it, and provides a solution to an outstanding question in cognitive control research-how the control system determines and motivates what tasks to perform. The theory derives from recent developments in the formal study of hierarchical control and learning that highlight computational efficiencies afforded when collections of actions are represented based on their conjoint goals. According to this position, the ACC utilizes reward information to select tasks that are then accomplished through top-down control over action selection by the striatum. Computational simulations capture animal lesion data that implicate the medial prefrontal cortex in regulating physical and cognitive effort. Overall, this theory provides a unifying theoretical framework for understanding the ACC in terms of the pivotal role it plays in the hierarchical organization of effortful behavior

A sharp image or a sharp knife: norms for the modality-exclusivity of 774 concept-property items

According to recent embodied cognition theories, mental concepts are represented by modality-specific sensory-motor systems. Much of the evidence for modality-specificity in conceptual processing comes from the property-verification task. When applying this and other tasks, it is important to select items based on their modality-exclusivity. We collected modality ratings for a set of 387 properties, each of which was paired with two different concepts, yielding a total of 774 concept-property items. For each item, participants rated the degree to which the property could be experienced through five perceptual modalities (vision, audition, touch, smell, and taste). Based on these ratings, we computed a measure of modality exclusivity, the degree to which a property is perceived exclusively through one sensory modality. In this paper, we briefly sketch the theoretical background of conceptual knowledge, discuss the use of the property-verification task in cognitive research, provide our norms and statistics, and validate the norms in a memory experiment. We conclude that our norms are important for researchers studying modality-specific effects in conceptual processing