Simulating Speed in Language: Contributions from vision, audition and action

Embodied theories propose that understanding meaning in language requires the mental simulation of entities being referred to. These mental simulations would make use of the same modality-specific systems involved in perceiving and acting upon such entities in the world, grounding language in the real world. However, embodied theories are currently underspecified in terms of how much information from an event is contained in mental simulations, and what features of experience are included. The thesis addresses comprehension of language that describes speed of events. Investigating speed allows embodied theories to be extended to a more complex feature of events. Further, speed is a fine-grained feature and thus testing an embodied theory of speed will reveal whether or not mental simulations include the fine details of real-world experience. Within the thesis four main methods of investigation were used, assessing simulation of speed with different types of speed language under different conditions: behavioural testing combining speed in language with speed in perception and action, eye-tracking investigating whether eye-movements to a visual scene are affected by speed in sentences, a psychophysics paradigm assessing whether speed in language affects visual perception processes, and finally, as a crucial test of embodiment, whether or not Parkinson’s patients, who have difficulty moving speedily, also have problems with comprehension of speed language. The main findings of the thesis are that: (1) speed, a fine-grained and abstract dimension, is simulated during comprehension, (2) simulations are dynamic and context-dependent, and (3) simulations of speed are specific to biological motion and can encode specific effectors used in an action. These results help to specify current embodied theories in terms of what the nature of simulations are and what factors they are sensitive to, in addition to broadly providing support for the sharing of cognitive/neural processes between language, action and perception

To dash or to dawdle: verb-associated speed of motion influences eye movements during spoken sentence comprehension

In describing motion events verbs of manner provide information about the speed of agents or objects in those events. We used eye tracking to investigate how inferences about this verb-associated speed of motion would influence the time course of attention to a visual scene that matched an event described in language. Eye movements were recorded as participants heard spoken sentences with verbs that implied a fast (“dash”) or slow (“dawdle”) movement of an agent towards a goal. These sentences were heard whilst participants concurrently looked at scenes depicting the agent and a path which led to the goal object. Our results indicate a mapping of events onto the visual scene consistent with participants mentally simulating the movement of the agent along the path towards the goal: when the verb implies a slow manner of motion, participants look more often and longer along the path to the goal; when the verb implies a fast manner of motion, participants tend to look earlier at the goal and less on the path. These results reveal that event comprehension in the presence of a visual world involves establishing and dynamically updating the locations of entities in response to linguistic descriptions of events

An integrated theory of language production and comprehension

Currently, production and comprehension are regarded as quite distinct in accounts of language processing. In rejecting this dichotomy, we instead assert that producing and understanding are interwoven, and that this interweaving is what enables people to predict themselves and each other. We start by noting that production and comprehension are forms of action and action perception. We then consider the evidence for interweaving in action, action perception, and joint action, and explain such evidence in terms of prediction. Specifically, we assume that actors construct forward models of their actions before they execute those actions, and that perceivers of others' actions covertly imitate those actions, then construct forward models of those actions. We use these accounts of action, action perception, and joint action to develop accounts of production, comprehension, and interactive language. Importantly, they incorporate well-defined levels of linguistic representation (such as semantics, syntax, and phonology). We show (a) how speakers and comprehenders use covert imitation and forward modeling to make predictions at these levels of representation, (b) how they interweave production and comprehension processes, and (c) how they use these predictions to monitor the upcoming utterances. We show how these accounts explain a range of behavioral and neuroscientific data on language processing and discuss some of the implications of our proposal

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task

Towards a complete multiple-mechanism account of predictive language processing [Commentary on Pickering & Garrod]

Although we agree with Pickering & Garrod (P&G) that prediction-by-simulation and prediction-by-association are important mechanisms of anticipatory language processing, this commentary suggests that they: (1) overlook other potential mechanisms that might underlie prediction in language processing, (2) overestimate the importance of prediction-by-association in early childhood, and (3) underestimate the complexity and significance of several factors that might mediate prediction during language processing

From Parallel Sequence Representations to Calligraphic Control: A Conspiracy of Neural Circuits

Calligraphic writing presents a rich set of challenges to the human movement control system. These challenges include: initial learning, and recall from memory, of prescribed stroke sequences; critical timing of stroke onsets and durations; fine control of grip and contact forces; and letter-form invariance under voluntary size scaling, which entails fine control of stroke direction and amplitude during recruitment and derecruitment of musculoskeletal degrees of freedom. Experimental and computational studies in behavioral neuroscience have made rapid progress toward explaining the learning, planning and contTOl exercised in tasks that share features with calligraphic writing and drawing. This article summarizes computational neuroscience models and related neurobiological data that reveal critical operations spanning from parallel sequence representations to fine force control. Part one addresses stroke sequencing. It treats competitive queuing (CQ) models of sequence representation, performance, learning, and recall. Part two addresses letter size scaling and motor equivalence. It treats cursive handwriting models together with models in which sensory-motor tmnsformations are performed by circuits that learn inverse differential kinematic mappings. Part three addresses fine-grained control of timing and transient forces, by treating circuit models that learn to solve inverse dynamics problems.National Institutes of Health (R01 DC02852

Mental Simulation in the Processing of Literal and Metaphorical Motion Language: An Eye Movement Study

An eye-tracking while listening study based on the blank screen paradigm was conducted to investigate the processing of literal and metaphorical verbs of motion. The study was based on two assumptions from the litera- ture: that language comprehension by default engages mental simulation, and that looking behavior (measured through patterns of eye movements) can provide a window into ongoing cognitive processes. This study specifi- cally compared the comprehension of sentences that depicted actual physi- cal motion (the curtain is rising) and sentences that described changes in quantity or emotional states in terms of vertical motion (prices are rising). Results showed that eye movements were selectively biased upward or downward in accordance with the direction implied by the verb, regardless of the context (literal or metaphorical) in which they appeared, and in the absence of any visual stimuli or explicit task. Thus, these findings suggest that literal and metaphorical language drive spontaneous, direction-specific mental simulations captured by eye movements and that at least in the case of verbs presented in the present progressive, which emphasizes the ongoing nature of actions, visual biases along the vertical axis may start during the verb itself

The implications of embodiment for behavior and cognition: animal and robotic case studies

In this paper, we will argue that if we want to understand the function of the brain (or the control in the case of robots), we must understand how the brain is embedded into the physical system, and how the organism interacts with the real world. While embodiment has often been used in its trivial meaning, i.e. 'intelligence requires a body', the concept has deeper and more important implications, concerned with the relation between physical and information (neural, control) processes. A number of case studies are presented to illustrate the concept. These involve animals and robots and are concentrated around locomotion, grasping, and visual perception. A theoretical scheme that can be used to embed the diverse case studies will be presented. Finally, we will establish a link between the low-level sensory-motor processes and cognition. We will present an embodied view on categorization, and propose the concepts of 'body schema' and 'forward models' as a natural extension of the embodied approach toward first representations.Comment: Book chapter in W. Tschacher & C. Bergomi, ed., 'The Implications of Embodiment: Cognition and Communication', Exeter: Imprint Academic, pp. 31-5