Neural correlates of the processing of co-speech gestures

In communicative situations, speech is often accompanied by gestures. For example, speakers tend to illustrate certain contents of speech by means of iconic gestures which are hand movements that bear a formal relationship to the contents of speech. The meaning of an iconic gesture is determined both by its form as well as the speech context in which it is performed. Thus, gesture and speech interact in comprehension. Using fMRI, the present study investigated what brain areas are involved in this interaction process. Participants watched videos in which sentences containing an ambiguous word (e.g. She touched the mouse) were accompanied by either a meaningless grooming movement, a gesture supporting the more frequent dominant meaning (e.g. animal) or a gesture supporting the less frequent subordinate meaning (e.g. computer device). We hypothesized that brain areas involved in the interaction of gesture and speech would show greater activation to gesture-supported sentences as compared to sentences accompanied by a meaningless grooming movement. The main results are that when contrasted with grooming, both types of gestures (dominant and subordinate) activated an array of brain regions consisting of the left posterior superior temporal sulcus (STS), the inferior parietal lobule bilaterally and the ventral precentral sulcus bilaterally. Given the crucial role of the STS in audiovisual integration processes, this activation might reflect the interaction between the meaning of gesture and the ambiguous sentence. The activations in inferior frontal and inferior parietal regions may reflect a mechanism of determining the goal of co-speech hand movements through an observation-execution matching process

The iconicity advantage in sign production: The case of bimodal bilinguals

Recent evidence demonstrates that pictures corresponding to iconic signs are named faster than pictures corresponding to non-iconic signs. The present study investigates the locus of the iconicity advantage in hearing bimodal bilinguals. A naming experiment with iconic and noniconic pictures in Italian Sign Language (LIS) was conducted. Bimodal bilinguals named the pictures either using a noun construction that involved the production of the sign corresponding to the picture or using a marked demonstrative pronoun construction replacing the picture sign. In this last condition, the pictures were colored and participants were instructed to name the pronoun together with the color. The iconicity advantage was reliable in the noun utterance but not in the marked demonstrative pronoun utterance. In a third condition, the colored pictures were presented as distractor stimuli and participants required to name the color. In this last condition, distractor pictures with iconic signs elicited faster naming latencies than non-iconic signs. The results suggest that the advantage of iconic signs in production arises at the level of semantic-tophonological links. In addition, we conclude that bimodal bilinguals and native signers do not differ in terms of the activation flow within the sign production system

Iconic store and partial report

The iconic store has recently been challenged on the grounds that data in its favor may have resulted from some procedural artifacts. The display-instruction compatibility and perceptual grouping hypotheses were reexamined in two experiments with the partial-report paradigm. When care was taken to rectify some procedural problems found in Merikle's (1980) study, it was established that the iconic store (as a hypothetical mechanism) can still be validly entertained. This report demonstrates one important procedural point in studying the iconic store with the partial-report task, namely, that subjects must be given more than token training on the partial-report task

Supporting Device Discovery and Spontaneous Interaction with Spatial References

The RELATE interaction model is designed to support spontaneous interaction of mobile users with devices and services in their environment. The model is based on spatial references that capture the spatial relationship of a user’s device with other co-located devices. Spatial references are obtained by relative position sensing and integrated in the mobile user interface to spatially visualize the arrangement of discovered devices, and to provide direct access for interaction across devices. In this paper we discuss two prototype systems demonstrating the utility of the model in collaborative and mobile settings, and present a study on usability of spatial list and map representations for device selection

Departure from the onset-onset rule

Using a signal-detection task, the generality of Turvey's (1973) onset-onset rule was tested in four experiments. After seeing, in succession, (1) one or two letters (target display), (2) a multiletter detection display, and (3) a mask display, subjects decided whether or not the letter or letters in the target display reappeared in the succeeding detection display at different levels of detection-display duration in various situations. The subjects' sensitivity was inconsistent with the onset-onset rule. More specifically, sensitivity increased with increases in display duration within a fixed stimulus onset asynchrony of 150 msec. Display duration, however, had no effect on response bias. Nor was there any interaction between display duration and display size in terms of either sensitivity or response bias. The more complicated relationship between display duration and display size does not invalidate the departure from the onset-onset rule

The discourse deictics ^ and <-- in a World of Warcraft community

In the written English variety used in a community of World of Warcraft players, two iconic lexical items created from symbols have undergone semantic change. The words analyzed are ^ and <--, which have shifted from iconic deictic items used for discourse reference to non-iconic epistemic meanings. ^ shifted from a discourse deictic to an affirmative of a previous utterance, and <-- shifted to a self-identifying meaning similar to a pronoun. The existence and evolution of these lexical items are related to the medium in which they were created, as their meanings are associated with a visual-spatial environment created by textual chat in the virtual world. The different meanings of ^ and <-- currently exist in polysemy in the community, and the continuum of meanings are documented using data from natural language use spanning three years. A statistical analysis is performed on the data, and a diachronic change in meaning is found; furthermore, the observed change follows the path of semantic shift processes previously documented in spoken language. © 2012 Elsevier Ltd

Iconic memory, location information, and partial report

It has been suggested that the systematic decline of partial report as the delay of the partial-report cue increases is due to a time-related loss of location information. Moreover, the backward masking effect is said to be precipitated by the disruption of location information before and after identification. Results from three experiments do not support these claims when new indices of location information and of item information are used. Instead, it was found that (a) the systematic decline in partial report was due to a time-related loss of item information, and (b) location information was affected neither by the delay of the partial-report cue nor by the delay of backward masking. Subjects adopted the "select-then-identify" mode of processing

Readout from iconic memory involves similar neural processes as selective spatial attention

Iconic memory and spatial attention are often considered as distinct topics, but may have functional similarities. Here we provide fMRI evidence for some common underlying neural effects. Participants judged three visual stimuli in one hemifield of a bilateral array comprising six stimuli. The relevant hemifield for partial report was indicated by an auditory cue, administered either before the visual array (pre-cues, spatial attention) or shortly after (post-cues, iconic memory). Pre- and post-cues led to similar activity modulations in lateral occipital cortex, contralateral to the cued side, indicating that readout from iconic memory can have similar neural effects to spatial attention. We also found common bilateral activation of a fronto-parietal network for post-cue and pre-cue trials. These neuroimaging data suggest that some common neural mechanisms underlie selective spatial attention and readout from iconic memory. Some differences were also found, with post-cues leading to higher activity in right middle frontal gyrus

Visualization in spatial modeling

This chapter deals with issues arising from a central theme in contemporary computer modeling - visualization. We first tie visualization to varieties of modeling along the continuum from iconic to symbolic and then focus on the notion that our models are so intrinsically complex that there are many different types of visualization that might be developed in their understanding and implementation. This focuses the debate on the very way of 'doing science' in that patterns and processes of any complexity can be better understood through visualizing the data, the simulations, and the outcomes that such models generate. As we have grown more sensitive to the problem of complexity in all systems, we are more aware that the twin goals of parsimony and verifiability which have dominated scientific theory since the 'Enlightenment' are up for grabs: good theories and models must 'look right' despite what our statistics and causal logics tell us. Visualization is the cutting edge of this new way of thinking about science but its styles vary enormously with context. Here we define three varieties: visualization of complicated systems to make things simple or at least explicable, which is the role of pedagogy; visualization to explore unanticipated outcomes and to refine processes that interact in unanticipated ways; and visualization to enable end users with no prior understanding of the science but a deep understanding of the problem to engage in using models for prediction, prescription, and control. We illustrate these themes with a model of an agricultural market which is the basis of modern urban economics - the von Thünen model of land rent and density; a model of urban development based on interacting spatial and temporal processes of land development - the DUEM model; and a pedestrian model of human movement at the fine scale where control of such movements to meet standards of public safety is intrinsically part of the model about which the controllers know intimately. © Springer-Verlag Berlin Heidelberg 2006