Filling Gaps On-Line: Use of Lexical and Semantic Information in Sentence Processing

Two experiments investigated how people assign an interpretation to question phrases. In order to determine the meaning of the WH-phrase (e.g., who, what), a "gap" must be located and the role associated with the gap assigned to the WH-phrase. Two experiments tested the Lexical Expectation model of Fodor (1978), according to which lexical properties of the verb determine when a gap is posited, and the All Resorts model of Stowe (1984), according to which all possibilities are considered and evaluated on their pragmatic appropriateness. In Experiment 1, subjects judged the meaningfulness of full sentences. The frequency with which verbs are used transitively determined whether there was an effect of the plausibility of the WH-phrase to act as an object of the verb. Effects of plausibility of the WH-phrase as an object showed up in just those cases where the object role should be assigned to the WH-phrase according to the Lexical Expectation model, rather than as predicted by the All Resorts model. In Experiment 2, these results were replicated using the word-by-word self-paced reading paradigm. The plausibility effect showed up at the verb itself when it is normally used transitively. This evidence suggests that a gap is preferred even over a lexically filled object for transitive expectation verbs

Utilizing visual attention for cross-modal coreference interpretation

Abstract. In this paper, we describe an exploratory study to develop a model of visual attention that could aid automatic interpretation of exophors in situated dialog. The model is intended to support the reference resolution needs of embodied conversational agents, such as graphical avatars and robotic collaborators. The model tracks the attentional state of one dialog participant as it is represented by his visual input stream, taking into account the recency, exposure time, and visual distinctness of each viewed item. The model correctly predicts the correct referent of 52 % of referring expressions produced by speakers in human-human dialog while they were collaborating on a task in a virtual world. This accuracy is comparable with reference resolution based on calculating linguistic salience for the same data.

Beyond the sentence given

A central and influential idea among researchers of language is that our language faculty is organized according to Fregean compositionality, which states that the meaning of an utterance is a function of the meaning of its parts and of the syntactic rules by which these parts are combined. Since the domain of syntactic rules is the sentence, the implication of this idea is that language interpretation takes place in a two-step fashion. First, the meaning of a sentence is computed. In a second step, the sentence meaning is integrated with information from prior discourse, world knowledge, information about the speaker and semantic information from extra-linguistic domains such as co-speech gestures or the visual world. Here, we present results from recordings of event-related brain potentials that are inconsistent with this classical two-step model of language interpretation. Our data support a one-step model in which knowledge about the context and the world, concomitant information from other modalities, and the speaker are brought to bear immediately, by the same fast-acting brain system that combines the meanings of individual words into a message-level representation. Underlying the one-step model is the immediacy assumption, according to which all available information will immediately be used to co-determine the interpretation of the speaker's message. Functional magnetic resonance imaging data that we collected indicate that Broca's area plays an important role in semantic unification. Language comprehension involves the rapid incorporation of information in a ‘single unification space’, coming from a broader range of cognitive domains than presupposed in the standard two-step model of interpretation