Comparing infrared and webcam eye tracking in the Visual World Paradigm

Visual World eye tracking is a temporally fine-grained method of monitoring attention, making it a popular tool in the study of online sentence processing. Recently, while infrared eye tracking was mostly unavailable, various web-based experiment platforms have rapidly developed webcam eye tracking functionalities, which are now in urgent need of testing and evaluation. We replicated a recent Visual World study on the incremental processing of verb aspect in English using ‘out of the box’ webcam eye tracking software (jsPsych; de Leeuw, 2015) and crowdsourced participants, and fully replicated both the offline and online results of the original study. We furthermore discuss factors influencing the quality and interpretability of webcam eye tracking data, particularly with regards to temporal and spatial resolution; and conclude that remote webcam eye tracking can serve as an affordable and accessible alternative to lab-based infrared eye tracking, even for questions probing the time-course of language processing

Verbal Symbols and Demonstrations Across Modalities

In this paper, I develop a new theory of the ingredients of semantic composition for the verb phrase, building on work in Ramchand (2018). I argue that the essential properties of this new approach make possible a new rapprochement between the theoretical analyses of symbolic verbal meaning across the visual and auditory modalities. The innovation of the new theory is that it partitions the verb phrase into a lower purely symbolic zone and a higher instantiational situation zone, mediated by the demonstrative act (cf. Davidson 2015, Henderson 2016). In the first part of the paper, I lay out the system, and then I show how it can be used to give a formal analysis of gestural and iconic elements in language. Along the way, I discuss the different ways in which iconicity in the two modalities plays out

Lexical items in complex predications : Selection as Underassociation

This paper examines the problem of selectional ‘matching’ effects in Bengali V-V complex predicates, and English denominal verbs within the context of a decompositional syntax/semantics for verbal meaning and a theory of lexical insertion under non-terminals. It argues that within the particular version of this kind of lexical insertion, as proposed by Ramchand 2008b, selection can be captured by the underassociation of category features constrained by Agree. In this way, I argue that we can achieve many of the effects of selection without any distinct lexical subcategorization frame, or sub-type of feature-checking, once we have a suitably articulated theory of lexical insertion

Licensing of Instrumental Case in Hindi/Urdu Causatives

In this paper, I revisit the licensing and interpretation of instrumental case-marked nominals in Hindi/Urdu causative constructions to argue against the hypothesis that the se-marked phrase corresponds to a demoted agent. Rather, I will argue that a more unified analysis of se-phrases can be achieved through an event-structural analysis, in line with the standard interpretation of other adverbials in the syntax. Since the ‘intermediate agent’ interpretation is only possible with indirect causatives in Hindi/Urdu, the event structural analysis proposed here also has implications for the direct vs. indirect causation distinction in the syntax

How (not) to look for meaning composition in the brain: A reassessment of current experimental paradigms

When we use language, we draw on a finite stock of lexical and functional meanings and grammatical structures to assign meanings to expressions of arbitrary complexity. According to the Principle of Compositionality, the meanings of complex expressions are a function of constituent meanings and syntax, and are generated by the recursive application of one or more composition operations. Given their central role in explanatory accounts of human language, it is surprising that relatively little is known about how the brain implements these composition operations in real time. In recent years, neurolinguistics has seen a surge of experiments investigating when and where in the brain meanings are composed. To date, however, neural correlates of composition have not been firmly established. In this article, we focus on studies that set out to find the correlates of linguistic composition. We critically examine the paradigms they employed, laying out the rationale behind each, their strengths and weaknesses. We argue that the still blurry picture of composition in the brain may be partly due to limitations of current experimental designs. We suggest that novel and improved paradigms are needed, and we discuss possible next steps in this direction. At the same time, rethinking the linguistic notion of composition, as based on a tight correspondence between syntax and semantics, might be in order

Deriving the Functional Hierarchy

There is a tension between Chomsky's recent Minimalist theory and the cartographic program initiated by Cinque. Cinque's cartography argues for a large number of fine-grained categories organized in one or more universal Rich Functional Hierarchies (RFH). The subtlety of the evidence and the richness of the inventory virtually force an innatist approach. In contrast, Chomsky argues for a minimal role for UG (MUG), shifting the burden to extralinguistic cognition, learning, and what he calls third factor principles such as principles of efficient computation. In this paper we reconcile the austere MUG vision of Chomsky with the impressive empirical evidence that Cinque and others have presented for RFH. We argue (building on previous work) that some Cartographic work overstates the universality of the orders observed, and furthermore conflates several different sources of ordering. Ordering sources include scope, polarity, and semantic category. Once these factors are properly understood, there remains an irreduceable universal functional hierarchy, for example that which orders epistemic modality and tense over root modality and aspect, and that which orders the latter over argument structure and Aktionsart (as discussed in much previous work). This residual core functional hierarchy (CFH) is unexplained so far by work which follows MUG. Rather than simply stipulating the CFH as part of UG, we reconcile CFH with MUG by detailing what nonlinguistic cognition must look like in order for MUG to derive the CFH. We furthermore show how an individual language develops a language-specific RFH which is consistent with the universal CFH, illustrating with a detailed account of the English auxiliary system

Fine-grained time course of verb aspect processing

Sentence processing is known to be highly incremental. Speakers make incremental commitments as the sentence unfolds, dynamically updating their representations based on the smallest pieces of information from the incoming speech stream. Less is known about linguistic processing on the sub-word level, especially with regard to abstract grammatical information. This study employs the Visual World Paradigm to investigate the processing of grammatical aspect by Russian-speaking adults (n = 124). Aspectual information is encoded relatively early within the Russian verb which makes this an ideal testing ground to investigate the incrementality of grammatical processing on the sub-word level. Participants showed preference for pictures of ongoing events when they heard sentences involving Imperfective verbs, and for pictures of completed events when they heard sentences involving Perfective verbs. Crucially, the analysis of the participants’ eye-movements showed that they exhibited preference for the target picture already before they heard the end of the verb. Moreover, the latency of this effect depended on where the aspectual information was encoded within the verb. These results indicate that the processing and integration of grammatical aspect information can happen rapidly and incrementally on a fine-grained wordinternal level. Methodologically, the study draws together a set of analytical techniques which can be fruitfully applied to the analysis of effect latencies in a wide range of studies within the Visual World eye-tracking paradigm

Aspect processing across languages: A visual world eye-tracking study

The study employed a combination of a picture selection task and Visual World eye-tracking to investigate the processing of grammatical aspect (perfective vs. imperfective) in three languages: Russian, Spanish and English. In order to probe into the cognitive representations triggered by the aspectual forms we contrasted visual representations of dierent temporal portions of telic events—a snapshot of the process stage (ongoing event) and a snapshot of the immediate aftermath of the event/the result state (completed event). In all three languages, the gaze patterns and o ine responses revealed a strong preference for representations of ongoing events in the imperfective condition. This confirms that the imperfective forms in all the three languages draw attention to the in-progress portion of a telic event. In the perfective condition, however, we found robust dierences. Russian uses verbal prefixes to mark perfective aspect, and our results suggest that perfective telic verbs in Russian strongly highlight the result state of an event. In Spanish, the perfective past tense form (Preterite) also highlights event completion, but to a lesser extent than in Russian—in line with its less restrictive semantics in not requiring an inherent boundary. In contrast to Russian and Spanish, English speakers did not show a preference for representations of completed events in the perfective (Simple Past) condition. This suggests that the English Simple Past form does not encode a preferential cognitive salience for either the activity portion of an event or its result state, and lends support to the analysis of the English Simple Past as a non-aspectual tense form

Temporal Information and Event Bounding Across Languages: Evidence from Visual World EyeTracking

We explore the typological question of what the interpretation of grammatical perfectivity is, and how it connects to the related aktionsartal notion of boundedness/telicity on the one hand, and the tense category Past on the other. We report on a comparative experimental paradigm of past tense accomplishment sentences in Russian, Spanish and English respectively, in which we use an online visual world paradigm -- comparing looks to an ongoing representation (OE) with a result state representation (CE) -- to track the triggering of entailments of culmination during auditory processing. In all three languages, the results revealed at-ceiling preference for OE in the imperfective condition both in the offline task and the online gaze patterns. In the perfective condition, we found robust differences. In Russian, the choice of the result state (CE) picture in the offline task was at ceiling (95 %); for Spanish it was high, but not quite at ceiling (83 %); in English there was no statistical preference for the CE picture in the Simple Past condition (54 %, not significantly different from chance, p=0.39). Analysis of the participants' online gaze patterns yielded parallel results. Our results for English suggest that even on telic predicates, the simple past form does not obligatorily enforce a completed-event interpretation, contrary to previous assumptions in the literature (Smith 1995)