Data convergence in syntactic theory and the role of sentence pairs

Most acceptability judgments reported in the syntactic literature are obtained by linguists being their own informants. For well-represented languages like English, this method of data collection is best described as a process of community agreement, given that linguists typically discuss their judgments with colleagues. However, the process itself is comparably opaque, and the reliability of its output has been questioned. Recent studies looking into this criticism have shown that judgments reported in the literature for English can be replicated in quantitative experiments to a near-perfect degree. However, the focus of those studies has been on testing sentence pairs. We argue that replication of only contrasts is not sufcient, because theory building necessarily includes comparison across pairs and across papers. Thus, we test items at large, i. e. independent of counterparts. We created a corpus of grammaticality judgments on sequences of American English reported in articles published in Linguistic Inquiry and then collected experimental ratings for a random subset of them. Overall, expert ratings and experimental ratings converge to a good degree, but there are numerous instances in which ratings do not converge. Based on this, we argue that for theory-critical data, the process of community agreement should be accompanied by quantitative methods whenever possible

Syntax-semantics interactions – seeking evidence from a synchronic analysis of 38 languages [version 1; peer review: 1 approved]

The notion that, to facilitate processing, as semantic complexity increases, syntactic complexity decreases, follows from various linguistic theories. This brief report presents the results of testing that notion, by analysing synchronic data from 38languages and correlating canonical measures of semantic and syntactic difficulty. We expected an overall positive tendency. However, the results came out mixed to negative. There is a notable degree of variation and there are no clear tendencies within language families. After detailing the theoretic and cognitive reasons that support the original hypothesis, we conclude with a short discussion about the potential causes and implications of our findings. A possible interpretation is that the interaction we are looking for is more subtle than one might have assumed

Semantic Influences on Syntactic Acceptability Ratings

A high prevalence of syntactic gradience is well attested, but a comprehensive explanation is still needed. In the present paper, we look into the question of whether semantic influences could account for parts of the observed gradience. Results from two experiments suggest that semantic influences can have a degrading effect on the acceptability of grammatical items. However, we did not observe that they had an ameliorating effect, which still leaves a good deal of the observed gradience in need of an explanation

Exploring syntactic and semantic acceptability: A case study on semantic restriction violations and aspectual mismatches

The present study investigates the interaction between syntax and semantics, and its effects on acceptability. The study compares ratings from two experiments – a syntactic rating task and a semantic one – the latter asking for meaningfulness/plausibility. The focus is on two phenomena: semantic restriction violations and aspectual mismatches with for-PPs. For comparison, the experiments also include two reference phenomena: resumptive pronouns, which are ungrammatical but in principle meaningful/plausible, and semantic contradictions, which are not meaningful/plausible but grammatical. Further, we include anchor items of various degrees of grammaticality and meaningfulness/plausibility, in order to calibrate the scale and probe the rating space. The results for the resumptive pronouns and the semantic contradictions, as well as the anchor items, indicate that our participants struggled to distinguish between the two tasks to some degree. Semantic deviations seem to drag down syntactic acceptability, and syntactic anomalies drag down perceived meaningfulness/plausibility. Importantly, however, the results remain interpretable. We observe that the impact of semantic anomaly on syntactic acceptability differs across phenomena, as did the impact of syntactic deviations on semantic acceptability. Furthermore, the semantic restriction violations seem to affect semantic acceptability more than syntactic acceptability. By contrast, the for-PPs received reduced ratings in both tasks. Our findings further substantiate the notion that the border between syntax and semantics is not clear-cut and that the interface between the two is complex

Hot Topics Surrounding Acceptability Judgement Tasks

This paper discusses various "hot topics" concerning methodological issues in experimental syntax, with a focus on acceptability judgement tasks. We first review the literature on the question whether formal methods are necessary at all and argue that this is indeed the case. We then address questions concerning running experiments, with a focus on running experiments via the internet and dealing with non-cooperative behaviour. We review strategies to fend-off and to detect non-cooperative behaviour. Strategies based on response times can be used effectively to do so, already during the actual experiment. We show how quick clicking through an experiment can be prevented by giving a warning when response times fall below a predefined threshold. Sometimes participants counterbalance extremely short response times by pausing. Therefore, median response times rather than mean response times should be used for excluding participants post-experiment. In the final section, we present some thoughts on gradience and argue that recent findings make a case that the observed gradience is not just a by-product, but comes from the grammar itself and should be modelled as such

A blend of MaLT: Selected contributions from the Methods and Linguistic Theories Symposium 2015

Over the past few decades, linguistic theorizing has benefited from an increasing trend towards empirical methodologies across all disciplines. Methodological know-how – both productive and receptive – has thus become one of the key qualifications for researchers. The empirical turn in linguistics has gone hand in hand with a considerable diversification of research methods. This diversity, which has come to be seen as a strength of linguistics as a field, has also benefited linguistic theory building. The present volume contains selected contributions from the 2015 Methods and Linguistic Theories (MaLT) symposium that address the aforementioned issues from an empirical and/or theoretical perspective. They can be seen as the essence of what MaLT was about, and illustrate the range of topics covered as well as the various concerns and approaches that featured during the event

Acceptability judgement tasks and grammatical theory

This thesis considers various questions about acceptability judgement tasks (AJTs). In Chapter 1, we compare the prevalent informal method of syntactic enquiry, researcher introspection, to formal judgement tasks. We randomly sample 200 sentences from Linguistic Inquiry and then compare the original author judgements to online AJT ratings. Sprouse et al., 2013, provided a similar comparison, but they limited their analysis to the comparison of sentence pairs and to extreme cases. We think a comparison at large, i.e. involving all items, is more sensible. We find only a moderate match between informal author judgements and formal online ratings and argue that the formal judgements are more reliable than the informal judgements. Further, the fact that many syntactic theories rely on questionable informal data calls the adequacy of those theories into question. In Chapter 2, we test whether ratings for constructions from spoken language and constructions from written language differ if presented as speech vs as text and if presented informally vs formally. We analyse the results with an LME model and find that neither mode of presentation nor formality are significant factors. Our results suggest that a speakerâs grammatical intuition is fairly robust. In Chapter 3, we quantitatively compare regular AJT data to their Z-scores and ranked data. For our analysis, we test resampled data for significant differences in statistical power. We find that Z-scores and ranked data are more powerful than raw data across most common measurement methods. Chapter 4 examines issues surrounding a common similarity test, the TOST. It has long been unclear how to set its controlling parameter &delta;. Based on data simulations, we outline a way to objectively set &delta;. Further results suggest that our guidelines hold for any kind of data. The thesis concludes with an appendix on non-cooperative participants in AJTs.<p

Exploring syntactic and semantic acceptability: A case study on semantic restriction violations and aspectual mismatches

Häussler J, Juzek TS. Exploring syntactic and semantic acceptability: A case study on semantic restriction violations and aspectual mismatches. Finnish Journal of Linguistics. 2021;34:39-76.The present study investigates the interaction between syntax and semantics, and its effects on acceptability. The study compares ratings from two experiments – a syntactic rating task and a semantic one – the latter asking for meaningfulness/plausibility. The focus is on two phenomena: semantic restriction violations and aspectual mismatches with for-PPs. For comparison, the experiments also include two reference phenomena: resumptive pronouns, which are ungrammatical but in principle meaningful/plausible, and semantic contradictions, which are not meaningful/plausible but grammatical. Further, we include anchor items of various degrees of grammaticality and meaningfulness/plausibility, in order to calibrate the scale and probe the rating space. The results for the resumptive pronouns and the semantic contradictions, as well as the anchor items, indicate that our participants struggled to distinguish between the two tasks to some degree. Semantic deviations seem to drag down syntactic acceptability, and syntactic anomalies drag down perceived meaningfulness/plausibility. Importantly, however, the results remain interpretable. We observe that the impact of semantic anomaly on syntactic acceptability differs across phenomena, as did the impact of syntactic deviations on semantic acceptability. Furthermore, the semantic restriction violations seem to affect semantic acceptability more than syntactic acceptability. By contrast, the for-PPs received reduced ratings in both tasks. Our findings further substantiate the notion that the border between syntax and semantics is not clear-cut and that the interface between the two is complex

Data convergence in syntactic theory and the role of sentence pairs

Juzek TS, Häussler J. Data convergence in syntactic theory and the role of sentence pairs. Zeitschrift für Sprachwissenschaft. 2020;39(2):109-147.Most acceptability judgments reported in the syntactic literature are obtained by linguists being their own informants. For well-represented languages like English, this method of data collection is best described as a process of community agreement, given that linguists typically discuss their judgments with colleagues. However, the process itself is comparably opaque, and the reliability of its output has been questioned. Recent studies looking into this criticism have shown that judgments reported in the literature for English can be replicated in quantitative experiments to a near-perfect degree. However, the focus of those studies has been on testing sentence pairs. We argue that replication of only contrasts is not sufficient, because theory building necessarily includes comparison across pairs and across papers. Thus, we test items at large, i. e. independent of counterparts. We created a corpus of grammaticality judgments on sequences of American English reported in articles published in Linguistic Inquiry and then collected experimental ratings for a random subset of them. Overall, expert ratings and experimental ratings converge to a good degree, but there are numerous instances in which ratings do not converge. Based on this, we argue that for theory-critical data, the process of community agreement should be accompanied by quantitative methods whenever possible

Variation in Participants and Stimuli in Acceptability Experiments

Häussler J, Juzek TS. Variation in Participants and Stimuli in Acceptability Experiments. In: Goodall G, ed. The Cambridge Handbook of Experimental Syntax. Cambridge: Cambridge University Press; 2021: 97-117.Experimental syntax is an area that is rapidly growing as linguistic research becomes increasingly focused on replicable language data, in both fieldwork and laboratory environments. The first of its kind, this handbook provides an in-depth overview of current issues and trends in this field, with contributions from leading international scholars. It pays special attention to sentence acceptability experiments, outlining current best practices in conducting tests, and pointing out promising new avenues for future research. Separate sections review research results from the past 20 years, covering specific syntactic phenomena and language types. The handbook also outlines other common psycholinguistic and neurolinguistic methods for studying syntax, comparing and contrasting them with acceptability experiments, and giving useful perspectives on the interplay between theoretical and experimental linguistics. Providing an up-to-date reference on this exciting field, it is essential reading for students and researchers in linguistics interested in using experimental methods to conduct syntactic research