Syntactic Data Augmentation Increases Robustness to Inference Heuristics

Pretrained neural models such as BERT, when fine-tuned to perform natural language inference (NLI), often show high accuracy on standard datasets, but display a surprising lack of sensitivity to word order on controlled challenge sets. We hypothesize that this issue is not primarily caused by the pretrained model's limitations, but rather by the paucity of crowdsourced NLI examples that might convey the importance of syntactic structure at the fine-tuning stage. We explore several methods to augment standard training sets with syntactically informative examples, generated by applying syntactic transformations to sentences from the MNLI corpus. The best-performing augmentation method, subject/object inversion, improved BERT's accuracy on controlled examples that diagnose sensitivity to word order from 0.28 to 0.73, without affecting performance on the MNLI test set. This improvement generalized beyond the particular construction used for data augmentation, suggesting that augmentation causes BERT to recruit abstract syntactic representations.Comment: ACL 202

Linear Connectivity Reveals Generalization Strategies

It is widely accepted in the mode connectivity literature that when two neural networks are trained similarly on the same data, they are connected by a path through parameter space over which test set accuracy is maintained. Under some circumstances, including transfer learning from pretrained models, these paths are presumed to be linear. In contrast to existing results, we find that among text classifiers (trained on MNLI, QQP, and CoLA), some pairs of finetuned models have large barriers of increasing loss on the linear paths between them. On each task, we find distinct clusters of models which are linearly connected on the test loss surface, but are disconnected from models outside the cluster -- models that occupy separate basins on the surface. By measuring performance on specially-crafted diagnostic datasets, we find that these clusters correspond to different generalization strategies: one cluster behaves like a bag of words model under domain shift, while another cluster uses syntactic heuristics. Our work demonstrates how the geometry of the loss surface can guide models towards different heuristic functions.Comment: Publushed as a conference paper at ICLR 202

O man do not scribble on the book : print and counter-print in a Scottish Englightenment university

The University of St. Andrews in the latter half of the eighteenth century was a small and under-funded institution. Nevertheless, it shared in the changes and accomplishments that made Scottish intellect in that period a European phenomenon. Some evidence of the university's intellectual vitality at that time can be seen in the very full records that survive from its contemporary Library: records of the Library's administration and of its everyday business, and also such records as the books themselves represent. It is with this last sort of evidence that I will be mainly concerned here. The books at St. Andrews unwittingly preserve a remarkable corpus of marginalia added by the students. In this article, I hope to relate these student writings to their educational context. Making use of the distinctions that Walter Ong has so instructively drawn between print, manuscript, and oral habits of mind, I will suggest that the marginalia oppose the Enlightenment ideology of their university with the values of an older style of discourse.Note, quotation marks removed from title to ensure alphabetical order. Difference as follows; "O man do not scribble on the book": Print and Counter-print in a Scottish Enlightenment Universit

On Diversity

This article is part of a special forum titled “Ethnic Diversity in Music Theory: Voices from the Field.” Diversity has a relatively short but complex history inseparable from a vexing politics of cultural recognition in, and economic access to, American higher-education institutions. The authors consider this history along three interrelated axes—juridical, socio-political, and subjective—in order to discern the relation of cultural recognition and economic access to the ethos of the neoliberal university and to the structure of democratic institutions in late capitalism. The programmatic labor of the Society for Music Theory’s Committee on Diversity (1996–2007) provides the empirical backdrop for their discussion

Cross-Lingual Neural Network Speech Synthesis Based on Multiple Embeddings

The paper presents a novel architecture and method for speech synthesis in multiple languages, in voices of multiple speakers and in multiple speaking styles, even in cases when speech from a particular speaker in the target language was not present in the training data. The method is based on the application of neural network embedding to combinations of speaker and style IDs, but also to phones in particular phonetic contexts, without any prior linguistic knowledge on their phonetic properties. This enables the network not only to efficiently capture similarities and differences between speakers and speaking styles, but to establish appropriate relationships between phones belonging to different languages, and ultimately to produce synthetic speech in the voice of a certain speaker in a language that he/she has never spoken. The validity of the proposed approach has been confirmed through experiments with models trained on speech corpora of American English and Mexican Spanish. It has also been shown that the proposed approach supports the use of neural vocoders, i.e. that they are able to produce synthesized speech of good quality even in languages that they were not trained on

The Law of White Spaces: Race, Culture, and Legal Education

The scene, drawn from memory, is a first-year law school classroom. It is the early 1980s and the class is on civil procedure. The teacher is a white woman. She is nervous, and the class is dominated by students who provide standard right answers to formulaic law school questions. Other points of view, particularly those of a critical or feminist nature, are either passed over quickly or ignored. Questions of color are never mentioned. More than that, the teacher never calls on any African-American students. Students of color are either ignored completely or told, when they have questions, “We are moving on.” What initially seemed to be nervousness or inexperience becomes accentuated over time as discrimination. As the semester progresses, the African-American students start to test in subtle and quiet ways the teacher\u27s practice of excluding them. Things come to a head when they organize a systematic protest. After every statement or question made by the teacher, at least two of the students raise their hands. After several unsuccessful attempts at asking or answering questions, one African-American student confronts the teacher. When she tells him that she is moving on, the student insists: “I have a question.” The teacher reiterates: “We are moving on.” The student persists, the teacher repeats. All the African-American students then stand up and walk out of the class. One white student stands up and leaves as well. The rest of the class stays. The course continues without any real interruption. Sometime later a curt apology—“I didn\u27t mean to offend anyone”—suffices to paper over the color lines that the incident revealed