Towards Incremental Parsing of Natural Language using Recursive Neural Networks

In this paper we develop novel algorithmic ideas for building a natural language parser grounded upon the hypothesis of incrementality. Although widely accepted and experimentally supported under a cognitive perspective as a model of the human parser, the incrementality assumption has never been exploited for building automatic parsers of unconstrained real texts. The essentials of the hypothesis are that words are processed in a left-to-right fashion, and the syntactic structure is kept totally connected at each step. Our proposal relies on a machine learning technique for predicting the correctness of partial syntactic structures that are built during the parsing process. A recursive neural network architecture is employed for computing predictions after a training phase on examples drawn from a corpus of parsed sentences, the Penn Treebank. Our results indicate the viability of the approach andlay out the premises for a novel generation of algorithms for natural language processing which more closely model human parsing. These algorithms may prove very useful in the development of eÆcient parsers

Computation and Linguistic Theory: A Government Binding Theory Parser Using Tree Adjoining Grammar (Master\u27s Thesis)

Government Binding (GB) theory, as a competence theory of grammar, is intended to define what a speaker\u27s knowledge of language consists of. The theory proposes a system of innate principles and constraints which determine the class of possible languages and, once instantiated by the parameter values for a given language, the class of well-formed sentences of that language [Chomsky, 1981]. In this thesis, I address the problem of how this knowledge of language is put to use. The answer I give to this question takes the shape of an implemented computational model, a parser, which utilizes the formulation of knowledge of language as proposed in GB theory. GB as a theory of grammar poses a particular problem for instantiation within a cognitively feasible computational model. It has a rich deductive structure whose obvious direct implementation as a set of axioms in a first order theorem prover runs up against the problem of undecidability. Thus, if we accept GB theory as psychologically real, and thus as functioning causally with respect to linguistic processing, there seems to be a paradox: we need a way of putting our knowledge of language, represented in GB theory, to use in a processing theory in an efficient manner. I will suggest a way out of this paradox. I propose to constrain the class of possible grammatical principles by requiring them to be statable over a linguistically and mathematically motivated domain, that of a tree adjoining grammar (TAG) elementary tree. The parsing process consists of the construction of such primitive structures, using a generalization of licensing relations as proposed in [Abney, 1986], and checking that the constraints are satisfied over these local domains. Since these domains are of bounded size, these constraints will be checkable in constant time and we will be guaranteed efficient, linear time, parsing. Additionally, the incrementality of the construction of the TAG elementary trees is consistent with intuitions of incremental semantic interpretation

Syntactic and Semantic Underspecification in the Verb Phrase.

This thesis is concerned with verbs and the relation between verbs and their complements. Syntactic evidence is presented which shows that the distinction between arguments and adjuncts reflects the optionality of adjuncts, but that adjuncts, once introduced, behave as arguments of the verb. An analysis is proposed which reflects this observation by assuming that verbal subcategorization is underspecified, so that optional constituents can be introduced into the verb phrase. The analysis is developed within a formal model of utterance interpretation. Labelled Deductive Systems for Natural Language (LDSNL), proposed in Kempson, Meyer-Viol & Gabbay (1999), which models the structural aspect of utterance interpretation as a dynamic process of tree growth during which lexical information is combined into more complex structures which provide vehicles for interpretation, propositional forms. The contribution of this thesis from the perspective of utterance interpretation is that it explores the notion of structural underspecification with respect to predicate-argument structure. After providing a formalization of underspecified verbal subcategorization, the thesis explores the consequences this analysis of verbs and verb phrases has for the process of tree growth, and how underspecified verbs are interpreted. The main argument developed is that verbs syntactically encode the possibilty for pragmatic enrichment; verbs address mental concepts only indirectly, so that the establishment of their eventual meaning, and, therefore, their eventual arity is mediated by the cognitive process of concept formation. Additional support for this view is provided by an analysis of applied verbs in Swahili which, from the perspective adopted here, can be seen to encode an explicit instruction for concept strengthening, an instruction to the hearer to derive additional inferential effects. The analysis presented in this thesis thus supports the view that natural language interpretation is a process in which structural properties and inferential activity are thoroughly intertwined

Modelling Incremental Self-Repair Processing in Dialogue.

PhDSelf-repairs, where speakers repeat themselves, reformulate or restart what they are saying, are pervasive in human dialogue. These phenomena provide a window into real-time human language processing. For explanatory adequacy, a model of dialogue must include mechanisms that account for them. Artificial dialogue agents also need this capability for more natural interaction with human users. This thesis investigates the structure of self-repair and its function in the incremental construction of meaning in interaction. A corpus study shows how the range of self-repairs seen in dialogue cannot be accounted for by looking at surface form alone. More particularly it analyses a string-alignment approach and shows how it is insufficient, provides requirements for a suitable model of incremental context and an ontology of self-repair function. An information-theoretic model is developed which addresses these issues along with a system that automatically detects self-repairs and edit terms on transcripts incrementally with minimal latency, achieving state-of-the-art results. Additionally it is shown to have practical use in the psychiatric domain. The thesis goes on to present a dialogue model to interpret and generate repaired utterances incrementally. When processing repaired rather than fluent utterances, it achieves the same degree of incremental interpretation and incremental representation. Practical implementation methods are presented for an existing dialogue system. Finally, a more pragmatically oriented approach is presented to model self-repairs in a psycholinguistically plausible way. This is achieved through extending the dialogue model to include a probabilistic semantic framework to perform incremental inference in a reference resolution domain. The thesis concludes that at least as fine-grained a model of context as word-by-word is required for realistic models of self-repair, and context must include linguistic action sequences and information update effects. The way dialogue participants process self-repairs to make inferences in real time, rather than filter out their disfluency effects, has been modelled formally and in practical systems.Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Account (DTA) scholarship from the School of Electronic Engineering and Computer Science at Queen Mary University of London

Learning categorial grammars

In 1967 E. M. Gold published a paper in which the language classes from the Chomsky-hierarchy were analyzed in terms of learnability, in the technical sense of identification in the limit. His results were mostly negative, and perhaps because of this his work had little impact on linguistics. In the early eighties there was renewed interest in the paradigm, mainly because of work by Angluin and Wright. Around the same time, Arikawa and his co-workers refined the paradigm by applying it to so-called Elementary Formal Systems. By making use of this approach Takeshi Shinohara was able to come up with an impressive result; any class of context-sensitive grammars with a bound on its number of rules is learnable. Some linguistically motivated work on learnability also appeared from this point on, most notably Wexler & Culicover 1980 and Kanazawa 1994. The latter investigates the learnability of various classes of categorial grammar, inspired by work by Buszkowski and Penn, and raises some interesting questions. We follow up on this work by exploring complexity issues relevant to learning these classes, answering an open question from Kanazawa 1994, and applying the same kind of approach to obtain (non)learnable classes of Combinatory Categorial Grammars, Tree Adjoining Grammars, Minimalist grammars, Generalized Quantifiers, and some variants of Lambek Grammars. We also discuss work on learning tree languages and its application to learning Dependency Grammars. Our main conclusions are: - formal learning theory is relevant to linguistics, - identification in the limit is feasible for non-trivial classes, - the `Shinohara approach' -i.e., placing a numerical bound on the complexity of a grammar- can lead to a learnable class, but this completely depends on the specific nature of the formalism and the notion of complexity. We give examples of natural classes of commonly used linguistic formalisms that resist this kind of approach, - learning is hard work. Our results indicate that learning even `simple' classes of languages requires a lot of computational effort, - dealing with structure (derivation-, dependency-) languages instead of string languages offers a useful and promising approach to learnabilty in a linguistic contex