Graph-based broad-coverage semantic parsing

Many broad-coverage meaning representations can be characterized as directed graphs, where nodes represent semantic concepts and directed edges represent semantic relations among the concepts. The task of semantic parsing is to generate such a meaning representation from a sentence. It is quite natural to adopt a graph-based approach for parsing, where nodes are identified conditioning on the individual words, and edges are labeled conditioning on the pairs of nodes. However, there are two issues with applying this simple and interpretable graph-based approach for semantic parsing: first, the anchoring of nodes to words can be implicit and non-injective in several formalisms (Oepen et al., 2019, 2020). This means we do not know which nodes should be generated from which individual word and how many of them. Consequently, it makes a probabilistic formulation of the training objective problematical; second, graph-based parsers typically predict edge labels independent from each other. Such an independence assumption, while being sensible from an algorithmic point of view, could limit the expressiveness of statistical modeling. Consequently, it might fail to capture the true distribution of semantic graphs. In this thesis, instead of a pipeline approach to obtain the anchoring, we propose to model the implicit anchoring as a latent variable in a probabilistic model. We induce such a latent variable jointly with the graph-based parser in an end-to-end differentiable training. In particular, we test our method on Abstract Meaning Representation (AMR) parsing (Banarescu et al., 2013). AMR represents sentence meaning with a directed acyclic graph, where the anchoring of nodes to words is implicit and could be many-to-one. Initially, we propose a rule-based system that circumvents the many-to-one anchoring by combing nodes in some pre-specified subgraphs in AMR and treats the alignment as a latent variable. Next, we remove the need for such a rule-based system by treating both graph segmentation and alignment as latent variables. Still, our graph-based parsers are parameterized by neural modules that require gradient-based optimization. Consequently, training graph-based parsers with our discrete latent variables can be challenging. By combing deep variational inference and differentiable sampling, our models can be trained end-to-end. To overcome the limitation of graph-based parsing and capture interdependency in the output, we further adopt iterative refinement. Starting with an output whose parts are independently predicted, we iteratively refine it conditioning on the previous prediction. We test this method on semantic role labeling (Gildea and Jurafsky, 2000). Semantic role labeling is the task of predicting the predicate-argument structure. In particular, semantic roles between the predicate and its arguments need to be labeled, and those semantic roles are interdependent. Overall, our refinement strategy results in an effective model, outperforming strong factorized baseline models

The mat sat on the cat : investigating structure in the evaluation of order in machine translation

We present a multifaceted investigation into the relevance of word order in machine translation. We introduce two tools, DTED and DERP, each using dependency structure to detect differences between the structures of machine-produced translations and human-produced references. DTED applies the principle of Tree Edit Distance to calculate edit operations required to convert one structure into another. Four variants of DTED have been produced, differing in the importance they place on words which match between the two sentences. DERP represents a more detailed procedure, making use of the dependency relations between words when evaluating the disparities between paths connecting matching nodes. In order to empirically evaluate DTED and DERP, and as a standalone contribution, we have produced WOJ-DB, a database of human judgments. Containing scores relating to translation adequacy and more specifically to word order quality, this is intended to support investigations into a wide range of translation phenomena. We report an internal evaluation of the information in WOJ-DB, then use it to evaluate variants of DTED and DERP, both to determine their relative merit and their strength relative to third-party baselines. We present our conclusions about the importance of structure to the tools and their relevance to word order specifically, then propose further related avenues of research suggested or enabled by our work

Vector Semantics

This open access book introduces Vector semantics, which links the formal theory of word vectors to the cognitive theory of linguistics. The computational linguists and deep learning researchers who developed word vectors have relied primarily on the ever-increasing availability of large corpora and of computers with highly parallel GPU and TPU compute engines, and their focus is with endowing computers with natural language capabilities for practical applications such as machine translation or question answering. Cognitive linguists investigate natural language from the perspective of human cognition, the relation between language and thought, and questions about conceptual universals, relying primarily on in-depth investigation of language in use. In spite of the fact that these two schools both have ‘linguistics’ in their name, so far there has been very limited communication between them, as their historical origins, data collection methods, and conceptual apparatuses are quite different. Vector semantics bridges the gap by presenting a formal theory, cast in terms of linear polytopes, that generalizes both word vectors and conceptual structures, by treating each dictionary definition as an equation, and the entire lexicon as a set of equations mutually constraining all meanings

Entwurf und Implementation einer auf Graph-Grammatiken beruhenden Sprache zur Funktions-Struktur-Modellierung von Pflanzen

Increasing biological knowledge requires more and more elaborate methods to translate the knowledge into executable model descriptions, and increasing computational power allows to actually execute these descriptions. Such a simulation helps to validate, extend and question the knowledge. For plant modelling, the well-established formal description language of Lindenmayer systems reaches its limits as a method to concisely represent current knowledge and to conveniently assist in current research. On one hand, it is well-suited to represent structural and geometric aspects of plant models - of which units is a plant composed, how are these connected, what is their location in 3D space -, but on the other hand, its usage to describe functional aspects - what internal processes take place in the plant structure, how does this interact with the structure - is not as convenient as desirable. This can be traced back to the underlying representation of structure as a linear chain of units, while the intrinsic nature of the structure is a tree or even a graph. Therefore, we propose to use graphs and graph grammars as a basis for plant modelling which combines structural and functional aspects. In the first part of this thesis, we develop the necessary theoretical framework. Starting with a presentation of the state of the art concerning Lindenmayer systems and graph grammars, we develop the formalism of relational growth grammars as a variant of graph grammars. We show that this formalism has a natural embedding of Lindenmayer systems which keeps all relevant properties, but represents branched structures directly as axial trees and not as linear chains with indirect encoding of branches. In the second part, we develop the main practical result, the XL programming language as an extension of the Java programming language by very general rule-based features. Short examples illustrate the application of the new language features. We describe the built-in pattern matching algorithm of the implemented run-time system for the XL programming language, and we sketch a possible implementation of an XL compiler. The third part is an application of relational growth grammars and the XL programming language. We show how the general XL interfaces can be customized for relational growth grammars. On top of this customization, several examples from a variety of disciplines demonstrate the usefulness of the developed formalism and language to describe plant growth, especially functional-structural plant models, but also artificial life, architecture or interactive games. Some examples operate on custom graphs like XML DOM trees or scene graphs of commercial 3D modellers, while the majority uses the 3D modelling platform GroIMP, a software developed in conjunction with this thesis. The appendix gives an overview of the GroIMP software. The practical usage of its plug-in for relational growth grammars is also illustrated.Das zunehmende Wissen über biologische Prozesse verlangt nach geeigneten Methoden, es in ausführbare Modelle zu übersetzen, und die zunehmende Rechenleistung der Computer ermöglicht es, diese Modelle auch tatsächlich auszuführen. Solche Simulationen dienen zur Validierung, Erweiterung und Hinterfragung des Wissens. Speziell für die Pflanzenmodellierung wurden Lindenmayer-Systeme mit Erfolg eingesetzt, jedoch stoßen diese bei aktuellen Modellierungsproblemen und Forschungsvorhaben an ihre Grenzen. Zwar sind sie gut geeignet, Pflanzenstruktur und Geometrie abzubilden - aus welchen Einheiten setzt sich eine Pflanze zusammen, wie sind diese verbunden, wie ist ihre räumliche Lage -, aber die lineare Datenstruktur erschwert die Integration von Funktionsmodellen, welche Prozesse innerhalb der verzweigten Struktur und des beanspruchten Raumes beschreiben. Daher wird in dieser Arbeit vorgeschlagen, anstelle der linearen Stuktur Graphen und Graph-Grammatiken als Grundlage für die kombinierte Funktions-Struktur-Modellierung von Pflanzen zu verwenden. Im ersten Teil der Dissertation wird der theoretische Unterbau entwickelt. Nach einer Vorstellung des aktuellen Wissensstandes auf dem Gebiet der Lindenmayer-Systeme und Graph-Grammatiken werden relationale Wachstumsgrammatiken eingeführt, die auf bekannten Mechanismen für parallele Graph-Grammatiken aufbauen und Lindenmayer-Systeme als Spezialfall enthalten, dabei jedoch verzweigte Strukturen direkt als axiale Bäume darstellen. Zur praktischen Anwendung wird im zweiten Teil die Programmiersprache XL entwickelt, die Java um allgemein gehaltene Sprachkonstrukte für Graph-Grammatiken erweitert. Kurze Beispiele zeigen die Anwendung der neuen Sprachmerkmale. Der Algorithmus zur Mustersuche wird erläutert, und die Implementation des XL-Compilers wird vorgestellt. Im dritten Teil werden mögliche Anwendungen relationaler Wachstumsgrammatiken aufgezeigt. Dazu werden zunächst die allgemeinen XL-Schnittstellen für relationale Wachstumsgrammatiken konkretisiert, um dieses System dann für Modelle aus verschiedenen Bereichen zu nutzen, darunter Funktions-Struktur-Modelle von Pflanzen, Künstliches Leben, Architektur und interaktive Spiele. Einige Beispiele nutzen spezifische Graphen wie XML-DOM-Bäume oder Szenengraphen kommerzieller 3D-Modellierprogramme, aber der überwiegende Teil baut auf der 3D-Plattform GroIMP auf, die zusammen mit dieser Dissertation entwickelt wurde. Im Anhang wird die Software GroIMP kurz vorgestellt und ihre praktische Anwendung für relationale Wachstumsgrammatiken erläutert

Evolutionary Computation

This book presents several recent advances on Evolutionary Computation, specially evolution-based optimization methods and hybrid algorithms for several applications, from optimization and learning to pattern recognition and bioinformatics. This book also presents new algorithms based on several analogies and metafores, where one of them is based on philosophy, specifically on the philosophy of praxis and dialectics. In this book it is also presented interesting applications on bioinformatics, specially the use of particle swarms to discover gene expression patterns in DNA microarrays. Therefore, this book features representative work on the field of evolutionary computation and applied sciences. The intended audience is graduate, undergraduate, researchers, and anyone who wishes to become familiar with the latest research work on this field

Graph Neural Networks for Natural Language Processing: A Survey

Deep learning has become the dominant approach in coping with various tasks in Natural LanguageProcessing (NLP). Although text inputs are typically represented as a sequence of tokens, there isa rich variety of NLP problems that can be best expressed with a graph structure. As a result, thereis a surge of interests in developing new deep learning techniques on graphs for a large numberof NLP tasks. In this survey, we present a comprehensive overview onGraph Neural Networks(GNNs) for Natural Language Processing. We propose a new taxonomy of GNNs for NLP, whichsystematically organizes existing research of GNNs for NLP along three axes: graph construction,graph representation learning, and graph based encoder-decoder models. We further introducea large number of NLP applications that are exploiting the power of GNNs and summarize thecorresponding benchmark datasets, evaluation metrics, and open-source codes. Finally, we discussvarious outstanding challenges for making the full use of GNNs for NLP as well as future researchdirections. To the best of our knowledge, this is the first comprehensive overview of Graph NeuralNetworks for Natural Language Processing.Comment: 127 page