PaLI: A Jointly-Scaled Multilingual Language-Image Model

Effective scaling and a flexible task interface enable large language models to excel at many tasks. PaLI (Pathways Language and Image model) extends this approach to the joint modeling of language and vision. PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages. To train PaLI, we make use of large pretrained encoder-decoder language models and Vision Transformers (ViTs). This allows us to capitalize on their existing capabilities and leverage the substantial cost of training them. We find that joint scaling of the vision and language components is important. Since existing Transformers for language are much larger than their vision counterparts, we train the largest ViT to date (ViT-e) to quantify the benefits from even larger-capacity vision models. To train PaLI, we create a large multilingual mix of pretraining tasks, based on a new image-text training set containing 10B images and texts in over 100 languages. PaLI achieves state-of-the-art in multiple vision and language tasks (such as captioning, visual question-answering, scene-text understanding), while retaining a simple, modular, and scalable design

Konstruktion von Lexika von relationalen Phrasen

Knowledge Bases are one of the key components of Natural Language Understanding systems. For example, DBpedia, YAGO, and Wikidata capture and organize knowledge about named entities and relations between them, which is often crucial for tasks like Question Answering and Named Entity Disambiguation. While Knowledge Bases have good coverage of prominent entities, they are often limited with respect to relations. The goal of this thesis is to bridge this gap and automatically create lexicons of textual representations of relations, namely relational phrases. The lexicons should contain information about paraphrases, hierarchy, as well as semantic types of arguments of relational phrases. The thesis makes three main contributions. The first contribution addresses disambiguating relational phrases by aligning them with the WordNet dictionary. Moreover, the alignment allows imposing the WordNet hierarchy on the relational phrases. The second contribution proposes a method for graph construction of relations using Probabilistic Graphical Models. In addition, we apply this model to relation paraphrasing. The third contribution presents a method for constructing a lexicon of relational paraphrases with fine-grained semantic typing of arguments. This method is based on information from a multilingual parallel corpus.Wissensbanken sind Schlüsselkomponenten für sprachverarbeitende Systeme. Prominente Vertreter wie zum Beispiel DBpedia, Yago und Wikidata enthalten und organisieren Wissen über benannte Entitäten und deren Relationen zueinander. Das so strukturierte Wissen spielt oft eine zentrale Rolle für Aufgaben wie automatische Fragebeantwortung (engl. Question Answering) oder Disambiguierung von Entitäten. Wissensbanken haben eine gute Abdeckung an Entitäten, sind aber hinsichtlich Relationen oft limitiert. Das Ziel dieser Dissertation ist es diese Lücke zu schließen und automatisch Lexika zu erstellen, die textuelle Repräsentationen von Relationen, so genannte relationale Phrasen, zur Verfügung stellen. Die Lexika sollten neben Informationen zu Paraphrasen und der Hierarchie relationaler Phrasen auch semantische Typisierung der Argumente einer Relation umfassen. Diese Dissertation leistet dafür drei wesentliche Beiträge. Der erste Beitrag behandelt die Disambiguierung relationaler Phrasen durch Verknüpfung mit Einträgen des WordNet Lexikons. Diese Verknüpfung ermöglicht es die WordNet Hierarchie auf relationale Phrasen zu übertragen. Im zweiten Beitrag wird eine Methode zur Konstruktion eines Graphen aus Relationen mittels probabilistischer graphischer Modelle vorgeschlagen. Das erzeugte Modell wird darüber hinaus zur Paraphrasierung von Relationen angewandt. Der dritte Beitrag ist eine Methode zur Lexikonkonstruktion relationaler Paraphrasen mit feingranularer semantischer Typisierung der Argumente von Relationen. Diese Methode basiert auf Informationen aus multilingualen parallelen Korpora

HARPY: Hypernyms and Alignment of Relational Paraphrases

Collections of relational paraphrases have been automatically constructed from \u000Alarge text corpora, as a WordNet counterpart for the realm of binary predicates \u000Aand their surface forms.\u000AHowever, these resources fall short in their coverage of hypernymy links \u000A(subsumptions) among the synsets of phrases. \u000AThis paper closes this gap by computing a high‐quality alignment between the \u000Arelational phrases of the Patty taxonomy, one of the largest collections of \u000Athis kind, and the verb senses of WordNet. To this end, we devise judicious \u000Afeatures and develop a graph‐based alignment algorithm by adapting and \u000Aextending the SimRank random‐walk method.\u000AThe resulting taxonomy of relational phrases and verb senses, coined HARPY, \u000Acontains 20,812 synsets organized into a \em Directed Acyclic Graph (DAG)} \u000Awith 616,792 hypernymy links. \u000AOur empirical assessment, indicates that the alignment links between Patty and \u000AWordNet have high accuracy, with {\em Mean Reciprocal Rank (MRR)} score 0.7 and \u000A{\em Normalized Discounted Cumulative Gain (NDCG) score 0.73. \u000AAs an additional extrinsic value, HARPY provides fine‐grained lexical types for \u000Athe arguments of verb senses in WordNet

RELLY: Inferring Hypernym Relationships Between Relational Phrases

Relational phrases (e.g., “got married to”) and their hypernyms (e.g., “is a relative of”) are central for many tasks including question answering, open information ex-traction, paraphrasing, and entailment de-tection. This has motivated the develop-ment of several linguistic resources (e.g. DIRT, PATTY, and WiseNet) which sys-tematically collect and organize relational phrases. These resources have demonstra-ble practical benefits, but are each limited due to noise, sparsity, or size. We present a new general-purpose method, RELLY, for constructing a large hypernymy graph of relational phrases with high-quality sub-sumptions using collective probabilistic programming techniques. Our graph in-duction approach integrates small high-precision knowledge bases together with large automatically curated resources, and reasons collectively to combine these re-sources into a consistent graph. Using RELLY, we construct a high-coverage, high-precision hypernymy graph consist-ing of 20K relational phrases and 35K hy-pernymy links. Our evaluation indicates a hypernymy link precision of 78%, and demonstrates the value of this resource for a document-relevance ranking task.