179,716 research outputs found
Latent Relation Language Models
In this paper, we propose Latent Relation Language Models (LRLMs), a class of
language models that parameterizes the joint distribution over the words in a
document and the entities that occur therein via knowledge graph relations.
This model has a number of attractive properties: it not only improves language
modeling performance, but is also able to annotate the posterior probability of
entity spans for a given text through relations. Experiments demonstrate
empirical improvements over both a word-based baseline language model and a
previous approach that incorporates knowledge graph information. Qualitative
analysis further demonstrates the proposed model's ability to learn to predict
appropriate relations in context
Extracting Multi-valued Relations from Language Models
The widespread usage of latent language representations via pre-trained
language models (LMs) suggests that they are a promising source of structured
knowledge. However, existing methods focus only on a single object per
subject-relation pair, even though often multiple objects are correct. To
overcome this limitation, we analyze these representations for their potential
to yield materialized multi-object relational knowledge. We formulate the
problem as a rank-then-select task. For ranking candidate objects, we evaluate
existing prompting techniques and propose new ones incorporating domain
knowledge. Among the selection methods, we find that choosing objects with a
likelihood above a learned relation-specific threshold gives a 49.5% F1 score.
Our results highlight the difficulty of employing LMs for the multi-valued
slot-filling task and pave the way for further research on extracting
relational knowledge from latent language representations.Comment: Accepted to Repl4NLP Workshop at ACL 202
Semantic spaces
Any natural language can be considered as a tool for producing large
databases (consisting of texts, written, or discursive). This tool for its
description in turn requires other large databases (dictionaries, grammars
etc.). Nowadays, the notion of database is associated with computer processing
and computer memory. However, a natural language resides also in human brains
and functions in human communication, from interpersonal to intergenerational
one. We discuss in this survey/research paper mathematical, in particular
geometric, constructions, which help to bridge these two worlds. In particular,
in this paper we consider the Vector Space Model of semantics based on
frequency matrices, as used in Natural Language Processing. We investigate
underlying geometries, formulated in terms of Grassmannians, projective spaces,
and flag varieties. We formulate the relation between vector space models and
semantic spaces based on semic axes in terms of projectability of subvarieties
in Grassmannians and projective spaces. We interpret Latent Semantics as a
geometric flow on Grassmannians. We also discuss how to formulate G\"ardenfors'
notion of "meeting of minds" in our geometric setting.Comment: 32 pages, TeX, 1 eps figur
Identifying Linear Relational Concepts in Large Language Models
Transformer language models (LMs) have been shown to represent concepts as
directions in the latent space of hidden activations. However, for any given
human-interpretable concept, how can we find its direction in the latent space?
We present a technique called linear relational concepts (LRC) for finding
concept directions corresponding to human-interpretable concepts at a given
hidden layer in a transformer LM by first modeling the relation between subject
and object as a linear relational embedding (LRE). While the LRE work was
mainly presented as an exercise in understanding model representations, we find
that inverting the LRE while using earlier object layers results in a powerful
technique to find concept directions that both work well as a classifier and
causally influence model outputs
Recommended from our members
Universal Schema for Knowledge Representation from Text and Structured Data
In data integration we transform information from a source into a target schema. A general problem in this task is loss of fidelity and coverage: the source expresses more knowledge than that can be fit into the target schema, or knowledge that is hard to fit into any schema at all. This problem is taken to an extreme in information extraction (IE) where the source is natural language---one of the most expressive forms of knowledge representation. To address this issue, one can either automatically learn a latent schema emergent in text (a brittle and ill-defined task), or manually define schemas. We propose instead to store data in a probabilistic representation of universal schema. This schema is simply the union of all source schemas, and we learn how to predict the cells of each source relation in this union. For example, we could store Freebase relations and relations that are expressed by natural language surface patterns. To populate such a database of universal schema, we present matrix factorization models that learn latent embedding vectors for entity tuples and relations.
We show that such latent models achieve substantially higher accuracy than a traditional classification approach on New York Times and Freebase data. Besides binary relations, we use universal schema for unary relations, i.e., entity types. We explore various facets of universal schema matrix factorization models on a large-scale web corpus, including implicature among the relations. We evaluate our approach on the task of question answering using features obtained from universal schema, achieving state-of-the-art accuracy on a benchmark dataset
A Deep Architecture for Semantic Parsing
Many successful approaches to semantic parsing build on top of the syntactic
analysis of text, and make use of distributional representations or statistical
models to match parses to ontology-specific queries. This paper presents a
novel deep learning architecture which provides a semantic parsing system
through the union of two neural models of language semantics. It allows for the
generation of ontology-specific queries from natural language statements and
questions without the need for parsing, which makes it especially suitable to
grammatically malformed or syntactically atypical text, such as tweets, as well
as permitting the development of semantic parsers for resource-poor languages.Comment: In Proceedings of the Semantic Parsing Workshop at ACL 2014
(forthcoming
- …