405 research outputs found
Representations Matter: Embedding Modes of Large Language Models using Dynamic Mode Decomposition
Existing large language models (LLMs) are known for generating "hallucinated"
content, namely a fabricated text of plausibly looking, yet unfounded, facts.
To identify when these hallucination scenarios occur, we examine the properties
of the generated text in the embedding space. Specifically, we draw inspiration
from the dynamic mode decomposition (DMD) tool in analyzing the pattern
evolution of text embeddings across sentences. We empirically demonstrate how
the spectrum of sentence embeddings over paragraphs is constantly low-rank for
the generated text, unlike that of the ground-truth text. Importantly, we find
that evaluation cases having LLM hallucinations correspond to ground-truth
embedding patterns with a higher number of modes being poorly approximated by
the few modes associated with LLM embedding patterns. In analogy to near-field
electromagnetic evanescent waves, the embedding DMD eigenmodes of the generated
text with hallucinations vanishes quickly across sentences as opposed to those
of the ground-truth text. This suggests that the hallucinations result from
both the generation techniques and the underlying representation
DCU 250 Arabic dependency bank: an LFG gold standard resource for the Arabic Penn treebank
This paper describes the construction of a dependency bank gold standard for Arabic, DCU 250 Arabic Dependency Bank (DCU 250), based on the Arabic Penn Treebank Corpus (ATB) (Bies and Maamouri, 2003; Maamouri and Bies, 2004) within the theoretical framework of Lexical Functional Grammar (LFG). For parsing and automatically extracting grammatical and lexical resources from treebanks, it is necessary to evaluate against established gold standard resources. Gold standards for various languages have been developed, but to our knowledge, such a resource has not yet been constructed for Arabic. The construction of the DCU 250 marks the first step
towards the creation of an automatic LFG f-structure annotation algorithm for the ATB,
and for the extraction of Arabic grammatical and lexical resources
- ā¦