2 research outputs found
ANALOGICAL - A New Benchmark for Analogy of Long Text for Large Language Models
Over the past decade, analogies, in the form of word-level analogies, have played a significant role as an intrinsic measure of evaluating the quality of word embedding methods such as word2vec. Modern large language models (LLMs), however, are primarily evaluated on extrinsic measures based on benchmarks such as GLUE and SuperGLUE, and there are only a few investigations on whether LLMs can draw analogies between long texts. In this paper, we present ANALOGICAL, a new benchmark to intrinsically evaluate LLMs across a taxonomy of analogies of long text with six levels of complexity – (i) word, (ii) word vs. sentence, (iii) syntactic, (iv) negation, (v) entailment, and (vi) metaphor. Using thirteen datasets and three different distance measures, we evaluate the abilities of eight LLMs in identifying analogical pairs in the semantic vector space (e.g., “I can speak two languages” should be closer to “I am bilingual” while “I like chocolate” and “I do not like chocolate” should be orthogonal). Our evaluation finds that it is increasingly challenging for LLMs to identify analogies when going up the analogy taxonomy
ANALOGICAL -- A New Benchmark for Analogy of Long Text for Large Language Models
Over the past decade, analogies, in the form of word-level analogies, have
played a significant role as an intrinsic measure of evaluating the quality of
word embedding methods such as word2vec. Modern large language models (LLMs),
however, are primarily evaluated on extrinsic measures based on benchmarks such
as GLUE and SuperGLUE, and there are only a few investigations on whether LLMs
can draw analogies between long texts. In this paper, we present ANALOGICAL, a
new benchmark to intrinsically evaluate LLMs across a taxonomy of analogies of
long text with six levels of complexity -- (i) word, (ii) word vs. sentence,
(iii) syntactic, (iv) negation, (v) entailment, and (vi) metaphor. Using
thirteen datasets and three different distance measures, we evaluate the
abilities of eight LLMs in identifying analogical pairs in the semantic vector
space. Our evaluation finds that it is increasingly challenging for LLMs to
identify analogies when going up the analogy taxonomy.Comment: Accepted as a long paper at Findings of ACL 202