7,425 research outputs found
Black-Box Analysis: GPTs Across Time in Legal Textual Entailment Task
The evolution of Generative Pre-trained Transformer (GPT) models has led to
significant advancements in various natural language processing applications,
particularly in legal textual entailment. We present an analysis of GPT-3.5
(ChatGPT) and GPT-4 performances on COLIEE Task 4 dataset, a prominent
benchmark in this domain. The study encompasses data from Heisei 18 (2006) to
Reiwa 3 (2021), exploring the models' abilities to discern entailment
relationships within Japanese statute law across different periods. Our
preliminary experimental results unveil intriguing insights into the models'
strengths and weaknesses in handling legal textual entailment tasks, as well as
the patterns observed in model performance. In the context of proprietary
models with undisclosed architectures and weights, black-box analysis becomes
crucial for evaluating their capabilities. We discuss the influence of training
data distribution and the implications on the models' generalizability. This
analysis serves as a foundation for future research, aiming to optimize
GPT-based models and enable their successful adoption in legal information
extraction and entailment applications.Comment: ISAILD@KSE 202
Defining Textual Entailment
Textual entailment is a relationship that obtains between fragments of text when one fragment in some sense implies the other fragment. The automation of textual entailment recognition supports a wide variety of text-based tasks, including information retrieval, information extraction, question answering, text summarization, and machine translation. Much ingenuity has been devoted to developing algorithms for identifying textual entailments, but relatively little to saying what textual entailment actually is. This article is a review of the logical and philosophical issues involved in providing an adequate definition of textual entailment. We show that many natural definitions of textual entailment are refuted by counterexamples, including the most widely cited definition of Dagan et al. We then articulate and defend the following revised definition: T textually entails H = df typically, a human reading T would be justified in inferring the proposition expressed by H from the proposition expressed by T. We also show that textual entailment is context-sensitive, nontransitive, and nonmonotonic
A Survey of Paraphrasing and Textual Entailment Methods
Paraphrasing methods recognize, generate, or extract phrases, sentences, or
longer natural language expressions that convey almost the same information.
Textual entailment methods, on the other hand, recognize, generate, or extract
pairs of natural language expressions, such that a human who reads (and trusts)
the first element of a pair would most likely infer that the other element is
also true. Paraphrasing can be seen as bidirectional textual entailment and
methods from the two areas are often similar. Both kinds of methods are useful,
at least in principle, in a wide range of natural language processing
applications, including question answering, summarization, text generation, and
machine translation. We summarize key ideas from the two areas by considering
in turn recognition, generation, and extraction methods, also pointing to
prominent articles and resources.Comment: Technical Report, Natural Language Processing Group, Department of
Informatics, Athens University of Economics and Business, Greece, 201
- …