10 research outputs found
Evaluating ChatGPT as a Question Answering System: A Comprehensive Analysis and Comparison with Existing Models
In the current era, a multitude of language models has emerged to cater to
user inquiries. Notably, the GPT-3.5 Turbo language model has gained
substantial attention as the underlying technology for ChatGPT. Leveraging
extensive parameters, this model adeptly responds to a wide range of questions.
However, due to its reliance on internal knowledge, the accuracy of responses
may not be absolute. This article scrutinizes ChatGPT as a Question Answering
System (QAS), comparing its performance to other existing QASs. The primary
focus is on evaluating ChatGPT's proficiency in extracting responses from
provided paragraphs, a core QAS capability. Additionally, performance
comparisons are made in scenarios without a surrounding passage. Multiple
experiments, exploring response hallucination and considering question
complexity, were conducted on ChatGPT. Evaluation employed well-known Question
Answering (QA) datasets, including SQuAD, NewsQA, and PersianQuAD, across
English and Persian languages. Metrics such as F-score, exact match, and
accuracy were employed in the assessment. The study reveals that, while ChatGPT
demonstrates competence as a generative model, it is less effective in question
answering compared to task-specific models. Providing context improves its
performance, and prompt engineering enhances precision, particularly for
questions lacking explicit answers in provided paragraphs. ChatGPT excels at
simpler factual questions compared to "how" and "why" question types. The
evaluation highlights occurrences of hallucinations, where ChatGPT provides
responses to questions without available answers in the provided context.Comment: 15 pages, 7 figure
Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction
A relation tuple consists of two entities and the relation between them, and
often such tuples are found in unstructured text. There may be multiple
relation tuples present in a text and they may share one or both entities among
them. Extracting such relation tuples from a sentence is a difficult task and
sharing of entities or overlapping entities among the tuples makes it more
challenging. Most prior work adopted a pipeline approach where entities were
identified first followed by finding the relations among them, thus missing the
interaction among the relation tuples in a sentence. In this paper, we propose
two approaches to use encoder-decoder architecture for jointly extracting
entities and relations. In the first approach, we propose a representation
scheme for relation tuples which enables the decoder to generate one word at a
time like machine translation models and still finds all the tuples present in
a sentence with full entity names of different length and with overlapping
entities. Next, we propose a pointer network-based decoding approach where an
entire tuple is generated at every time step. Experiments on the publicly
available New York Times corpus show that our proposed approaches outperform
previous work and achieve significantly higher F1 scores.Comment: Accepted at AAAI 202