3,554 research outputs found
CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning
To accelerate software development, much research has been performed to help
people understand and reuse the huge amount of available code resources. Two
important tasks have been widely studied: code retrieval, which aims to
retrieve code snippets relevant to a given natural language query from a code
base, and code annotation, where the goal is to annotate a code snippet with a
natural language description. Despite their advancement in recent years, the
two tasks are mostly explored separately. In this work, we investigate a novel
perspective of Code annotation for Code retrieval (hence called `CoaCor'),
where a code annotation model is trained to generate a natural language
annotation that can represent the semantic meaning of a given code snippet and
can be leveraged by a code retrieval model to better distinguish relevant code
snippets from others. To this end, we propose an effective framework based on
reinforcement learning, which explicitly encourages the code annotation model
to generate annotations that can be used for the retrieval task. Through
extensive experiments, we show that code annotations generated by our framework
are much more detailed and more useful for code retrieval, and they can further
improve the performance of existing code retrieval models significantly.Comment: 10 pages, 2 figures. Accepted by The Web Conference (WWW) 201
Leveraging 2-hop Distant Supervision from Table Entity Pairs for Relation Extraction
Distant supervision (DS) has been widely used to automatically construct
(noisy) labeled data for relation extraction (RE). Given two entities, distant
supervision exploits sentences that directly mention them for predicting their
semantic relation. We refer to this strategy as 1-hop DS, which unfortunately
may not work well for long-tail entities with few supporting sentences. In this
paper, we introduce a new strategy named 2-hop DS to enhance distantly
supervised RE, based on the observation that there exist a large number of
relational tables on the Web which contain entity pairs that share common
relations. We refer to such entity pairs as anchors for each other, and collect
all sentences that mention the anchor entity pairs of a given target entity
pair to help relation prediction. We develop a new neural RE method REDS2 in
the multi-instance learning paradigm, which adopts a hierarchical model
structure to fuse information respectively from 1-hop DS and 2-hop DS.
Extensive experimental results on a benchmark dataset show that REDS2 can
consistently outperform various baselines across different settings by a
substantial margin
Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
Large language models (LLMs) such as ChatGPT and GPT-4 have shown impressive
performance in complex reasoning tasks. However, it is difficult to know
whether the models are reasoning based on deep understandings of truth and
logic, or leveraging their memorized patterns in a relatively superficial way.
In this work, we explore testing LLMs' reasoning by engaging with them in a
debate-like conversation, where given a question, the LLM and the user need to
discuss to make the correct decision starting from opposing arguments. Upon
mitigating the Clever Hans effect, our task requires the LLM to not only
achieve the correct answer on its own, but also be able to hold and defend its
belief instead of blindly believing or getting misled by the user's (invalid)
arguments and critiques, thus testing in greater depth whether the LLM grasps
the essence of the reasoning required to solve the problem. Across a range of
complex reasoning benchmarks spanning math, commonsense, logic and BIG-Bench
tasks, we find that despite their impressive performance as reported in
existing work on generating correct step-by-step solutions in the beginning,
LLMs like ChatGPT cannot maintain their beliefs in truth for a significant
portion of examples when challenged by oftentimes absurdly invalid arguments.
Our work points to danger zones of model alignment, and also suggests more
careful treatments and interpretations of the recent findings that LLMs can
improve their responses based on feedback.Comment: EMNLP-23 (findings
- …