2 research outputs found
Linguistic representations for fewer-shot relation extraction across domains
Recent work has demonstrated the positive impact of incorporating linguistic
representations as additional context and scaffolding on the in-domain
performance of several NLP tasks. We extend this work by exploring the impact
of linguistic representations on cross-domain performance in a few-shot
transfer setting. An important question is whether linguistic representations
enhance generalizability by providing features that function as cross-domain
pivots. We focus on the task of relation extraction on three datasets of
procedural text in two domains, cooking and materials science. Our approach
augments a popular transformer-based architecture by alternately incorporating
syntactic and semantic graphs constructed by freely available off-the-shelf
tools. We examine their utility for enhancing generalization, and investigate
whether earlier findings, e.g. that semantic representations can be more
helpful than syntactic ones, extend to relation extraction in multiple domains.
We find that while the inclusion of these graphs results in significantly
higher performance in few-shot transfer, both types of graph exhibit roughly
equivalent utility.Comment: ACL 202
LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs
LLMs have shown promise in replicating human-like behavior in crowdsourcing
tasks that were previously thought to be exclusive to human abilities. However,
current efforts focus mainly on simple atomic tasks. We explore whether LLMs
can replicate more complex crowdsourcing pipelines. We find that modern LLMs
can simulate some of crowdworkers' abilities in these "human computation
algorithms," but the level of success is variable and influenced by requesters'
understanding of LLM capabilities, the specific skills required for sub-tasks,
and the optimal interaction modality for performing these sub-tasks. We reflect
on human and LLMs' different sensitivities to instructions, stress the
importance of enabling human-facing safeguards for LLMs, and discuss the
potential of training humans and LLMs with complementary skill sets. Crucially,
we show that replicating crowdsourcing pipelines offers a valuable platform to
investigate (1) the relative strengths of LLMs on different tasks (by
cross-comparing their performances on sub-tasks) and (2) LLMs' potential in
complex tasks, where they can complete part of the tasks while leaving others
to humans