2 research outputs found
ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews
Revising scientific papers based on peer feedback is a challenging task that
requires not only deep scientific knowledge and reasoning, but also the ability
to recognize the implicit requests in high-level feedback and to choose the
best of many possible ways to update the manuscript in response. We introduce
this task for large language models and release ARIES, a dataset of review
comments and their corresponding paper edits, to enable training and evaluating
models. We study two versions of the task: comment-edit alignment and edit
generation, and evaluate several baselines, including GPT-4. We find that
models struggle even to identify the edits that correspond to a comment,
especially in cases where the comment is phrased in an indirect way or where
the edit addresses the spirit of a comment but not the precise request. When
tasked with generating edits, GPT-4 often succeeds in addressing comments on a
surface level, but it rigidly follows the wording of the feedback rather than
the underlying intent, and includes fewer technical details than human-written
edits. We hope that our formalization, dataset, and analysis will form a
foundation for future work in this area.Comment: 11 pages, 2 figure
The Semantic Reader Project: Augmenting Scholarly Documents through AI-Powered Interactive Reading Interfaces
Scholarly publications are key to the transfer of knowledge from scholars to
others. However, research papers are information-dense, and as the volume of
the scientific literature grows, the need for new technology to support the
reading process grows. In contrast to the process of finding papers, which has
been transformed by Internet technology, the experience of reading research
papers has changed little in decades. The PDF format for sharing research
papers is widely used due to its portability, but it has significant downsides
including: static content, poor accessibility for low-vision readers, and
difficulty reading on mobile devices. This paper explores the question "Can
recent advances in AI and HCI power intelligent, interactive, and accessible
reading interfaces -- even for legacy PDFs?" We describe the Semantic Reader
Project, a collaborative effort across multiple institutions to explore
automatic creation of dynamic reading interfaces for research papers. Through
this project, we've developed ten research prototype interfaces and conducted
usability studies with more than 300 participants and real-world users showing
improved reading experiences for scholars. We've also released a production
reading interface for research papers that will incorporate the best features
as they mature. We structure this paper around challenges scholars and the
public face when reading research papers -- Discovery, Efficiency,
Comprehension, Synthesis, and Accessibility -- and present an overview of our
progress and remaining open challenges