Reviewing Natural Language Processing Research

International audienceThis tutorial will cover the goals, processes, and evaluation of reviewing research in natural language processing. As has been pointed out for years by leading figures in our community (Web-ber, 2007), researchers in the ACL community face a heavy-and growing-reviewing burden. Initiatives to lower this burden have been discussed at the recent ACL general assembly in Florence (ACL 2019) 1. Simultaneously, notable "false negatives"-rejection by our conferences of work that was later shown to be tremendously important after acceptance by other conferences (Church, 2005)-has raised awareness of the fact that our reviewing practices leave something to be desired.. . and we do not often talk about "false positives" with respect to conference papers, but conversations in the hallways at *ACL meetings suggest that we have a publication bias towards papers that report high performance, with perhaps not much else of interest in them (Manning, 2015). It need not be this way. There is good reason to think that reviewing is a learnable (and teachable)

The Other Reviewer: RoboReviewer

The peer review process is a mainstay for informing publication decisions at many journals and conferences. It has several strengths that are well-accepted, such as providing a signal about the quality of published papers. Nonetheless, it has several limitations that have been documented extensively, such as reviewer biases affecting paper appraisals. To date, attempts to mitigate these limitations have had limited success. Accordingly, I consider how developments in artificial intelligence technologies—in particular, pretrained large language models with downstream fine-tuning—might be used to automate peer reviews. I discuss several challenges that are likely to arise if these systems are built and deployed and some ways to address these challenges. If the systems are deemed successful, I describe some characteristics of a highly competitive, lucrative marketplace for these systems that is likely to emerge. I discuss some ramifications of such a marketplace for authors, reviewers, editors, conference chairs, conference program committees, publishers, and the peer review process

Design science research in PhD education: designing for assistance tools

170 p.scholars. Major conferences have tracks dedicated to DSR, and even leading journals have publishedspecial issues on it. In line with this momentum, DSR has also gained acceptance among PhD students.Indeed, DSR is well regarded for its ability to bring together theoretical and practical knowledge,addressing both rigor and relevance. But in exchange, DSR calls for high levels of commitment andmaturity. PhD students, as they are transitioning towards becoming independent researchers, usually lacksuch maturity. On top of that, the lack of widely accepted software tools for conducting DSR does nothelp.This Thesis is aimed at providing PhD students with tool support for carrying out DSR. To thatend, we focus on problematic situations related to four basic activities conducted throughout thedoctorate: inquiry, reading, writing and peer review. For each of these problems, a purposeful artifact isdesigned, developed and evaluated with real stakeholders. The outcome: DScaffolding and Review&Go,two browser extensions for Google Chrome currently in use by practitioners