Computational support for academic peer review:a perspective from artificial intelligence

New tools tackle an age-old practice.</jats:p

STARC: Structured Annotations for Reading Comprehension

We present STARC (Structured Annotations for Reading Comprehension), a new annotation framework for assessing reading comprehension with multiple choice questions. Our framework introduces a principled structure for the answer choices and ties them to textual span annotations. The framework is implemented in OneStopQA, a new high-quality dataset for evaluation and analysis of reading comprehension in English. We use this dataset to demonstrate that STARC can be leveraged for a key new application for the development of SAT-like reading comprehension materials: automatic annotation quality probing via span ablation experiments. We further show that it enables in-depth analyses and comparisons between machine and human reading comprehension behavior, including error distributions and guessing ability. Our experiments also reveal that the standard multiple choice dataset in NLP, RACE, is limited in its ability to measure reading comprehension. 47% of its questions can be guessed by machines without accessing the passage, and 18% are unanimously judged by humans as not having a unique correct answer. OneStopQA provides an alternative test set for reading comprehension which alleviates these shortcomings and has a substantially higher human ceiling performance.Comment: ACL 2020. OneStopQA dataset, STARC guidelines and human experiments data are available at https://github.com/berzak/onestop-q

Submission-Aware Reviewer Profiling for Reviewer Recommender System

Assigning qualified, unbiased and interested reviewers to paper submissions is vital for maintaining the integrity and quality of the academic publishing system and providing valuable reviews to authors. However, matching thousands of submissions with thousands of potential reviewers within a limited time is a daunting challenge for a conference program committee. Prior efforts based on topic modeling have suffered from losing the specific context that help define the topics in a publication or submission abstract. Moreover, in some cases, topics identified are difficult to interpret. We propose an approach that learns from each abstract published by a potential reviewer the topics studied and the explicit context in which the reviewer studied the topics. Furthermore, we contribute a new dataset for evaluating reviewer matching systems. Our experiments show a significant, consistent improvement in precision when compared with the existing methods. We also use examples to demonstrate why our recommendations are more explainable. The new approach has been deployed successfully at top-tier conferences in the last two years

Robust filtering for bilinear uncertain stochastic discrete-time systems

Copyright [2002] IEEE. This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of Brunel University's products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to [email protected]. By choosing to view this document, you agree to all provisions of the copyright laws protecting it.This paper deals with the robust filtering problem for uncertain bilinear stochastic discrete-time systems with estimation error variance constraints. The uncertainties are allowed to be norm-bounded and enter into both the state and measurement matrices. We focus on the design of linear filters, such that for all admissible parameter uncertainties, the error state of the bilinear stochastic system is mean square bounded, and the steady-state variance of the estimation error of each state is not more than the individual prespecified value. It is shown that the design of the robust filters can be carried out by solving some algebraic quadratic matrix inequalities. In particular, we establish both the existence conditions and the explicit expression of desired robust filters. A numerical example is included to show the applicability of the present method

THE NATURE OF FEEDBACK:HOW DIFFERENT TYPES OF PEER FEEDBACK AFFECT WRITING PERFORMANCE

Although providing feedback is commonly practiced in education, there is general agreement regarding what type of feedback is most helpful and why it is helpful. This study examined the relationship between various types of feedback, potential internal mediators, and the likelihood of implementing feedback. Five main predictions were developed from the feedback literature in writing, specifically regarding feedback features (summarization, identifying problems, providing solutions, localization, explanations, scope, praise, and mitigating language) as they relate to potential causal mediators of problem or solution understand and problem or solution agreement, leading to the final outcome of feedback implementation.To empirically test the proposed feedback model, 1073 feedback segments from writing assessed by peers was analyzed. Feedback was collected using SWoRD, an online peer review system. Each segment was coded for each of the feedback features, implementation, agreement, and understanding. The correlations between the feedback features, levels of mediating variables, and implementation rates revealed several significant relationships. Understanding was the only significant mediator of implementation. Several feedback features were associated with understanding: including solutions, a summary of the performance, and the location of the problem were associated with increased understanding; and explanations to problems were associated with decreased understanding. Implications of these results are discussed

A Democracy of Children’s Literature Critics? The Opportunities and Risks of Paying Attention to Open Reviews and Mass Discussion

Drawing on the outputs of a wider democracy of online reviewers presents the academic study of children’s literature with opportunities and challenges, and can enhance critical discussion. As it is now easy to locate a large number of online reviews, it is argued that children’s literature studies needs to make room for a wider range of critical voices. This article reports on the work of two cohorts of over a thousand students. Each cohort, in consecutive years, researched online reviews as part of their studies in contemporary children’s literature on a one year part-time module at a distance learning university. Despite the perceived lack of status of non-academic, non-professional critiques, students’ and tutors’ experiences of these tasks showed the value of researching online reviews. This work also allowed for alternative forms of writing and assessment alongside more conventional academic essays, and encouraged students to develop their skills of critical digital literacy. Module leaders recommended basic initial research methods for student use, but for more extensive or larger scale research it will be important to address methodological issues and understand how online reviewer communities operate. Such changes in approaches to teaching and learning also need to take into account the issues surrounding social media usage, ownership and control