Peer Review Analyze: A Novel Benchmark Resource for Computational Analysis of Peer Reviews

Peer Review is at the heart of scholarly communications and the cornerstone of scientific publishing. However, academia often criticizes the peer review system as non-transparent, biased, arbitrary, a flawed process at the heart of science, leading to researchers arguing with its reliability and quality. These problems could also be due to the lack of studies with the peer-review texts for various proprietary and confidentiality clauses. Peer review texts could serve as a rich source of Natural Language Processing (NLP) research on understanding the scholarly communication landscape, and thereby build systems towards mitigating those pertinent problems. In this work, we present a first of its kind multi-layered dataset of 1199 open peer review texts manually annotated at the sentence level (*17k sentences) across the four layers, viz. Paper Section Correspondence, Paper Aspect Category, Review Functionality, and Review Significance. Given a text written by the reviewer, we annotate: to which sections (e.g., Methodology, Experiments, etc.), what aspects (e.g., Originality/Novelty, Empirical/Theoretical Soundness, etc.) of the paper does the review text correspond to, what is the role played by the review text (e.g., appreciation, criticism, summary, etc.), and the importance of the review statement (major, minor, general) within the review. We also annotate the sentiment of the reviewer (positive, negative, neutral) for the first two layers to judge the reviewer’s perspective on the different sections and aspects of the paper. We further introduce four novel tasks with this dataset, which could serve as an indicator of the exhaustiveness of a peer review and can be a step towards the automatic judgment of review quality. We also present baseline experiments and results for the different tasks for further investigations. We believe our dataset would provide a benchmark experimental testbed for automated systems to leverage on current NLP state-of-the-art techniques to address different issues with peer review quality, thereby ushering increased transparency and trust on the holy grail of scientific research validation

Online Reviews in B2B Markets: A Qualitative Study of Underlying Motives

While online reviews in Business-to-Consumer (B2C) and Consumer-to Consumer (C2C) markets have reached an advanced state of maturity in both academia and practice, the study of the dynamics of online reviews in the B2B market is still in its early stages. For this market, there are numerous unanswered questions concerning online reviews. The growing number of B2B review platforms and reviews makes it increasingly important to better understand the heterogeneous motives for writing online reviews for a business partner. Structured by the scales of the Motivation Sources Inventory, the literature on online reviews in the B2C market, and specifically the motivation underpinning review writing, and the characteristics of B2B review platforms, a semi-structured interview protocol is derived and presented. This research-in-progress describes the concept and proposed next steps of a qualitative study aimed at identifying the underlying motives for writing B2B online reviews

Editorial : A five year retrospective

Non peer reviewedProo

Characterizing industry-academia collaborations in software engineering: evidence from 101 projects

Research collaboration between industry and academia supports improvement and innovation in industry and helps ensure the industrial relevance of academic research. However, many researchers and practitioners in the community believe that the level of joint industry-academia collaboration (IAC) projects in Software Engineering (SE) research is relatively low, creating a barrier between research and practice. The goal of the empirical study reported in this paper is to explore and characterize the state of IAC with respect to industrial needs, developed solutions, impacts of the projects and also a set of challenges, patterns and anti-patterns identified by a recent Systematic Literature Review (SLR) study. To address the above goal, we conducted an opinion survey among researchers and practitioners with respect to their experience in IAC. Our dataset includes 101 data points from IAC projects conducted in 21 different countries. Our findings include: (1) the most popular topics of the IAC projects, in the dataset, are: software testing, quality, process, and project managements; (2) over 90% of IAC projects result in at least one publication; (3) almost 50% of IACs are initiated by industry, busting the myth that industry tends to avoid IACs; and (4) 61% of the IAC projects report having a positive impact on their industrial context, while 31% report no noticeable impacts or were “not sure”. To improve this situation, we present evidence-based recommendations to increase the success of IAC projects, such as the importance of testing pilot solutions before using them in industry. This study aims to contribute to the body of evidence in the area of IAC, and benefit researchers and practitioners. Using the data and evidence presented in this paper, they can conduct more successful IAC projects in SE by being aware of the challenges and how to overcome them, by applying best practices (patterns), and by preventing anti-patterns.The authors would like to thank the researchers and practitioners who participated in this survey. João M. Fernandes was supported by FCT (Fundação para a Ciência e Tecnologia) within the Project Scope UID/CEC/00319/2013. Dietmar Pfahl was supported by the institutional research grant IUT20-55 of the Estonian Research Council. Andrea Arcuri was supported by the Research Council of Norway (grant agreement No 274385). Mika Mäntylä was partially supported by Academy of Finland grant and ITEA3 / TEKES grant

The lab management practices of “Research Exemplars” that foster research rigor and regulatory compliance: A qualitative study of successful principal investigators

IntroductionConducting rigorous scientific inquiry within the bounds of research regulation and acceptable practice requires a principal investigator to lead and manage research processes and personnel. This study explores the practices used by investigators nominated as exemplars of research excellence and integrity to produce rigorous, reproducible research and comply with research regulations.MethodsUsing a qualitative research design, we interviewed 52 principal investigators working in the United States at top research universities and the National Institutes of Health Intramural Research Program. We solicited nominations of researchers meeting two criteria: (1) they are federally-funded researchers doing high-quality, high-impact research, and (2) have reputations for professionalism and integrity. Each investigator received an initial nomination addressing both criteria and at least one additional endorsement corroborating criteria 2. A panel of researchers and our research team reviewed the nominations to select finalists who were invited to participate. The cohort of "Research Exemplars" includes highly accomplished researchers in diverse scientific disciplines. The semi-structured interview questions asked them to describe the routine practices they employ to foster rigor and regulatory compliance. We used inductive thematic analysis to identify common practices.ResultsThe exemplars identified a core set of 8 practices and provided strategies for employing them. The practices included holding regular team meetings, encouraging shared ownership, providing supervision, ensuring adequate training, fostering positive attitudes about compliance, scrutinizing data and findings, and following standard operating procedures. Above all, the use of these practices aim to create a psychologically safe work environment in which lab members openly collaborate to scrutinize their work and share in accountability for rigorous, compliant research.ConclusionsResearchers typically receive limited systematic training in how to lead and manage their research teams. Training and education for principal investigators should include essential leadership and management practices and strategies that support doing high-quality research with integrity

Editorial for the special issue: “Novel Solutions or Novel Approaches in Operational Research”

This special issue of Business Systems Research (SI of BSR) is co-published by the Slovenian Society INFORMATIKA – Section for Operational Research (SSI – SOR) and highlights recent advances in operations research and management science (OR/MS), with a focus on linking OR/MS with other areas of quantitative and qualitative methods in a multidisciplinary framework. Nine papers that have been selected for this SI of BSR present improvements and new techniques (methodology) in operations research (OR) and their use in various fields of business, economics, spatial science and location