Peer Review of Reviewers: The Author's Perspective

The aim of this study was to investigate the opinion of authors on the overall quality and effectiveness of reviewers' contributions to reviewed papers. We employed an on-line survey of thirteen journals which publish articles in the field of life, social or technological sciences. Responses received from 193 authors were analysed using a mixed-effects model in order to determine factors deemed the most important in the authors' evaluation of the reviewers. Qualitative content analysis of the responses to open questions was performed as well. The mixed-effects model revealed that the authors' assessment of the competence of referees strongly depended on the final editorial decision and that the speed of the review process was influential as well. In Ordinary Least Squares (OLS) analysis on seven questions detailing authors' opinions, perception of review speed remained a significant predictor of the assessment. In addition, both the perceived competence and helpfulness of the reviewers significantly and positively affected the authors' evaluation. New models were used to re-check the value of these two factors and it was confirmed that the assessment of the competence of reviewers strongly depended on the final editorial decision

Registered Reports in Software Engineering

Registered reports are scientific publications which begin the publication process by first having the detailed research protocol, including key research questions, reviewed and approved by peers. Subsequent analysis and results are published with minimal additional review, even if there was no clear support for the underlying hypothesis, as long as the approved protocol is followed. Registered reports can prevent several questionable research practices and give early feedback on research designs. In software engineering research, registered reports were first introduced in the International Conference on Mining Software Repositories (MSR) in 2020. They are now established in three conferences and two pre-eminent journals, including Empirical Software Engineering. We explain the motivation for registered reports, outline the way they have been implemented in software engineering, and outline some ongoing challenges for addressing high quality software engineering research.Comment: in press as EMSE J. commen

Methodological criteria for the assessment of moderators in systematic reviews of randomised controlled trials : a consensus study

Background: Current methodological guidelines provide advice about the assessment of sub-group analysis within RCTs, but do not specify explicit criteria for assessment. Our objective was to provide researchers with a set of criteria that will facilitate the grading of evidence for moderators, in systematic reviews. Method: We developed a set of criteria from methodological manuscripts (n = 18) using snowballing technique, and electronic database searches. Criteria were reviewed by an international Delphi panel (n = 21), comprising authors who have published methodological papers in this area, and researchers who have been active in the study of sub-group analysis in RCTs. We used the Research ANd Development/University of California Los Angeles appropriateness method to assess consensus on the quantitative data. Free responses were coded for consensus and disagreement. In a subsequent round additional criteria were extracted from the Cochrane Reviewers’ Handbook, and the process was repeated. Results: The recommendations are that meta-analysts report both confirmatory and exploratory findings for subgroups analysis. Confirmatory findings must only come from studies in which a specific theory/evidence based apriori statement is made. Exploratory findings may be used to inform future/subsequent trials. However, for inclusion in the meta-analysis of moderators, the following additional criteria should be applied to each study: Baseline factors should be measured prior to randomisation, measurement of baseline factors should be of adequate reliability and validity, and a specific test of the interaction between baseline factors and interventions must be presented. Conclusions: There is consensus from a group of 21 international experts that methodological criteria to assess moderators within systematic reviews of RCTs is both timely and necessary. The consensus from the experts resulted in five criteria divided into two groups when synthesising evidence: confirmatory findings to support hypotheses about moderators and exploratory findings to inform future research. These recommendations are discussed in reference to previous recommendations for evaluating and reporting moderator studies

MINORITY STRESS: DOES SEXUAL ORIENTATION REALLY MATTER ?

The objective of this study was to compare the degree of work stress experienced by heterosexual and non-heterosexual men. For this, a survey was conducted with 225 white male executive employees at different companies. Sexual orientation was self-declared which made possible to isolate two groups of respondents: 104 of whom defined their sexual orientation as heterosexual, and 121 as non-heterosexual. A work stress scale was applied. Results indicated that there was no statistically significant difference between the work stress experienced by the two groups. The analysis developed indicated that for groups that are like the sample used, being heterosexual or non-heterosexual is not enough to determine the degree of stress that one can suffer.DOI: 10.21714/2178-8030gep.v.21.6194El objetivo de este estudio fue comparar el grado de estrés laboral experimentado por hombres heterosexuales y no heterosexuales. Para esto, se realizó una encuesta con 225 empleados ejecutivos blancos de diferentes compañías. La orientación sexual fue auto declarada, lo que permitió aislar a dos grupos de encuestados: 104 de ellos definieron su orientación como heterosexual y 121 como no heterosexual. Se aplicó una escala de estrés laboral. Los resultados no indicaron diferencias estadísticamente significativas entre el estrés laboral experimentado por ambos grupos. El análisis desarrollado indicó que, para grupos similares a la muestra utilizada, ser heterosexual o no heterosexual no es suficiente para determinar el grado de estrés que uno puede sufrir.O objetivo deste estudo foi comparar o grau de estresse no trabalho vivenciado por homens heterossexuais e não heterossexuais. Para isso foi realizada pesquisa com 225 funcionários executivos, homens, brancos, de diferentes empresas. A orientação sexual foi autodeclarada, o que tornou possível isolar dois grupos de respondentes: 104 deles definiram sua orientação como heterossexual e 121 como não heterossexual.  Uma escala de estresse no trabalho foi aplicada. Os resultados indicaram não haver diferença estatisticamente significante entre o estresse no trabalho vivenciado pelos dois grupos. A análise desenvolvida indicou que para grupos similares ao da amostra usada, ser heterossexual ou não heterossexual não é suficiente para determinar o grau de stress que alguém pode sofrer.DOI: 10.21714/2178-8030gep.v.21.619

Open science and modified funding lotteries can impede the natural selection of bad science.

Assessing scientists using exploitable metrics can lead to the degradation of research methods even without any strategic behaviour on the part of individuals, via 'the natural selection of bad science.' Institutional incentives to maximize metrics like publication quantity and impact drive this dynamic. Removing these incentives is necessary, but institutional change is slow. However, recent developments suggest possible solutions with more rapid onsets. These include what we call open science improvements, which can reduce publication bias and improve the efficacy of peer review. In addition, there have been increasing calls for funders to move away from prestige- or innovation-based approaches in favour of lotteries. We investigated whether such changes are likely to improve the reproducibility of science even in the presence of persistent incentives for publication quantity through computational modelling. We found that modified lotteries, which allocate funding randomly among proposals that pass a threshold for methodological rigour, effectively reduce the rate of false discoveries, particularly when paired with open science improvements that increase the publication of negative results and improve the quality of peer review. In the absence of funding that targets rigour, open science improvements can still reduce false discoveries in the published literature but are less likely to improve the overall culture of research practices that underlie those publications

Key issues on partial least squares (PLS) in operations management research: A guide to submissions

Purpose: This work aims to systematise the use of PLS as an analysis tool via a usage guide or recommendation for researchers to help them eliminate errors when using this tool. Design/methodology/approach: A recent literature review about PLS and discussion with experts in the methodology. Findings: This article considers the current situation of PLS after intense academic debate in recent years, and summarises recommendations to properly conduct and report a research work that uses this methodology in its analyses. We particularly focus on how to: choose the construct type; choose the estimation technique (PLS or CB-SEM); evaluate and report the measurement model; evaluate and report the structural model; analyse statistical power. Research limitations: It was impossible to cover some relevant aspects in considerable detail herein: presenting a guided example that respects all the report recommendations presented herein to act as a practical guide for authors; does the specification or evaluation of the measurement model differ when it deals with first-order or second-order constructs?; how are the outcomes of the constructs interpreted with the indicators being measured with nominal measurement levels?; is the Confirmatory Composite Analysis approach compatible with recent proposals about the Confirmatory Tetrad Analysis (CTA)? These themes will the object of later publications. Originality/value: We provide a check list of the information elements that must contain any article using PLS. Our intention is for the article to act as a guide for the researchers and possible authors who send works to the JIEM (Journal of Industrial and Engineering Management). This guide could be used by both editors and reviewers of JIEM, or other journals in this area, to evaluate and reduce the risk of bias (Losilla, Oliveras, Marin-Garcia & Vives, 2018) in works using PLS as an analysis procedure

Accuracy of medical billing data against the electronic health record in the measurement of colorectal cancer screening rates.

ObjectiveMedical billing data are an attractive source of secondary analysis because of their ease of use and potential to answer population-health questions with statistical power. Although these datasets have known susceptibilities to biases, the degree to which they can distort the assessment of quality measures such as colorectal cancer screening rates are not widely appreciated, nor are their causes and possible solutions.MethodsUsing a billing code database derived from our institution's electronic health records, we estimated the colorectal cancer screening rate of average-risk patients aged 50-74 years seen in primary care or gastroenterology clinic in 2016-2017. 200 records (150 unscreened, 50 screened) were sampled to quantify the accuracy against manual review.ResultsOut of 4611 patients, an analysis of billing data suggested a 61% screening rate, an estimate that matches the estimate by the Centers for Disease Control. Manual review revealed a positive predictive value of 96% (86%-100%), negative predictive value of 21% (15%-29%) and a corrected screening rate of 85% (81%-90%). Most false negatives occurred due to examinations performed outside the scope of the database-both within and outside of our institution-but 21% of false negatives fell within the database's scope. False positives occurred due to incomplete examinations and inadequate bowel preparation. Reasons for screening failure include ordered but incomplete examinations (48%), lack of or incorrect documentation by primary care (29%) including incorrect screening intervals (13%) and patients declining screening (13%).ConclusionsBilling databases are prone to substantial bias that may go undetected even in the presence of confirmatory external estimates. Caution is recommended when performing population-level inference from these data. We propose several solutions to improve the use of these data for the assessment of healthcare quality

Degrees of freedom in planning, running, analyzing, and reporting psychological studies:A checklist to avoid p-hacking

The designing, collecting, analyzing, and reporting of psychological studies entail many choices that are often arbitrary. The opportunistic use of these so-called researcher degrees of freedom aimed at obtaining statistically significant results is problematic because it enhances the chances of false positive results and may inflate effect size estimates. In this review article, we present an extensive list of 34 degrees of freedom that researchers have in formulating hypotheses, and in designing, running, analyzing, and reporting of psychological research. The list can be used in research methods education, and as a checklist to assess the quality of preregistrations and to determine the potential for bias due to (arbitrary) choices in unregistered studies

Is Law a Discipline? Forays into Academic Culture

This Article explores academic culture. It addresses the reluctance in academic circles to accord law the full stature of a discipline. It forms doubts that have been raised into a series of four criticisms. Each attacks an academic feature of law, inviting the question: Is law different from the rest of the university in a way damaging its stature as an academic discipline? The Article concludes that, upon careful examination of each criticism, none establishes a difference between law and other disciplines capable of damaging law’s stature

Is Law a Discipline? Forays into Academic Culture

This Article explores academic culture. It addresses the reluctance in academic circles to accord law the full stature of a discipline. It forms doubts that have been raised into a series of four criticisms. Each attacks an academic feature of law, inviting the question: Is law different from the rest of the university in a way damaging its stature as an academic discipline? The Article concludes that, upon careful examination of each criticism, none establishes a difference between law and other disciplines capable of damaging law’s stature