Towards a research agenda for promoting responsible research practices

This opinion piece aims to inform future research funding programs on responsible research practices (RRP) based on three specific objectives: (1) to give a sketch of the current international discussion on responsible research practices (RRPs); (2) to give an overview of current initiatives and already obtained results regarding RRP; and (3) to give an overview of potential future needs for research on RRP. In this opinion piece, we have used seven iterative methodological steps (including literature review, ranking, and sorting exercises) to create the proposed research agenda. We identified six main themes that we believe need attention in future research: (1) responsible evaluation of research and researchers, (2) the influence of open science and transparency on RRP, (3) research on responsible mentoring, supervision, and role modeling, (4) the effect of education and training on RRP, (5) checking for reproducibility, and (6) responsible and fair peer review. These themes have in common that they address aspects of research that are mostly on the level of the scientific system, more than on the level of the individual researcher. Some current initiatives are already gathering substantial empirical evidence to start filling these gaps. We believe that with sufficient support from all relevant stakeholders, more progress can be made

Costs of managing adverse events in the treatment of first-line metastatic renal cell carcinoma: Bevacizumab in combination with interferon-α2a compared with sunitinib

Background: Bevacizumab plus interferon-α2a (IFN) prolongs progression-free survival to>10 months, which is comparable with sunitinib as first-line treatment of metastatic renal cell carcinoma (RCC). The two regimens have different tolerability profiles; therefore, costs for managing adverse events may be an important factor in selecting therapy.Methods: Costs of managing adverse events affecting patients with metastatic RCC eligible for treatment with bevacizumab plus IFN or sunitinib were evaluated using a linear decision analytical model. Management costs were calculated from the published incidence of adverse events and health-care costs for treating adverse events in the United Kingdom, Germany, France and Italy.Results: Adverse event management costs were higher for sunitinib than for bevacizumab plus IFN. The average cost per patient for the management of grade 3-4 adverse events was markedly lower with bevacizumab plus IFN compared with sunitinib in the United Kingdom (\[euro]1475 vs \[euro]804), Germany (\[euro]1785 vs \[euro]1367), France (\[euro]2590 vs \[euro]1618) and Italy (\[euro]891 vs \[euro]402). The main cost drivers were lymphopaenia, neutropaenia, thrombocytopaenia, leucopaenia and fatigue/asthaenia for sunitinib; and proteinuria, fatigue/asthaenia, bleeding, anaemia and gastrointestinal perforation for bevacizumab plus IFN.Conclusion: The costs of managing adverse events are lower for bevacizumab plus IFN than for sunitinib. The potential for cost savings should be considered when selecting treatments for RCC

The validity of the tool “statcheck” in discovering statistical reporting inconsistencies

The R package “statcheck” (Epskamp & Nuijten, 2016) is a tool to extract statistical results from articles and check whether the reported p-value matches the accompanying test statistic and degrees of freedom. A previous study showed high interrater reliabilities (between .76 and .89) between statcheck and manual coding of inconsistencies (.76 - .89; Nuijten, Hartgerink, Van Assen, Epskamp, & Wicherts, 2016). Here we present an additional, detailed study of the validity of statcheck. In Study 1, we calculated its sensitivity and specificity. We found that statcheck’s sensitivity (true positive rate) and specificity (true negative rate) were high: between 85.3% and 100%, and between 96.0% and 100%, respectively, depending on the assumptions and settings. The overall accuracy of statcheck ranged from 96.2% to 99.9%. In Study 2, we investigated statcheck’s ability to deal with statistical corrections for multiple testing or violations of assumptions in articles. We found that the prevalence of corrections for multiple testing or violations of assumptions in psychology was higher than we initially estimated in Nuijten et al. (2016). Although we found numerous reporting inconsistencies in results corrected for violations of the sphericity assumption, we demonstrate that inconsistencies associated with statistical corrections are not what is causing the high estimates of the prevalence of statistical reporting inconsistencies in psychology

The dire disregard of measurement invariance testing in psychological science

In psychological science, self-report scales are widely used to compare means in targeted latent constructs across time points, groups, or experimental conditions. For these scale mean comparisons (SMC) to be meaningful and unbiased, the scales should be measurement invariant across the compared time points or (experimental) groups. Measurement invariance (MI) testing checks whether the latent constructs are measured equivalently across groups or time points. Since MI is essential for meaningful comparisons, we conducted a systematic review to check whether MI is taken seriously in psychological research. Specifically, we sampled 426 psychology articles with openly available data that involved a total of 918 SMCs to (1) investigate common practices in conducting and reporting of MI testing, (2) check whether reported MI test results can be reproduced, and (3) conduct MI tests for the SMCs that enabled sufficiently powerful MI testing with the shared data. Our results indicate that (1) 4% of the 918 scales underwent MI testing across groups or time and that these tests were generally poorly reported, (2) none of the reported MI tests could be successfully reproduced, and (3) of 161 newly performed MI tests, a mere 46 (29%) reached sufficient MI (scalar invariance), and MI often failed completely (89; 55%). Thus, MI tests were rarely done and poorly reported in psychological studies, and the frequent violations of MI indicate that reported group differences cannot be solely attributed to group differences in the latent constructs. We offer recommendations on reporting MI tests and improving computational reproducibility practices