Reproducibility in Machine Learning-Driven Research

Research is facing a reproducibility crisis, in which the results and findings of many studies are difficult or even impossible to reproduce. This is also the case in machine learning (ML) and artificial intelligence (AI) research. Often, this is the case due to unpublished data and/or source-code, and due to sensitivity to ML training conditions. Although different solutions to address this issue are discussed in the research community such as using ML platforms, the level of reproducibility in ML-driven research is not increasing substantially. Therefore, in this mini survey, we review the literature on reproducibility in ML-driven research with three main aims: (i) reflect on the current situation of ML reproducibility in various research fields, (ii) identify reproducibility issues and barriers that exist in these research fields applying ML, and (iii) identify potential drivers such as tools, practices, and interventions that support ML reproducibility. With this, we hope to contribute to decisions on the viability of different solutions for supporting ML reproducibility.Comment: This research is supported by the Horizon Europe project TIER2 under grant agreement No 10109481

OPUCEM: A Library with Error Checking Mechanism for Computing Oblique Parameters

After a brief review of the electroweak radiative corrections to gauge-boson self-energies, otherwise known as the direct and oblique corrections, a tool for calculation of the oblique parameters is presented. This tool, named OPUCEM, brings together formulas from multiple physics models and provides an error-checking machinery to improve reliability of numerical results. It also sets a novel example for an "open-formula" concept, which is an attempt to improve the reliability and reproducibility of computations in scientific publications by encouraging the authors to open-source their numerical calculation programs. Finally, we demonstrate the use of OPUCEM in two detailed case studies related to the fourth Standard Model family. The first is a generic fourth family study to find relations between the parameters compatible with the EW precision data and the second is the particular study of the Flavor Democracy predictions for both Dirac and Majorana-type neutrinos.Comment: 10 pages, 19 figures, section 3 and 4 reviewed, results unchanged, typo correction

Sharing Social Network Data: Differentially Private Estimation of Exponential-Family Random Graph Models

Motivated by a real-life problem of sharing social network data that contain sensitive personal information, we propose a novel approach to release and analyze synthetic graphs in order to protect privacy of individual relationships captured by the social network while maintaining the validity of statistical results. A case study using a version of the Enron e-mail corpus dataset demonstrates the application and usefulness of the proposed techniques in solving the challenging problem of maintaining privacy \emph{and} supporting open access to network data to ensure reproducibility of existing studies and discovering new scientific insights that can be obtained by analyzing such data. We use a simple yet effective randomized response mechanism to generate synthetic networks under $\epsilon$-edge differential privacy, and then use likelihood based inference for missing data and Markov chain Monte Carlo techniques to fit exponential-family random graph models to the generated synthetic networks.Comment: Updated, 39 page

Methodological shortcomings of bibliometric papers published in the journal Sustainability (2019-2020)

Using Sustainability (the journal outside the IS area that publishes the most bibliometric articles) as a case study, this study uses content analysis to determine various parameters relating to the methodological rigour and reproducibility of the papers published in this journal in 2019 and 2020. In particular, analysis has been performed of the samples and time periods used in the analyses, and whether the authors adequately report the search strategy and the data sources used. Results show that 181 of the 204 studies analysed (88.7%) have one or more methodological limitations which hinder or prevent their reproducibility. This shows that there is considerable room for improvement in the methodological quality of the bibliometric papers published in Sustainability

Measuring reproducibility of high-throughput experiments

Reproducibility is essential to reliable scientific discovery in high-throughput experiments. In this work we propose a unified approach to measure the reproducibility of findings identified from replicate experiments and identify putative discoveries using reproducibility. Unlike the usual scalar measures of reproducibility, our approach creates a curve, which quantitatively assesses when the findings are no longer consistent across replicates. Our curve is fitted by a copula mixture model, from which we derive a quantitative reproducibility score, which we call the "irreproducible discovery rate" (IDR) analogous to the FDR. This score can be computed at each set of paired replicate ranks and permits the principled setting of thresholds both for assessing reproducibility and combining replicates. Since our approach permits an arbitrary scale for each replicate, it provides useful descriptive measures in a wide variety of situations to be explored. We study the performance of the algorithm using simulations and give a heuristic analysis of its theoretical properties. We demonstrate the effectiveness of our method in a ChIP-seq experiment.Comment: Published in at http://dx.doi.org/10.1214/11-AOAS466 the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org

The Reproducibility of Lists of Differentially Expressed Genes in Microarray Studies

Reproducibility is a fundamental requirement in scientific experiments and clinical contexts. Recent publications raise concerns about the reliability of microarray technology because of the apparent lack of agreement between lists of differentially expressed genes (DEGs). In this study we demonstrate that (1) such discordance may stem from ranking and selecting DEGs solely by statistical significance (P) derived from widely used simple t-tests; (2) when fold change (FC) is used as the ranking criterion, the lists become much more reproducible, especially when fewer genes are selected; and (3) the instability of short DEG lists based on P cutoffs is an expected mathematical consequence of the high variability of the t-values. We recommend the use of FC ranking plus a non-stringent P cutoff as a baseline practice in order to generate more reproducible DEG lists. The FC criterion enhances reproducibility while the P criterion balances sensitivity and specificity