Measuring reproducibility of high-throughput experiments

Reproducibility is essential to reliable scientific discovery in high-throughput experiments. In this work we propose a unified approach to measure the reproducibility of findings identified from replicate experiments and identify putative discoveries using reproducibility. Unlike the usual scalar measures of reproducibility, our approach creates a curve, which quantitatively assesses when the findings are no longer consistent across replicates. Our curve is fitted by a copula mixture model, from which we derive a quantitative reproducibility score, which we call the "irreproducible discovery rate" (IDR) analogous to the FDR. This score can be computed at each set of paired replicate ranks and permits the principled setting of thresholds both for assessing reproducibility and combining replicates. Since our approach permits an arbitrary scale for each replicate, it provides useful descriptive measures in a wide variety of situations to be explored. We study the performance of the algorithm using simulations and give a heuristic analysis of its theoretical properties. We demonstrate the effectiveness of our method in a ChIP-seq experiment.Comment: Published in at http://dx.doi.org/10.1214/11-AOAS466 the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org

Reproducible Econometric Simulations

Reproducibility of economic research has attracted considerable attention in recent years. So far, the discussion has focused on reproducibility of empirical analyses. This paper addresses a further aspect of reproducibility, the reproducibility of computational experiments. We examine the current situation in econometrics and derive a set of guidelines from our findings. To illustrate how computational experiments could be conducted and reported we present an example from time series econometrics that explores the finite-sample power of certain structural change tests.computational experiment, reproducibility, simulation, software.

Reproducibility of a noisy limit-cycle oscillator induced by a fluctuating input

Reproducibility of a noisy limit-cycle oscillator driven by a random piecewise constant signal is analyzed. By reducing the model to random phase maps, it is shown that the reproducibility of the limit cycle generally improves when the phase maps are monotonically increasing.Comment: 4 pages, 3 figures, Prog. Theoret. Phys. Suppl. 200

Reproducibility of deep inspiration breath hold for prone left-sided whole breast irradiation

Background: Investigating reproducibility and instability of deep inspiration breath hold (DIBH) in the prone position to reduce heart dose for left-sided whole breast irradiation. Methods: Thirty patients were included and underwent 2 prone DIBH CT-scans during simulation. Overlap indices were calculated for the ipsilateral breast, heart and lungs to evaluate the anatomical reproducibility of the DIBH maneuver. The breathing motion of 21 patients treated with prone DIBH were registered using magnetic probes. These breathing curves were investigated to gain data on intra-fraction reproducibility and instability of the different DIBH cycles during treatment. Results: Overlap index was 0.98 for the ipsilateral breast and 0.96 for heart and both lungs between the 2 prone DIBH-scans. The magnetic sensors reported population amplitudes of 2.8 +/- 1.3 mm for shallow breathing and 11.7 +/- 4.7 mm for DIBH, an intra-fraction standard deviation of 1.0 +/- 0.4 mm for DIBH, an intra-breath hold instability of 1.0 +/- 0.6 mm and a treatment time of 300 +/- 69 s. Conclusion: Prone DIBH can be accurately clinically implemented with acceptable reproducibility and instability

Sharing and Preserving Computational Analyses for Posterity with encapsulator

Open data and open-source software may be part of the solution to science's "reproducibility crisis", but they are insufficient to guarantee reproducibility. Requiring minimal end-user expertise, encapsulator creates a "time capsule" with reproducible code in a self-contained computational environment. encapsulator provides end-users with a fully-featured desktop environment for reproducible research.Comment: 11 pages, 6 figure

Predicting computational reproducibility of data analysis pipelines in large population studies using collaborative filtering

Evaluating the computational reproducibility of data analysis pipelines has become a critical issue. It is, however, a cumbersome process for analyses that involve data from large populations of subjects, due to their computational and storage requirements. We present a method to predict the computational reproducibility of data analysis pipelines in large population studies. We formulate the problem as a collaborative filtering process, with constraints on the construction of the training set. We propose 6 different strategies to build the training set, which we evaluate on 2 datasets, a synthetic one modeling a population with a growing number of subject types, and a real one obtained with neuroinformatics pipelines. Results show that one sampling method, "Random File Numbers (Uniform)" is able to predict computational reproducibility with a good accuracy. We also analyze the relevance of including file and subject biases in the collaborative filtering model. We conclude that the proposed method is able to speedup reproducibility evaluations substantially, with a reduced accuracy loss

Replicability is not Reproducibility:\ud Nor is it Good Science

At various machine learning conferences, at various times, there have been discussions arising from the inability to replicate the experimental results published in a paper. There seems to be a wide spread view that we need to do something to address this problem, as it is essential to the advancement of our field. The most compelling argument would seem to be that reproducibility of experimental results is the hallmark of science. Therefore, given that most of us regard machine learning as a scientific discipline, being able to replicate experiments is paramount. I want to challenge this view by separating the notion of reproducibility, a generally desirable property, from replicability, its poor cousin. I claim there are important differences between the two. Reproducibility requires changes; replicability avoids them. Although reproducibility is desirable, I contend that the impoverished version, replicability, is one not worth having

Reproducibility of lymphovascular space invasion (LVSI) assessment in endometrial cancer

Aims Lymphovascular space invasion (LVSI) in endometrial cancer (EC) is an important prognostic variable impacting on a patient's individual recurrence risk and adjuvant treatment recommendations. Recent work has shown that grading the extent of LVSI further improves its prognostic strength in patients with stage I endometrioid EC. Despite this, there is little information on the reproducibility of LVSI assessment in EC. Therefore, we designed a study to evaluate interobserver agreement in discriminating true LVSI from LVSI mimics (Phase I) and reproducibility of grading extent of LVSI (Phase II). Methods and results Scanned haematoxylin and eosin (H&E) slides of endometrioid EC (EEC) with a predefined possible LVSI focus were hosted on a website and assessed by a panel of six European gynaecological pathologists. In Phase I, 48 H&E slides were included for LVSI assessment and in Phase II, 42 H&E slides for LVSI grading. Each observer was instructed to apply the criteria for LVSI used in daily practice. The degree of agreement was measured using the two-way absolute agreement average-measures intraclass correlation coefficient (ICC). Reproducibility of LVSI assessment (ICC = 0.64, P < 0.001) and LVSI grading (ICC = 0.62, P < 0.001) in EEC was substantial among the observers. Conclusions Given the good reproducibility of LVSI, this study further supports the important role of LVSI in decision algorithms for adjuvant treatment

Reproducibility Performance Test of Multi Metal Oxide Catalyst in Selective Oxidation of Propane using Combinatorial Technology

The concept of rapid catalyst screening using combinatorial technology was applied in the development of selective oxidation catalyst. In this paper, the design, Design of Experiment (DOE) and catalytic results are discussed to demonstrate the importance and versatility of such technology. The instrument is implaced in COMBICAT (Universiti Malaya) to rapidly support parallel testing of catalytic material using continuous fixed bed reactor technology programme. It is used for the automated parallel testing of selective oxidation of propane to acrylic acid over some types of multi metal oxide catalysts. The configuration of the ‘nanoflow’ is shown to be suitable to screen catalytic performance, and its operating conditions were mimicked closely to conventional laboratory as well as to industrial conditions. The results obtained gave very good reproducibility