Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer fiveoriginal research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams renderedstatistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.</div

The distributions of fitted selection probabilities on the extended apoptosis model.

<p>The distributions of fitted selection probabilities on the extended apoptosis model.</p

Run-time analysis of grid-based <i>optPBN</i> pipeline.

<p>Run-time analysis of grid-based <i>optPBN</i> pipeline.</p

An example model with the corresponding Boolean rules, truth table and model simulation results.

<p>[A] The example model consists of 3 nodes with one activation edge and one partial inhibition edge. The weights of both edges are expressed as selection probability next to the arrow. [B] Two representative Boolean rules were assigned with the corresponding selection probabilities () to represent the example model in PBN format. [C] The truth table of the example model demonstrates the state values according to different inputs. Once both inputs (N1 and N2) are active, the output (N3) has a probability of being ON at 0.6 and of being OFF at 0.4 according to the selection probability of Boolean rules. [D] Three separated Monte-Carlo simulations were performed on an instantaneously random PBN of the example model in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0098001#pone-0098001-g001" target="_blank">Figure 1</a>. The state values of N3 are shown on the y-axis as a function of time. The mean of the N3 state values over 20 time steps is given on the upper right corner of each run.</p

Optimisation pipeline of the <i>optPBN</i> toolbox.

<p>A preliminary model structure is required as an input for the generation of a PBN model. The generated PBN models from different experimental conditions together with the corresponding measurement data are subsequently combined to generate an integrated optimisation problem which can be solved by various optimisation algorithms. Once the optimisation algorithm(s) generate sufficient amount of good parameter sets, a statistical analysis of the optimised parameter sets (i.e., of PBN's selection probabilities) is performed to indicate the identifiability and the sensitivity of parameters through the consideration on parameters' distribution. The <i>optPBN</i> scripts used for each task are given in parentheses.</p

Case study 2.

<p>[A] Case study 2 deals with a probabilistic Boolean network that consists of 3 nodes with an unknown type and weight of interaction from PTEN to PIP3. [B] The table contains artificial experimental data from four different combinations of input states (Experiments ‘A’, ‘B’, ‘C’, and ‘D’) of case study 2.</p

Results from the <i>optPBN</i> toolbox for case study 2 compared to the original network.

<p>Results from the <i>optPBN</i> toolbox for case study 2 compared to the original network.</p

Case study 1.

<p>[A] Case study 1 deals with a Boolean network that consists of 3 nodes with an unknown Boolean interaction from the two inputs. [B] The table contains artificial experimental data from four different combinations of input states (Experiments ‘A’, ‘B’, ‘C’, and ‘D’) of case study 1.</p

Modified toy model of Saez-Rodriguez <i>et al.</i> and corresponding artificial experimental data (case study 3).

<p>[A] The modified toy model from Saez-Rodriguez <i>et al.</i><a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0098001#pone.0098001-SaezRodriguez2" target="_blank">[18]</a> is a probabilistic Boolean network that consists of 8 nodes with two unknown weights of Boolean interactions for NFkB and ERK. [B] The table describes the states of inputs and inhibitor treatments for 6 experimental conditions. [C] The corresponding normalised artificial experimental data of the experimental conditions as described in [B]. The 6 experimental conditions based on the combination of stimulus and inhibitor treatments yield different readouts on four downstream molecules.</p

Results from the <i>optPBN</i> toolbox for case study 3 compared to the original network.

<p>Results from the <i>optPBN</i> toolbox for case study 3 compared to the original network.</p