Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer fiveoriginal research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams renderedstatistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.</div

The Effect of Time Pressure and Cognitive Load on Moral Judgment.

<p>The Effect of Time Pressure and Cognitive Load on Moral Judgment.</p

Logistic and OLS regressions on altruistic behavior in the dictator game in the time-pressure experiment, effects shown as marginal effects (ME) and betas.

<p>Logistic and OLS regressions on altruistic behavior in the dictator game in the time-pressure experiment, effects shown as marginal effects (ME) and betas.</p

Logistic and OLS regressions on utilitarian judgements in moral dilemmas in the cognitive load experiment, effects shown as marginal effects (ME) and betas.

<p>Logistic and OLS regressions on utilitarian judgements in moral dilemmas in the cognitive load experiment, effects shown as marginal effects (ME) and betas.</p

List of Analyzed SNPs.

<p>Notes: Tests of Hardy-Weinberg conducted using likelihood ratio tests using only a sample of genetically unrelated individuals (one twin from each pair was randomly selected if genotypic data was available for both twins).</p

Summary Statistics for the Sample.

<p>Notes: Years of education estimated from categorical variable produced by Statistics Sweden (with seven categories ranging from middle school to PhD).</p

Regression Results: Non-Additive Model.

<p>Notes: This table reports regression coefficients from the non-additive model, estimated separately for each individual SNP. All regressions include age and sex controls. The reported p-values are for the F-test of the joint hypothesis that the additive and dominance coefficients are both equal to zero.</p

Regression Results: Additive Model with Sex Specific Effects.

<p>Notes: This table reports regression coefficients from the additive model, estimated separately for each individual SNP and allowing different coefficients in men and women. All regressions include age and sex controls. Three stars (***) denote statistical significance at the one percent level, two stars (**) denote statistical significance at the five percent level and one star (*) denotes statistical significance at the ten percent level.</p

Regression Results: Additive Model.

<p>Notes: This table reports regression coefficients from the additive model, estimated separately for each individual SNP. All regressions include age and sex controls. One star (*) denotes statistical significance at the ten percent level (three coefficient estimates are statistically significant at the ten percent level, and none is significant at the five percent level).</p

Trivariate ADE Cholesky decomposition for Flow Proneness (FP), Behavioural Inhibition (BI), and Locus of Control (LOC) showing non-significant pathways (dashed lines) for all additive (A) and most dominant (D) genetic influences indicating low power to distinguish between A and D.

<p>Trivariate ADE Cholesky decomposition for Flow Proneness (FP), Behavioural Inhibition (BI), and Locus of Control (LOC) showing non-significant pathways (dashed lines) for all additive (A) and most dominant (D) genetic influences indicating low power to distinguish between A and D.</p