Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer fiveoriginal research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams renderedstatistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.</div

A multidimensional IRT model (MIRT) used in simulating data.

<p>All latent variables were normally distributed with standard deviation of 1 and all symptoms were binary. The edges in this model correspond to item discrimination parameters.</p

Examples of estimated network structures when the true network is sparse, using different sample sizes and estimation methods.

<p>Graphs were drawn using the <i>qgraph</i> package without setting a maximum value (i.e., the strongest edge in each network has full saturation and width).</p

Estimated network structures based on data generated by the MIRT model in Fig 4.

<p>Estimated network structures based on data generated by the MIRT model in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0179891#pone.0179891.g004" target="_blank">Fig 4</a>.</p

True network structures used in simulation study.

<p>The first network is a Curie-Weiss network: a fully connected network in which all edges have the same strength. The second network is a random sparse network. All edge weights are 0.2.</p

Acceptance rates for the SVE algorithm with different bin sizes <i>a</i>.

<p>The average proportion of accepted points when proposals are generated until <i>t</i>(<i>x</i>*) falls in the range (<i>t</i>(<i>x</i>) − <i>a</i>, <i>t</i>(<i>x</i>) + <i>a</i>) using <i>a</i> ∈ {∞, 5, 3, 2}. The gray bars reflect both the range (left and right endpoints) and the proportion of accepted points (top).</p

Number of rejected samples for the rejection algorithms.

<p>The average number of rejected samples per posterior <i>π</i>(<i>θ</i> ∣ <i>x</i>₊) out of <i>n</i> target distributions in the original Rejection algorithm (solid line) and when rejected values are recycled among the target distributions (points).</p

Acceptance rates of the original SVE and SVE with matching.

<p>The average proportion of accepted points when simultaneously sampling from <i>n</i> target distributions <i>π</i>(<i>θ</i> ∣ <i>x</i>₊) in the original SVE algorithm (solid line) and the proposal matching procedure (points).</p

Turning Simulation into Estimation: Generalized Exchange Algorithms for Exponential Family Models

<div><p>The Single Variable Exchange algorithm is based on a simple idea; any model that can be simulated can be estimated by producing draws from the posterior distribution. We build on this simple idea by framing the Exchange algorithm as a mixture of Metropolis transition kernels and propose strategies that automatically select the more efficient transition kernels. In this manner we achieve significant improvements in convergence rate and autocorrelation of the Markov chain without relying on more than being able to simulate from the model. Our focus will be on statistical models in the Exponential Family and use two simple models from educational measurement to illustrate the contribution.</p></div

Acceptance rates for the original SVE algorithm and SVE using as proposal.

<p>The average acceptance rate for sampling from posterior distributions <i>π</i>(<i>θ</i> ∣ <i>x</i>₊) when using the original SVE algorithm (left panel) and when using the proposal distribution (right panel).</p