Crowdsourcing prior information to improve study design and data analysis.

Though Bayesian methods are being used more frequently, many still struggle with the best method for setting priors with novel measures or task environments. We propose a method for setting priors by eliciting continuous probability distributions from naive participants. This allows us to include any relevant information participants have for a given effect. Even when prior means are near-zero, this method provides a principle way to estimate dispersion and produce shrinkage, reducing the occurrence of overestimated effect sizes. We demonstrate this method with a number of published studies and compare the effect of different prior estimation and aggregation methods

Densities of raw best guess responses to concrete elicitation, colored by condition and faceted by study question.

<p>The horizontal axis is logarithmically scaled to minimize the visual effect of extreme estimates.</p

Sums of squared error by participant, collapsing across conditions, for each of the eight questions.

<p>Sums of squared error by participant, collapsing across conditions, for each of the eight questions.</p

Densities of participant responses to abstract prompt.

<p>Superimposed quantities give the mean probability for the estimated truth of each statement.</p

Distributions of for each model.

<p>Distributions of for each model.</p

t-values as reported by the original studies and as recalculated using our elicitation and aggregation methods.

<p>t-values as reported by the original studies and as recalculated using our elicitation and aggregation methods.</p

Cumulative normal curves generated from maximum likelihood estimates for each participant and condition, faceted by question.

<p>Participant curves for both conditions are in gray, while the curves corresponding the median mean and standard parameters in each condition are used to create consensus curves, colored by condition.</p

True parameter values (red) for each parameter compared with the HBM posterior estimates for the four top-level hyperparameters fit to one set of twenty subjects simulated from fixed parameters.

<p>True parameter values (red) for each parameter compared with the HBM posterior estimates for the four top-level hyperparameters fit to one set of twenty subjects simulated from fixed parameters.</p

Example pooling coefficients for m and s parameters in the control condition of question 4.

<p>The pooling coefficient is a measure of how much a given parameter is shrunk toward the grand means, in this case <i>μ</i> and <i>σ</i>: 0 is no pooling and 1 is infinite pooling.</p

Sample size required to reach 80 percent power for based on the empirical and consensus-modeled effect sizes.

<p>Sample size required to reach 80 percent power for based on the empirical and consensus-modeled effect sizes.</p