Flow chart of crowdsourcing procedures.

<p>Steps are separated into sections based on separate tasks. The more granular of a process that can be made, the more amenable the process is to crowdsourcing. MTurk: process completed on MTurk. PI: process completed by AWB. R: process completed with custom R scripts. Gray boxes include tasks automated through R or MTurk, while steps outside of the gray boxes were manually completed. Circled numbers represent the number of times a task was completed. In box E, 304 preliminarily included abstracts (including the 158 ultimately included) had citation counts extracted while the final abstract ratings were concurrently being completed from box A.</p

Results of the Performance of Two Tests for Detecting Fiddling in a Distribution of p-Values^a.

a<p>Test 1 considers the total of <i>N</i> p-values and Test 2 considers only those in the interval (0.05, 0.1]. Type I error and power for the two tests are reported for various sample sizes.</p

Results of a Testing Scenario for Evaluating Simulated Data^a.

a<p>Two tests were conducted to evaluate type I error and power for various sample sizes.</p

Boxplots of OBJ and DIFF for the conditions of fiddling or no fiddling.

<p>Boxplots compare the distributions (from 1000 simulations) of comparison statistics from mixture models fitted to a distribution of p-values for which no fiddling has occurred (i.e., a level 2 null distribution) and to a distribution of p-values for which fiddling did occur. OBJ is the objective function calculated at maximum likelihood estimates of parameters of the model, and DIFF is a difference in the fitted model to 0.05 versus 0.1.</p

Using Crowdsourcing to Evaluate Published Scientific Literature: Methods and Example

<div><p>Systematically evaluating scientific literature is a time consuming endeavor that requires hours of coding and rating. Here, we describe a method to distribute these tasks across a large group through online crowdsourcing. Using Amazon's Mechanical Turk, crowdsourced workers (microworkers) completed four groups of tasks to evaluate the question, “Do nutrition-obesity studies with conclusions concordant with popular opinion receive more attention in the scientific community than do those that are discordant?” 1) Microworkers who passed a qualification test (19% passed) evaluated abstracts to determine if they were about human studies investigating nutrition and obesity. Agreement between the first two raters' conclusions was moderate (κ = 0.586), with consensus being reached in 96% of abstracts. 2) Microworkers iteratively synthesized free-text answers describing the studied foods into one coherent term. Approximately 84% of foods were agreed upon, with only 4 and 8% of ratings failing manual review in different steps. 3) Microworkers were asked to rate the perceived obesogenicity of the synthesized food terms. Over 99% of responses were complete and usable, and opinions of the microworkers qualitatively matched the authors' expert expectations (e.g., sugar-sweetened beverages were thought to cause obesity and fruits and vegetables were thought to prevent obesity). 4) Microworkers extracted citation counts for each paper through Google Scholar. Microworkers reached consensus or unanimous agreement for all successful searches. To answer the example question, data were aggregated and analyzed, and showed no significant association between popular opinion and attention the paper received as measured by Scimago Journal Rank and citation counts. Direct microworker costs totaled $221.75, (estimated cost at minimum wage:$312.61). We discuss important points to consider to ensure good quality control and appropriate pay for microworkers. With good reliability and low cost, crowdsourcing has potential to evaluate published literature in a cost-effective, quick, and reliable manner using existing, easily accessible resources.</p></div

Citations increase with journal quality and time since publication.

<p>To confirm that extracted citation counts, journal rank, and publication date conformed to expected patterns, citation counts were fit as a function of publication date controlling for journal quality (SJR; upper panel) and citation counts were fit as a function of journal quality (lower panel). No differences were seen between papers that were “known” (included) and “unknown” (excluded).</p

A Novel Generalized Normal Distribution for Human Longevity and other Negatively Skewed Data

<div><p>Negatively skewed data arise occasionally in statistical practice; perhaps the most familiar example is the distribution of human longevity. Although other generalizations of the normal distribution exist, we demonstrate a new alternative that apparently fits human longevity data better. We propose an alternative approach of a normal distribution whose scale parameter is conditioned on attained age. This approach is consistent with previous findings that longevity conditioned on survival to the modal age behaves like a normal distribution. We derive such a distribution and demonstrate its accuracy in modeling human longevity data from life tables. The new distribution is characterized by 1. An intuitively straightforward genesis; 2. Closed forms for the pdf, cdf, mode, quantile, and hazard functions; and 3. Accessibility to non-statisticians, based on its close relationship to the normal distribution.</p> </div

Plots of , with μ = 0 and σ = 1.

<p>Plots of , with μ  = 0 and σ  = 1.</p

Pseudo-Code to Generate p-Values When the Level 2 Null Hypothesis of No Fiddling is False.

<p>Pseudo-Code to Generate p-Values When the Level 2 Null Hypothesis of No Fiddling is False.</p

Results of varying initial parameter values.

<p>DNC  =  Did not converge. “Distance” was defined as the Euclidean distance between the initial values and final estimates.</p