Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer fiveoriginal research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams renderedstatistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.</div

CORT and Bd lineage

<p>This is the CORT and Bd lineage data from experiment 2. See Gabor et al. PLOSOne</p

Bd, CORT and behavior

<p>This is CORT, chytrid and behavior data from Experiment 1 and 3 in our PlosOne publication Gabor et al. 2015</p

Gabor et al. Bd, CORT and behavior data

<p>These are the data sets for our Gabor et al. 2105 PlosOne paper</p

A general RT-PCR approach using primers to exon 3 or 4 and 3'UTR failed to detect VEGFxxxb isoforms in mouse and human cDNA extracts respectively.

<p><b>A</b>, PCR products corresponding to VEGF188 (474 bp), VEGF164 (402 bp), VEGF164/120 heteroduplex (arrowed) and VEGF120 (270 bp) are evident in the panel of mouse cDNAs amplified using the 3′UTR reverse primer and a forward primer to exon 3. <b>B</b> & <b>C</b>, PCR products corresponding to VEGF189 (371 bp), VEGF165 (299 bp) and VEGF121 (167 bp) are evident in the commercial human tissue cDNAs amplified using the 3′UTR reverse primer (DB 3'UTR) and a forward primer to exon 4 (DB exon4). <b>D</b>, Amplification of the heteroduplex species (arrowed) a lso occurred when VEGF164 and VEGF120 tumour cDNAs were pooled and analysed by RT-PCR using exon3/3'UTR primers (lanes labelled 120+164). <b>E</b>, The same heteroduplex species was generated when cDNAs from VEGF164 and VEGF188 expressing fibrosarcoma cells were pooled and amplified as above (lanes labelled 120+164). M corresponds to a 100 bp ladder, whilst -tem represents a control PCR reaction in which water was used instead of cDNA template. The Figure contains data we published previously <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0035231#pone.0035231-Tozer1" target="_blank">[25]</a>.</p

A general RT-PCR approach with primers to exon 7a and 3'UTR failed to detect VEGFxxxb isoforms in human and mouse cDNA extracts.

<p>RT-PCR reactions were performed using forward/reverse primers designed to amplify both VEGFxxx and VEGFxxxb isoforms from cDNAs extracted from mouse fibrosarcoma cell lines & tumours and normal mouse tissues (heart, lung, liver and kidney) as well as commercially sourced human tissue cDNAs. <b>A</b>, A single product corresponding to VEGF164/188 (194 bp) is evident in the panel of mouse cDNAs using forward and reverse primers designed to exon 7 and the 3'UTR (exon7/3′UTR) with no evidence of VEGF164b/188b (128 bp) products. <b>B</b> & <b>C</b>, A single product corresponding to VEGF165/189 (194 bp) is evident in human brain, bladder and kidney (AMS Biotechnology Ltd), and prostate and kidney (Primer Design Ltd) using forward and reverse primers designed to exon 7 (DB exon7a) and the 3'UTR (DB 3'UTR) respectively. No products corresponding to VEGF165b/189b (128 bp) are evident in any of these samples.</p

Murine VEGF-A splice variants.

<p>Several splice variants of VEGF are produced, by alternative splicing of exons 6 & 7. Exons 3 & 4 encode dimerisation and VEGF receptor binding sites, exon 7 encodes neuropilin binding sites and exons 6 & 7 encode heparin binding sites.</p

Effect of IGF-1 and TGFβ-1 on expression of VEGF isoforms in mouse podocytes.

<p>Cell lysates were prepared from untreated control (C) podocyte cells or cells treated with either 100 nM IGF-1 (IGF) or 1 nM TGFβ-1 (TGF) for 12 hours in serum free media. Results from three independent experiments are shown. RT-PCR using general primers designed to simultaneously amplify both VEGFxxx and VEGFxxxb isoforms (exon7a/3'UTR) revealed only VEGFxxx (194 bp). Qualitative assessment of the PCR products suggests that treatment with either growth factor increased VEGFxxx expression. The same RT-PCR strategy using three different extracts (1, 2, 3) from HEK 293 cells similarly revealed only VEGFxxx (194 bp). (-tem) corresponds to reactions containing water instead of cDNA. (-RT) corresponds to reactions containing water instead of reverse transcriptase.</p

C-terminal exon sequences of murine and human VEGF-A genes.

<p>Top two panels show the predicted splicing reactions for murine VEGF188b, 164b and 120b (sequence in bold and arrowed), highlighting the exon 7/exon 8b and exon 5/exon 8b splice junction sequences. These splicing reactions would generate putative VEGFxxxb protein isoforms with a <b><i>PLTGKTD</i></b> C-terminus. The reverse PCR primers designed for this study are indicated below the exon sequences. The lower panel shows the corresponding sequence of the human VEGF-A gene C-terminus highlighting the exon 7/exon 8b splice junction that generates VEGFxxxb isoforms with a <b><i>SLTRKD</i></b> C-terminus. The reverse PCR primers used in the amplification reactions are indicated below the exon sequences and DB 165b/188b-1 and DB 3'UTR are as previously published <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0035231#pone.0035231-Bates1" target="_blank">[26]</a>. The bases highlighted in bold exhibit mismatches from the published human VEGF-A sequence.</p

Discriminating VEGF isoform-specific RT-PCR using human cDNA extracts further highlights the importance of primer design.

<p><b>A</b>, Human cDNAs from colon, prostate and kidney (Primer Design Ltd) were amplified in RT-PCR reactions using an exon 4 forward primer together with either DB165b/189b-1 (lane 1), h165b/189b-2 (lane 2), h165b/189b-3 (lane 3), h165b/189b-4 (lane 4) or h165b/189b-5 (lane 5). Primer sequences are highlighted in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0035231#pone-0035231-t001" target="_blank">Table 1</a> & <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0035231#pone-0035231-g002" target="_blank">Fig 2</a>. A putative VEGF165b (∼212 bp) product is evident in lanes 2 & 3, but absent in lane 1. All three reverse primers spanned 5 bases across the 8b/7 junction, however differed in their melting temperatures (see main text), with h165b/189b-2 & 3 exhibiting more compatible Tms with the forward primer. A similar 165b product was more abundantly amplified in lanes 4 & 5 using reverse primers spanning more bases (13 or 14 respectively) across the 8b/7 splice junction. In addition, these latter reverse primers also amplified a product corresponding to 188b (∼283 bp). However DNA sequencing revealed that the product amplified in lanes 2 & 3 was not VEGF but LIM domain only, protein isoform 7, confirming that detection of apparent VEGFxxxb species was only evident using 8b/7 reverse primers with increasing complementary sequence to exon7. <b>B</b>, Similar results were obtained in RT-PCR reactions using cDNA extracted from a normal kidney patient biopsy and amplified with the exon 4 forward primer and the DB165b/189b-1 (lane1), h165b/189b-2 (lane 2) and h165b/189b-4 (lane 4) reverse primers. <b>C</b>, RT-PCR using the DBexon7/DB3'UTR forward/reverse primers capable of simultaneously amplifying VEGFxxx & VEGFxxxb, confirmed only VEGFxxx (194 bp) amplification in the human samples tested.</p