Data from a pre-publication independent replication initiative examining ten moral judgement effects

We present the data from a crowdsourced project seeking to replicate findings in independent laboratories before (rather than after) they are published. In this Pre-Publication Independent Replication (PPIR) initiative, 25 research groups attempted to replicate 10 moral judgment effects from a single laboratory's research pipeline of unpublished findings. The 10 effects were investigated using online/lab surveys containing psychological manipulations (vignettes) followed by questionnaires. Results revealed a mix of reliable, unreliable, and culturally moderated findings. Unlike any previous replication project, this dataset includes the data from not only the replications but also from the original studies, creating a unique corpus that researchers can use to better understand reproducibility and irreproducibility in science

The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline

This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had “in the pipeline” as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed

Enumerating pelvic recurrence following radical cystectomy for bladder cancer: A canadian multi-institutional study

Introduction: We aimed to enumerate the rate of pelvic recurrence following radical cystectomy at university-affiliated hospitals in Canada. Methods: Canadian, university-affiliated hospitals were invited to participate. They were asked to identify the first 10 consecutive patients undergoing radical cystectomy starting January 1, 2005, who had urothelial carcinoma stages pT3/T4 N0-2 M0. The first 10 consecutive cases starting January 1, 2005 who met these criteria were the patients submitted by that institution with information regarding tumour stage, age, number of nodes removed, and last known clinical status in regard to recurrence and patterns of failure. Results: Of the 111 patients, 80% had pT3 and 20% pT4 disease, with 62% being node-negative, 14% pN1, and 27% pN2; 57% had 10 or more nodes removed. Cumulative incidence of pelvic relapse was 40% among the entire group Conclusions: This review demonstrates a high rate of pelvic tumour recurrence following radical cystectomy for pT3/T4 urothelial cancer

The pipeline project:Pre-publication independent replications of a single laboratory's research pipeline

This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had "in the pipeline" as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed. (C) 2015 The Authors. Published by Elsevier Inc.</p

The neuropathology of autism: defects of neurogenesis and neuronal migration, and dysplastic changes

Autism is characterized by a broad spectrum of clinical manifestations including qualitative impairments in social interactions and communication, and repetitive and stereotyped patterns of behavior. Abnormal acceleration of brain growth in early childhood, signs of slower growth of neurons, and minicolumn developmental abnormalities suggest multiregional alterations. The aim of this study was to detect the patterns of focal qualitative developmental defects and to identify brain regions that are prone to developmental alterations in autism. Formalin-fixed brain hemispheres of 13 autistic (4–60 years of age) and 14 age-matched control subjects were embedded in celloidin and cut into 200-μm-thick coronal sections, which were stained with cresyl violet and used for neuropathological evaluation. Thickening of the subependymal cell layer in two brains and subependymal nodular dysplasia in one brain is indicative of active neurogenesis in two autistic children. Subcortical, periventricular, hippocampal and cerebellar heterotopias detected in the brains of four autistic subjects (31%) reflect abnormal neuronal migration. Multifocal cerebral dysplasia resulted in local distortion of the cytoarchitecture of the neocortex in four brains (31%), of the entorhinal cortex in two brains (15%), of the cornu Ammonis in four brains and of the dentate gyrus in two brains. Cerebellar flocculonodular dysplasia detected in six subjects (46%), focal dysplasia in the vermis in one case, and hypoplasia in one subject indicate local failure of cerebellar development in 62% of autistic subjects. Detection of flocculonodular dysplasia in only one control subject and of a broad spectrum of focal qualitative neuropathological developmental changes in 12 of 13 examined brains of autistic subjects (92%) reflects multiregional dysregulation of neurogenesis, neuronal migration and maturation in autism, which may contribute to the heterogeneity of the clinical phenotype

Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer fiveoriginal research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams renderedstatistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.</div

Examining the generalizability of research findings from archival data

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples

Data from a pre-publication independent replication initiative examining ten moral judgement effects

We present the data from a crowdsourced project seeking to replicate findings in independent laboratories before (rather than after) they are published. In this Pre-Publication Independent Replication (PPIR) initiative, 25 research groups attempted to replicate 10 moral judgment effects from a single laboratory's research pipeline of unpublished findings. The 10 effects were investigated using online/lab surveys containing psychological manipulations (vignettes) followed by questionnaires. Results revealed a mix of reliable, unreliable, and culturally moderated findings. Unlike any previous replication project, this dataset includes the data from not only the replications but also from the original studies, creating a unique corpus that researchers can use to better understand reproducibility and irreproducibility in science.Link_to_subscribed_fulltex