Assessing the reproducibility of discriminant function analyses

Data are the foundation of empirical research, yet all too often the datasets underlying published papers are unavailable, incorrect, or poorly curated. This is a serious issue, because future researchers are then unable to validate published results or reuse data to explore new ideas and hypotheses. Even if data files are securely stored and accessible, they must also be accompanied by accurate labels and identifiers. To assess how often problems with metadata or data curation affect the reproducibility of published results, we attempted to reproduce Discriminant Function Analyses (DFAs) from the field of organismal biology. DFA is a commonly used statistical analysis that has changed little since its inception almost eight decades ago, and therefore provides an opportunity to test reproducibility among datasets of varying ages. Out of 100 papers we initially surveyed, fourteen were excluded because they did not present the common types of quantitative result from their DFA or gave insufficient details of their DFA. Of the remaining 86 datasets, there were 15 cases for which we were unable to confidently relate the dataset we received to the one used in the published analysis. The reasons ranged from incomprehensible or absent variable labels, the DFA being performed on an unspecified subset of the data, or the dataset we received being incomplete. We focused on reproducing three common summary statistics from DFAs: the percent variance explained, the percentage correctly assigned and the largest discriminant function coefficient. The reproducibility of the first two was fairly high (20 of 26, and 44 of 60 datasets, respectively), whereas our success rate with the discriminant function coefficients was lower (15 of 26 datasets). When considering all three summary statistics, we were able to completely reproduce 46 (65%) of 71 datasets. While our results show that a majority of studies are reproducible, they highlight the fact that many studies still are not the carefully curated research that the scientific community and public expects

ripr_analysis_code_annotated

The R code used for the analysis in the paper, including drawing the figures

reviewer data from Molecular Ecology

This file contains the reviewer data for Molecular Ecology. The other data files called by the analysis code are from Petchey et al (2014), see http://dx.doi.org/10.5061/dryad.36r6

Data from: Is it becoming harder to secure reviewers for peer review? A test with data from five ecology journals

Background: There is concern in the academic publishing community that it is becoming more difficult to secure reviews for peer-reviewed manuscripts, but much of this concern stems from anecdotal and rhetorical evidence. Methods: We examined the proportion of review requests that led to a completed review over a 6-year period (2009–2015) in a mid-tier biology journal (Molecular Ecology). We also re-analyzed previously published data from four other mid-tier ecology journals (Functional Ecology, Journal of Ecology, Journal of Animal Ecology, and Journal of Applied Ecology), looking at the same proportion over the period 2003 to 2010. Results: The data from Molecular Ecology showed no significant decrease through time in the proportion of requests that led to a review (proportion in 2009 = 0.47 (95 % CI = 0.43 to 0.52), proportion in 2015 = 0.44 (95 % CI = 0.40 to 0.48)). This proportion did decrease for three of the other ecology journals (changes in proportions from 2003 to 2010 = −0.10, −0.18, and −0.09), while the proportion for the fourth (Functional Ecology) stayed roughly constant (change in proportion = −0.04). Conclusions: Overall, our data suggest that reviewer agreement rates have probably declined slightly but not to the extent suggested by the anecdotal and rhetorical evidence

Assessing the reproducibility of discriminant function analyses

Data are the foundation of empirical research, yet all too often the datasets underlying published papers are unavailable, incorrect, or poorly curated. This is a serious issue, because future researchers are then unable to validate published results or reuse data to explore new ideas and hypotheses. Even if data files are securely stored and accessible, they must also be accompanied by accurate labels and identifiers. To assess how often problems with metadata or data curation affect the reproducibility of published results, we attempted to reproduce Discriminant Function Analyses (DFAs) from the field of organismal biology. DFA is a commonly used statistical analysis that has changed little since its inception almost eight decades ago, and therefore provides an opportunity to test reproducibility among datasets of varying ages. Out of 100 papers we initially surveyed, fourteen were excluded because they did not present the common types of quantitative result from their DFA or gave insufficient details of their DFA. Of the remaining 86 datasets, there were 15 cases for which we were unable to confidently relate the dataset we received to the one used in the published analysis. The reasons ranged from incomprehensible or absent variable labels, the DFA being performed on an unspecified subset of the data, or the dataset we received being incomplete. We focused on reproducing three common summary statistics from DFAs: the percent variance explained, the percentage correctly assigned and the largest discriminant function coefficient. The reproducibility of the first two was fairly high (20 of 26, and 44 of 60 datasets, respectively), whereas our success rate with the discriminant function coefficients was lower (15 of 26 datasets). When considering all three summary statistics, we were able to completely reproduce 46 (65%) of 71 datasets. While our results show that a majority of studies are reproducible, they highlight the fact that many studies still are not the carefully curated research that the scientific community and public expects

16S rRNA gene sequence and phylogenetic tree of lactobacillus species from the vagina of healthy Nigerian women

Background: The vaginal microbial community plays a vital role in maintaining women\u27s health. Understanding the precise bacterial composition is challenging because of the diverse and difficult-to-culture nature of many bacterial constituents, necessitating culture-independent methodology. During a natural menstrual cycle, physiological changes could have an impact on bacterial growth, colonization, and community structure. The objective of this study was to assess the stability of the vaginal microbiome of healthy Canadian women throughout a menstrual cycle by using cpn60-based microbiota analysis. Vaginal swabs from 27 naturally cycling reproductive-age women were collected weekly through a single menstrual cycle. Polymerase chain reaction (PCR) was performed to amplify the universal target region of the cpn60 gene and generate amplicons representative of the microbial community. Amplicons were pyrosequenced, assembled into operational taxonomic units, and analyzed. Samples were also assayed for total 16S rRNA gene content and Gardnerella vaginalis by quantitative PCR and screened for the presence of Mollicutes by using family and genus-specific PCR.Results: Overall, the vaginal microbiome of most women remained relatively stable throughout the menstrual cycle, with little variation in diversity and only modest fluctuations in species richness. Microbiomes between women were more different than were those collected consecutively from individual women. Clustering of microbial profiles revealed the expected groupings dominated by Lactobacillus crispatus, Lactobacillus iners, and Lactobacillus jensenii. Interestingly, two additional clusters were dominated by either Bifidobacterium breve or a heterogeneous mixture of nonlactobacilli. Direct G. vaginalis quantification correlated strongly with its pyrosequencing-read abundance, and Mollicutes, including Mycoplasma hominis, Ureaplasma parvum, and Ureaplasma urealyticum, were detected in most samples.Conclusions: Our cpn60-based investigation of the vaginal microbiome demonstrated that in healthy women most vaginal microbiomes remained stable through their menstrual cycle. Of interest in these findings was the presence of Bifidobacteriales beyond just Gardnerella species. Bifidobacteriales are frequently underrepresented in 16S rRNA gene-based studies, and their detection by cpn60-based investigation suggests that their significance in the vaginal community may be underappreciated