Towards evidence-based computational statistics: lessons from clinical research on the role and design of real-data benchmark studies

Abstract Background The goal of medical research is to develop interventions that are in some sense superior, with respect to patient outcome, to interventions currently in use. Similarly, the goal of research in methodological computational statistics is to develop data analysis tools that are themselves superior to the existing tools. The methodology of the evaluation of medical interventions continues to be discussed extensively in the literature and it is now well accepted that medicine should be at least partly “evidence-based”. Although we statisticians are convinced of the importance of unbiased, well-thought-out study designs and evidence-based approaches in the context of clinical research, we tend to ignore these principles when designing our own studies for evaluating statistical methods in the context of our methodological research. Main message In this paper, we draw an analogy between clinical trials and real-data-based benchmarking experiments in methodological statistical science, with datasets playing the role of patients and methods playing the role of medical interventions. Through this analogy, we suggest directions for improvement in the design and interpretation of studies which use real data to evaluate statistical methods, in particular with respect to dataset inclusion criteria and the reduction of various forms of bias. More generally, we discuss the concept of “evidence-based” statistical research, its limitations and its impact on the design and interpretation of real-data-based benchmark experiments. Conclusion We suggest that benchmark studies—a method of assessment of statistical methods using real-world datasets—might benefit from adopting (some) concepts from evidence-based medicine towards the goal of more evidence-based statistical research

Essential guidelines for computational method benchmarking

In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the performance of different methods using well-characterized benchmark datasets, to determine the strengths of each method or to provide recommendations regarding suitable choices of methods for an analysis. However, benchmarking studies must be carefully designed and implemented to provide accurate, unbiased, and informative results. Here, we summarize key practical guidelines and recommendations for performing high-quality benchmarking analyses, based on our experiences in computational biology.Comment: Minor update

Essential guidelines for computational method benchmarking

In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the performance of different methods using well-characterized benchmark datasets, to determine the strengths of each method or to provide recommendations regarding suitable choices of methods for an analysis. However, benchmarking studies must be carefully designed and implemented to provide accurate, unbiased, and informative results. Here, we summarize key practical guidelines and recommendations for performing high-quality benchmarking analyses, based on our experiences in computational biology

On the optimistic performance evaluation of newly introduced bioinformatic methods

Most research articles presenting new data analysis methods claim that “the new method performs better than existing methods,” but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of novel data analysis methods, that is, all biases resulting from, for example, selection of datasets or competing methods, better ability to fix bugs in a preferred method, and selective reporting of method variants. We quantitatively investigate this bias using an example from epigenetic analysis: normalization methods for data generated by the Illumina HumanMethylation450K BeadChip microarray

Making Complex Prediction Rules Applicable for Readers: Current Practice in Random Forest Literature and Recommendations

Ideally, prediction rules (including classifiers as a special case) should be published in such a way that readers may apply them, for example to make predictions for their own data. While this is straightforward for simple prediction rules, such as those based on the logistic regression model, this is much more difficult for complex prediction rules derived by machine learning tools. We conducted a survey of articles reporting prediction rules that were constructed using the random forest algorithm and published in PLOS ONE in 2014-2015 with the aim to identify issues related to their applicability. The presented prediction rules were applicable in only 2 of 30 identified papers, while for further 8 prediction rules it was possible to obtain the necessary information by contacting the authors. Various problems, such as non-response of the authors, hampered the applicability of prediction rules in the other cases. Based on our experiences from the survey, we formulate a set of recommendations for authors publishing complex prediction rules to ensure their applicability for readers

The aminotransferase Aat initiates 3-phenyllactic acid biosynthesis in Pediococcus acidilactici

The function of the aminotransferase Aat (GenBank Protein WP_159211138) from Pediococcus acidilactici FAM 18098 was studied in vivo. For this purpose, the gene was replaced with an erythromycin resistance gene using the temperature-sensitive Escherichia coli-Pediococcus shuttle plasmid pSET4T_Δaat. The knockout was verified by PCR and genome sequencing. Subsequently, the differences between the metabolism of the knockout and of the wild-type strain were investigated by determining the free amino acids and organic acids in culture supernatants. It was found that the knockout mutant no longer synthesized 3-phenyllactic acid (PLA) and 4-hydroxyphenyllactic acid (HPLA). Additionally, the mutant strain no longer catabolized phenylalanine. Metabolic pathway analysis using the KEGG database indicate that P. acidilactici cannot synthesize α-ketoglutarate that is a predominant amino-group acceptor in many transamination reactions. To study the transfer of the amino group of phenylalanine, the wild-type strain was incubated with [15N] phenylalanine. Mass spectrometry showed that during fermentation, [15N] alanine was formed, indicating that pyruvic acid is an amino group acceptor in P. acidilactici. The present study shows that Aat plays a crucial role in PLA/HPLA biosynthesis and pyruvic acid is an amino acceptor in transamination reactions in P. acidilactici

Faint Infrared Flares from the Microquasar GRS 1915+105

We present simultaneous infrared and X-ray observations of the Galactic microquasar GRS 1915+105 using the Palomar 5-m telescope and Rossi X-ray Timing Explorer on July 10, 1998 UT. Over the course of 5 hours, we observed 6 faint infrared (IR) flares with peak amplitudes of $\sim 0.3-0.6$ mJy and durations of $\sim 500-600$ seconds. These flares are associated with X-ray soft-dip/soft-flare cycles, as opposed to the brighter IR flares associated with X-ray hard-dip/soft-flare cycles seen in August 1997 by Eikenberry et al. (1998). Interestingly, the IR flares begin {\it before} the X-ray oscillations, implying an ``outside-in'' origin of the IR/X-ray cycle. We also show that the quasi-steady IR excess in August 1997 is due to the pile-up of similar faint flares. We discuss the implications of this flaring behavior for understanding jet formation in microquasars.Comment: 10 pages, 4 figures Accepted for publication in ApJ Letter

Unique Resistance of I/LnJ Mice to a Retrovirus Is Due to Sustained Interferon γ–dependent Production of Virus-neutralizing Antibodies

Selection of immune escape variants impairs the ability of the immune system to sustain an efficient antiviral response and to control retroviral infections. Like other retroviruses, mouse mammary tumor virus (MMTV) is not efficiently eliminated by the immune system of susceptible mice. In contrast, MMTV-infected I/LnJ mice are capable of producing IgG2a virus-neutralizing antibodies, sustain this response throughout their life, and secrete antibody-coated virions into the milk, thereby preventing infection of their progeny. Antibodies were produced in response to several MMTV variants and were cross-reactive to them. Resistance to MMTV infection was recessive and was dependent on interferon (IFN)-γ production, because I/LnJ mice with targeted deletion of the INF-γ gene failed to produce any virus-neutralizing antibodies. These findings reveal a novel mechanism of resistance to retroviral infection that is based on a robust and sustained IFN-γ–dependent humoral immune response

Absence of up-regulation for a proliferation-inducing ligand in Sjögren's sialadenitis lesions

Objective. To determine whether a proliferation-inducing ligand (APRIL) has a role in the survival of plasma cells infiltrating salivary glands from SS patients. Methods. We performed immunological staining for APRIL in minor salivary glands from SS with a pair of antibodies specifically recognizing APRIL-producing cells and secreted APRIL. Results. Despite high leucocyte infiltration, APRIL-producing cells, identified as neutrophils, were rare in SS salivary glands. Keratinocytes from the adjacent oral epithelium also produced APRIL, but we never detected significant levels of secreted APRIL in SS salivary glands. We obtained similar results with B-cell lymphomas associated with SS. In fact, there was no significant difference in APRIL production and the level of secreted APRIL in these pathological samples compared with normal corresponding tissues. Conclusion. The combined observation that APRIL production is not up-regulated in lesions from SS patients, and that secreted APRIL is not retained in these lesions, indicates that plasma cells frequently present in SS lesions may not rely on APRIL for survival, as they do in other rheumatic disease