Christopher E. Mason

Doron Betel

Franck Rapaport

Nicholas D. Socci

Raya Khanin

PubMed

Springer - Publisher Connector

Response to Zhou and Robinson

Crossref

CORRESPONDENCE Open AccessResponse to Zhou and RobinsonDoron Betel1*, Nicholas D. Socci2, Raya Khanin2, Christopher E. Mason1 and Franck Rapaport2The choice of algorithmic tool for analysis of genomicdata is often guided by personal preference and conveni-ence rather than clear scientific criteria. In a recentpaper (Rapaport et al. Genome Biol. 2013;14:R95) weperformed an objective analysis of widely used RNA-seqdifferential expression (DE) programs in order to under-stand the performance of the various tools with respectto different evaluation measures. One such measure wasthe interpretation of DE when there is no detectable sig-nal from one of the contrasting conditions. Zhou andRobinson present insightful reanalysis of this compari-son. They present two claims. The first is that there wasa coding error when calculating the normalized edgeRcount values. Second, that our use of signal-to-noiseratio (S/N) as a measure of sensitivity and specificity(in the form of ROC) is incorrect. This is because themean and variance are correlated in count data andtherefore the signal (mean) and noise (standard devi-ation) are not two independent measures. Perhaps,more importantly, the ROC analysis was not based onthe same set of truth set for all methods since thesame S/N threshold value corresponds to very differ-ent genes sets for each method.The authors are correct in both these points and weappreciate the effort invested in bringing those to light.Our coding error was omitting scaling by library sizewhen calculating the normalized edgeR values. However,this error has very little impact on the analysis as pre-sented in Fig. 4 of our manuscript and does not changeits conclusions(see erratum to the manuscript).The most salient point raised is the use of the sameS/N threshold in the ROC evaluation. The main argumentis that the scale of mean and variance is different betweenthe different methods and therefore the methods are notevaluating the same sets of genes. That is correct and weagree with these points. The second point raised is thatsince the mean and variance are highly correlated in countdata calculating S/N using transformed mean variancevalues is more appropriate (Fig. 2 in Zhou and Robinson’sresponse). It is notable that the ROC performances ofedgeR and DESeq2 relative to limma in Fig. 2d are similarto those presented in Fig. 4b of Rapaport et al. suggestingthat selecting a fixed set of labels is not the major reasonfor the differences in performances. The improvement inthe ROC performance for edgeR (and DESeq2) relative tolimma is attributed to the transformation of the meanvariance values (Fig. 2b and c).The ROC analysis is only one part of much broaderanalyses of this subset of genes. Our analysis was origin-ally motivated by the observation that often genes withsimilar mean expression values, in the condition inwhich they are expressed, had different P values. Thiswas contrary to our initial expectation, since these geneshad similar mean expression values and therefore similarlog expression ratios we anticipated similar P values.Moreover, we noticed that some genes with higher ex-pression values had less significant P values than geneswith lower expression values (and similar standard devi-ation). This introduces a problem when selecting genesfor subsequent expression validation and functional im-portance. We were interested in characterizing this fur-ther and gain a better understanding of this discrepancy.As noted in both Rapaport et al. and in the responsefrom Zhou and Robinson, most of these genes areclearly differentially expressed and are correctly detectedas such by all methods. Hence the challenge is not incorrect identification of DE (that is, rejecting the nullhypothesis) but rather meaningful ordering of the DEgenes. Therefore, ranking this specific set of genes bytheir P values of differential expression indicates theconfidence in experimental measurement rather than re-jection of the null hypothesis (which is trivial in this casesince there is a clear change in expression). In our paper(Fig. 4a) we show that limma and PoissonSeq packageshave a clear monotonic relationship between the S/Nand P value assigned to difference in expression for thisparticular subset of genes with zero value in one con-dition. In contrast, DESeq, edgeR, and cuffdiff did notdisplay this monotonic relationship. It would be inter-esting to see the impact of mean variance scaling onthis correlation.* Correspondence: dob2014@med.cornell.edu1Weill Cornell Medical College, 1305 York Avenue, New York, NY 10021, USAFull list of author information is available at the end of the article© 2015 Betel et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.Betel et al. Genome Biology  (2015) 16:223 DOI 10.1186/s13059-015-0782-2Competing interestsThe authors declare that they have no competing interests.Authors' contributionDB wrote the response with contribution from NC. RK, CM and FR providedinput. All authors contributed and were authors of the orignal publication.All authors read and approved the final responseAuthor details1Weill Cornell Medical College, 1305 York Avenue, New York, NY 10021, USA.2Memorial Sloan Kettering Cancer Center, 1275 York Avenue, New York, NY10065, USA.Received: 8 September 2015 Accepted: 15 September 2015Submit your next manuscript to BioMed Centraland take full advantage of: • Convenient online submission• Thorough peer review• No space constraints or color ﬁgure charges• Immediate publication on acceptance• Inclusion in PubMed, CAS, Scopus and Google Scholar• Research which is freely available for redistributionSubmit your manuscript at www.biomedcentral.com/submitBetel et al. Genome Biology  (2015) 16:223 Page 2 of 2

https://genomebiology.biomedcentral.com/track/pdf/10.1186/s13059-015-0782-2?site=genomebiology.biomedcentral.com

Response to Zhou and Robinson

Abstract

Similar works

Full text

Available Versions

Springer - Publisher Connector

Crossref