We describe why the Choe et al. control dataset should not be used to assess GeneChip expression measures

Cope, Leslie

Irizarry, Rafael A

Wu, Zhijin

Collection Of Biostatistics Research Archive

Johns Hopkins University, Dept. of Biostatistics Working Papers3-17-2006FEATURE-LEVEL EXPLORATION OF THECHOE ET AL. AFFYMETRIX GENECHIPCONTROL DATASETRafael A. IrizarryJohns Hopkins Bloomberg School of Public Health, Department of Biostatistics, rafa@jhu.eduLeslie CopeJohns Hopkins University School of Medicine, Oncology Center, Lcope1@jhmi.eduZhijin WuBrown University, Center for Statistical Science, Zhijin_Wu@brown.eduThis working paper is hosted by The Berkeley Electronic Press (bepress) and may not be commercially reproduced without the permission of thecopyright holder.Copyright © 2011 by the authorsSuggested CitationIrizarry, Rafael A.; Cope, Leslie; and Wu, Zhijin, "FEATURE-LEVEL EXPLORATION OF THE CHOE ET AL. AFFYMETRIXGENECHIP CONTROL DATASET" (March 2006). Johns Hopkins University, Dept. of Biostatistics Working Papers. Working Paper102.http://biostats.bepress.com/jhubiostat/paper102Feature-level exploration of the Choe et al. Affymetrix GeneChipcontrol datasetRafael A. Irizarry, Leslie Cope, and Zhijin WuHosted by The Berkeley Electronic PressHeadlineWe describe why the Choe et al. control dataset should not be used to assess GeneChip expression measures.IntroductionIn [1] a spike-in experiment is described which the authors use to compare expression measures for AffymetrixGeneChip technology. Two sets of triplicates were created to represent control (C) and experimental (S)samples. In [2] and [3] we describe a benchmark for such measures based on experiments developed byAffymetrix and a GeneLogic. These datasets are described in detail in [2]. A web-based implementation ofthe benchmark, is available at affycomp.biostat.jhsph.edu. There are various inconsistencies betweenthe conclusions reached by [1] and [3]. In this letter we describe certain characteristics of the feature-leveldata produced by [1] which we believe explain these inconsistencies. These can be divided into 1) inducedby the experimental design and 2) an artifact.Experimental designThere are three characteristics of the experimental design described by [1] make the resulting data inappro-priate for assessment. Below we enumerate these problems and explain how they lead to unfair assessments.Other problems with the experimental design are described by [4].1. The spike-in concentrations are unrealistically high. In [3] we demonstrate that background noisemakes it harder to detect differentially expression for genes that are present in low concentrations. In [3] wepoint out that in the Affymetrix spike-in experiments the concentrations for spiked-in features are artificiallyhigh but that a large number of these are actually in a usable range (See Figure 1A). Figure 1B demonstratesthat in a typical experiment, features related to differentially expressed genes show intensities with a similarrange as the rest of the genes. However, Figures 1C-D suggest that none of the genes spiked-in by [1] are ina usable range since less than 1% of the data would reach the intensity levels seen for the spiked-in genes.This implies that expression measure assessments based on this dataset only apply to unlikely situationswhere we expect differentially expressed genes to be in the top 1% of overall expression.2. A large percentage of the genes (about 10%) are spiked-in to be differentially expressed and allof these are expected to be up-regulated. This design makes this spike-in data very different from thoseproduced by typical experiments where at least one of the following assumptions is expected to hold: 1) asmall percentage of genes are differentially expressed, 2) there is a balance between up and down regulation,and 3) the gene expression distribution across arrays is roughly the same. Most preprocessing algorithmsimplement normalization routines motivated by one or more of these assumptions, thus we should not expectexisting expression measure methodology to perform well with the Choe et al. data.3. A careful look at Table 1 in [1] shows that nominal concentrations and fold change sizes are confounded.This is better demonstrated by a graphical representation (Figure 2). This problem will not permit usto distinguish ability to detect small fold changes from the ability to detect differential expression whenconcentration is low. [3] show why this distinction is important.The artifactFigure 1 are based on raw feature-level data. No preprocessing or normalization was performed. A typicalexpression data will produce MA-plots with most of the points in the lower-range of concentration and anexponential tapering as concentration grows. However, the Choe et. al data shows a second cluster centeredat a high concentration and a negative log ratio. Figure 3 reveals that the features spiked-in to be at 1:1ratios behave very differently from the features from non-spiked-in genes which should also have a nominallog fold change of 0. Without more sample preparation information it is impossible to give an explanation1http://biostats.bepress.com/jhubiostat/paper102for this artifact. This problem implies that what [1] define as false positive might in fact be true positives.Figure 3 shows that this problem persists even after quantile normalization [5]. This artifact makes thisdataset particularly difficult to use in assessment because the distinction between true and false positives isnot clear.Abbreviations• MA-plot - Log expression in treatment minus (M) log expression in control versus average (A) logexpression plot.AcknowledgmentsThe work of R.A.I. is partially funded by the National Institutes of Health Specialized Centers of ClinicallyOriented Research (SCCOR) translational research funds (212-2492 and 212-2496).References[1] Choe SE, Boutros M, Michelson AM, Church GM, Halfon MS: Preferred analysis methods forAffymetrix GeneChips revealed by a wholly defined control dataset. Genome Biol 2005,6(2):R16.[2] Cope L, Irizarry R, Jaffee H, Wu Z, Speed T: A benchmark for Affymetrix GeneChip expressionmeasures. Bioinformatics 2004, 20(3):323–331.[3] Irizarry R, Wu Z, Jaffee H: Comparison of Affymetrix GeneChip Expression Measures. Dept. ofbiostatistics working papers, Johns Hopkins University 2005. [www.bepress.com/jhubiostat/paper86].[4] Dabney A, Storey J: A re-analysis of the Choe et al. Affymetrix GeneChip control dataset.Submitted 2005.[5] Bolstad B, Irizarry R, A˚strand M, Speed T: A comparison of normalization methods for highdensity oligonucleotide array data based on variance and bias. Bioinfromatics 2003, 19(2):185–193.2Hosted by The Berkeley Electronic Press6 8 10 12 14−1.0−0.50.00.51.01.5A) Affymetrix Spike−InAM6 8 10 12 14−1.0−0.50.00.51.01.5B) Typical ExperimentAM6 8 10 12 14−1.0−0.50.00.51.01.5C) Choe et al. Spike−InAM6 8 10 12 140.000.100.200.30D) Densities for Choe et al.Log (base 2) intensityDensityFigure 1: A) For two sets of triplicates from the Affymetrix HGU133A spike-in experiment we calculated theaverage log ratio across the three comparisons (M) and the average log intensity (A) across all six arrays foreach feature. This figure shows M plotted against A. However, because there are hundreds of thousands offeature, instead of plotting each point, we use shades of blue to denote the amount of points in each regionof the plot. About 90% of the data is contained in the dark blue regions. 405 features from the 36 geneswith nominal fold-changes of 2 are shown as orange points. B) As Figure A) but using two sets of biologicaltriplicates from a study comparing three trisomic human brains to three normal human brains. The orangedots are 385 features representing 35 genes on chromosome 21 for which we expect fold changes of 3/2. C)As A) but showing the two sets of triplicates described by Choe et al. The orange dots are 375 featuresrandomly sampled from those that were spiked-in to have fold changes greater than 1. The yellow ellipseis used to illustrate an artifact: among the data with nominal fold changes of 1, there appears to be twoclusters having different overall observed log ratios. D) Empirical density estimators for the log intensitydata, in one of S samples in the Choe el al data, for non-spiked in features (solid/gray), features spiked-inbut to be at a 1:1 ratio (dashed/purple), and features spiked-in to have nominal fold changes greater than1 (dotted/orange).3http://biostats.bepress.com/jhubiostat/paper1020.2 0.4 0.6 0.8 1.0 1.2−0.50.00.51.01.52.02.5concentration in C chiplog 2 (assigned fold change)differential expressionconstant expresssionsize of circle indicate number of probesetsFigure 2: Graphical representation of Table 1 in [1]. The coordinates of the points represent nominal foldchanges and concentrations that were used. The size of the points are proportional to the number of geneshaving those values.4Hosted by The Berkeley Electronic Press−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−2−1012A)            Replicate 1                       Replicate 2                       Replicate 3                MNo SpikeInFC=1Small FCMedium FCHigh FCNo SpikeInFC=1Small FCMedium FCHigh FCNo SpikeInFC=1Small FCMedium FCHigh FC−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−2−1012B)            Replicate 1                       Replicate 2                       Replicate 3                MNo SpikeInFC=1Small FCMedium FCHigh FCNo SpikeInFC=1Small FCMedium FCHigh FCNo SpikeInFC=1Small FCMedium FCHigh FCFigure 3: A) For the raw probe-level data in [1] we computed log fold changes comparing the control andspike-in arrays for each of the three replicates. The C and S arrays were paired according to their filenames:C1-S1, C2-S2, and C3-S3. Boxplots are shown for five groups of probes: not spiked-in (gray), spiked-in atequal concentrations (purple), spiked-in with nominal fold-changes between 1 and 2, 2 and 3, and 3 and 4(orange). B) As A) but after quantile normalizing the probes.5http://biostats.bepress.com/jhubiostat/paper102

FEATURE-LEVEL EXPLORATION OF THE CHOE ET AL. AFFYMETRIX GENECHIP CONTROL DATASET

http://biostats.bepress.com/cgi/viewcontent.cgi?article=1102&context=jhubiostat

FEATURE-LEVEL EXPLORATION OF THE CHOE ET AL. AFFYMETRIX GENECHIP CONTROL DATASET

Abstract

Similar works

Full text

Available Versions

Collection Of Biostatistics Research Archive