We are now well into the sequencing era of genetic analysis, and methods to investigate rare variants associated with disease remain in high demand. Currently, the more common rare variant analysis methods are burden tests
and variance component tests. This report introduces a burden test known as the modified replication based sum statistic and evaluates its performance, and the performance of other common burden and variance component tests under the setting of a small sample size (103 total cases and controls) using the Genetic Analysis Workshop 18 simulated data with complete knowledge of the simulation model. Specifically we look at the variable threshold sum statistic, replication-based sum statistics, the C-alpha, and sequence kernel association test. Using minor allele frequency thresholds of less than 0.05, we find that the modified replication based sum statistic is competitive with all methods and that using 103 individuals leads to all methods being vastly underpowered. Much larger sample sizes are needed to confidently find truly associated genes

Ionita-Laza, Iuliana

Kim, Taebeom

Niu, Jiangong

Shete, Sanjay

Swartz, Michael

Yu, Robert

BMC Proceedings

PubMed

Columbia University Academic Commons

PROCEEDINGS Open AccessSmall sample properties of rare variant analysismethodsMichael D Swartz1,3*, Taebeom Kim1, Jiangong Niu2, Robert K Yu3, Sanjay Shete3, Iuliana Ionita-Laza4From Genetic Analysis Workshop 18Stevenson, WA, USA. 13-17 October 2012AbstractWe are now well into the sequencing era of genetic analysis, and methods to investigate rare variants associatedwith disease remain in high demand. Currently, the more common rare variant analysis methods are burden testsand variance component tests. This report introduces a burden test known as the modified replication based sumstatistic and evaluates its performance, and the performance of other common burden and variance componenttests under the setting of a small sample size (103 total cases and controls) using the Genetic Analysis Workshop18 simulated data with complete knowledge of the simulation model. Specifically we look at the variable thresholdsum statistic, replication-based sum statistics, the C-alpha, and sequence kernel association test. Using minor allelefrequency thresholds of less than 0.05, we find that the modified replication based sum statistic is competitive withall methods and that using 103 individuals leads to all methods being vastly underpowered. Much larger samplesizes are needed to confidently find truly associated genes.BackgroundWe are now well into the sequencing era of genetic analy-sis, and methods to investigate rare variants associatedwith disease remain in high demand. The typical methodsfor detecting single variants associated with disease are notsuited for the rare variants, owing to the low minor allelefrequency (MAF) of rare variants. Most rare variant analy-sis methods fall into two categories, burden tests [1] andvariance component tests [2]. Both of these classes of testsfind association between rare variants and disease by pool-ing rare variants in a defined region in some sense.Whereas burden tests pool the change in risk (positive ornegative) caused by total rare variants in a region, variancecomponent methods compare the distribution of rarevariant counts with that expected under the null.We begin by briefly summarizing some of the morecommon and more recent methods for rare variant ana-lyses and give their primary references. We consider thefollowing burden tests: the variable threshold sum statistic(VT) [3] and the replication-based weighted sum statistic(RBS) [4] and the following variance component tests:C-alpha [5] and the sequence kernel association test(SKAT) for rare variants [2]. The variable threshold sumstatistic computes a score for each region or gene basedon a weighted sum of variant counts. The weights aredefined by a combination of variant importance score, orfrequency weight, and allow the minor allele threshold tovary according to locus to define the rare variant [3]. TheVT method assumes the variants in each gene affect riskin the same direction. RBS constructs a weighted sum ofrare variant scores, defining weights as a function of theminor allele being more frequent in the cases or controlsand constructing the appropriate weighted sum score ofthose variants overrepresented in the controls (S+) orcases (S-) to accommodate both protective or deleteriousrare variants within the same gene [4]. Then the final sta-tistic to assess the association of rare variants to diseasecan be constructed in one of two ways. Take the maxi-mum of the weighted sum of the variants more frequentin the cases and the weighted sum in the controls (Smax =max[S-,S+]) or sum the weighted sums (Scomb = S- + S+)[4]. In this paper, we propose an additional RBS that com-bines S- and S+ based on the data. The C-alpha statistic isa variance component method that compares the* Correspondence: Michael.D.Swartz@uth.tmc.edu1Division of Biostatistics, University of Texas School of Public Health,Houston, TX 77025, USAFull list of author information is available at the end of the articleSwartz et al. BMC Proceedings 2014, 8(Suppl 1):S13http://www.biomedcentral.com/1753-6561/8/S1/S13© 2014 Swartz et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative CommonsAttribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction inany medium, provided the original work is properly cited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.distribution of the rare variants in the sample with thatexpected under the null, assuming a binomial distributionof each rare variant [5]. And finally, SKAT uses a linearmodel type approach to construct an aggregate weightedscore test statistic for variants in a given region of interest[2]. Both C-alpha and SKAT can accommodate both riskand protective variants in the same gene.Additionally, a major issue faced with researchersinvestigating rare variants, beyond the low MAF, is sam-ple size considerations. Although some consortia exist,providing larger sample sizes for rare variant analyses,many researchers following up genome-wide associationstudies (GWAS) do not have the resources to sequencethe full GWAS sample, leading to sample sizes on theorder of only a few hundred for rare variant analysis.One recent study used 40 individuals in total [6].As with all methods, larger samples give more power,and limited sample size limits the power. With rare var-iants, this is compounded because only a few individualsin the sample will exhibit a rare variant. For smallersamples, the MAF of the rare variant must be larger toeven have a chance of appearing in the sample. Butwould a rare variant analysis still be useful for the rarevariants that do appear in the sample? And whichmethod would be effective in detecting the rare variantswith smaller sample size? In addition to examining ournew method, we investigate these questions using thesimulated data of the Genetic Analysis Workshop 18(GAW18) (see [7] for full details).MethodsIn our study, we propose a modified RBS. We compareits performance and that of the burden and variancecomponent tests mentioned earlier. The details of ourmodified RBS follow.The modified replication-based weighted sum statistic:StauWe follow the notation from [4]. Let nk′k+denote thenumber of variants in group (k’,k) where k’ denotescopies of the minor allele that appear in the cases and kdenotes copies of the minor alleles that appear in thecontrols, and k’ > k. Let nk′k− be defined as a similarcount, except k’ < k. We define S+ and S- as in [4],which are the two statistics for variants with k’ > k andk’ < k, respectively. Then we can define Sτ as theweighted sum of weighted sums: Stau = τS+ + (1 − τ )S− ,where τ =∑k′>knk′k+∑k′>knk′k++∑k′<knk′k−. Defining τ in this way com-bines the concepts touched on in the Smax and Scomb.Smax models the scenario of extremes, either all the rarevariants are risk or all are protective, while the Scombassumes equal risk and protective rare variants. Stauallows the data to weight the impact of risk and protec-tive variants according to the data, modeling unequalprotective and risk variants in the combined statistic.Data preparation and phenotype definitionBecause the methods detailed are designed for case-control data, we focus on the unrelated individuals.Using all of the longitudinal data, we define as a caseany individual who became hypertensive over the courseof observation, and we define as a control any individualwho did not exhibit signs of hypertension. Across the200 replicates, we have an average of 48 controls and 55cases for a total of 103 unrelated individuals. (Of thetotal 159 unrelated individuals, 103 had all the informa-tion needed for this analysis.) Because we are comparingthe performance of methods on the same data, wefocused on genes on chromosome 3 that were used inthe simulation model. We used NCBI dbSNP to identifyall the single-nucleotide polymorphisms (SNPs) belong-ing to each gene on chromosome 3 involved in thesimulation model.We evaluated type 1 error, based on a resamplingapproach, specifically, we simulated a dichotomous phe-notype to be independent of the genotype following themethod in [8]. We simulate a Bernoulli random variablewith event probability 0.5. If the variable = 1, we changethe original phenotype to the alternate group, and if thevariable =0, we keep the original phenotype status.Data analysisFor the burden tests, we computed a burden statistic foreach gene. We computed Smax, Scomb, Stau, C-alpha, andVT for chromosome 3 for each replicate using the simu-lated phenotype. We also used the freely availableR-package for computing SKAT using the default valueswith the small sample size option [2]. For each replicate,we recorded whether each method declared one of thesolution genes as significant and reported the power foreach method as the proportion of the 200 replicateswhere each method identified the gene as associatedwith the disease. We focused our analysis on rarer SNPswith MAFs less than 5%.ResultsThe signal strength for each gene on chromosome3 ranged from 0% to 4.5% (the number of causal rare var-iants divided by the total rare variants). The actual rawnumber of causal variants per gene ranged from 0 to 9,and the number of total variants per gene ranged from 5to 2531. Thus, the total number of rare variants (denomi-nator) controlled the signal to noise ratio (see Figure 1,horizontal axis). To compare the performance of eachSwartz et al. BMC Proceedings 2014, 8(Suppl 1):S13http://www.biomedcentral.com/1753-6561/8/S1/S13Page 2 of 5method on the nongenetic dichotomous phenotype, thetype 1 error was controlled at the 0.05 level. Across all 200simulated replicates, the average proportion of significancefor all genes for each method was 0.05.The modified replication based sum statistic, Stauperforms competitively, especially compared with theother replication based statistics, Smax and Scomb,within 5% power of each other. In fact, most of the 6methods performed similarly for each gene, (within5%). Power exceeded 10% for only 9 of the 30 causalgenes: ARHGEF3, FBLN2, GPR160, PAK2, PTPLB(SKAT only), RAD18 (VT only), RYBP (VT only)SERP1, TFDP2, and TUSC2 (Figure 1). Additionally,for some genes (GRP160, SERP1, TFDP2, TUSC2), theVT method is substantially more powerful than theother methods, but for others (e.g., SEMA3F), it is lesspowerful. Regardless of the variance in power acrossmethods across genes, when averaging each method’spower across all genes, the averages were close to 0.05(Table 1).Figure 1 Power for analysis with minor allele frequency (MAF) less than 0.05. This figure shows the power of each method to detect eachgene as causal for the 200 replicates of the simulation data when including only rarer variants with empirical MAF less than 0.05. The methodsperform similarly within each gene. Especially note the performance of the modified replication based sum statistic Stau (solid black bar) relativeto its predecessors Scomb and Smax. For these data, all 3 methods are very competitive. Below the horizontal axis is the percent signal for eachgene (percent of causal rare variants divided by the total number of rare variants multiplied by 100).Table 1 Mean power for each method across all genesMethod Hypertension NongeneticStau 0.056 0.048Scomb 0.083 0.049Smax 0.055 0.046C-alpha 0.043 0.050VT 0.069 0.050SKAT 0.051 0.055This table reports the mean power for each of the 6 methods investigated forthose variants with minor allele frequencies less than 0.05. We report thepower for the simulated hypertension phenotype as well as for a nongeneticphenotype for type 1 error.SKAT, sequence kernel association test; VT, variable threshold.Swartz et al. BMC Proceedings 2014, 8(Suppl 1):S13http://www.biomedcentral.com/1753-6561/8/S1/S13Page 3 of 5DiscussionThe new method we introduced in this paper, Stau, per-forms competitively with other popular methods forrare variant analysis. It does not exhibit false positivesmuch higher than the other methods, nor does it standalone in its detecting or failing to detect the causal var-iants. However, given the small sample size, it is difficultto identify any new advantages or disadvantages of thismodified replication-based method.In fact, all methods investigated here are underpow-ered for this case-control study with a sample size of103 individuals. These methods fail to detect the causalrare variants most of the time (Figure 1). Even morecomplex, there is not an easily identified pattern orrationale to when these methods do well for this samplesize. For instance, all methods exhibited poor power(<0.05) to detect MAP4, which was simulated to be oneof the top 15 genes with the largest effect sizes for anygene on chromosome 3. And yet TUSC2 has only onecausal variant according to the simulation model para-meters, whose minor allele was not present in our sam-ple. As a gene, it is much smaller with only 19 rarevariants compared with MAP4 with more than 400 var-iants. Yet it only has a few variants analyzed, and 2 ofthem are in linkage disequilibrium (LD) (R2 >0.25) witha causal variant in MAP4 that has a large effect size forboth diastolic blood pressure (DBP) and systolic bloodpressure (SBP). Because this is a gene detected by allmethods with power greater than 0.2, the power couldbe driven through the LD with the large effect size com-bined with the small number of total variants includedin the weighted sum. Beyond gene size and LD, using adichotomous hypertension variable rather than the con-tinuous trait could have cost power and added complex-ity to determine the efficacies of these methods withsuch a small sample size versus using SBP or DBP as acontinuous trait.ConclusionsThe main conclusion of this paper is that for all meth-ods, 103 individuals are not enough for a rare variantanalysis of a complex qualitative disease such as hyper-tension. Perhaps power can be increased using bloodpressure as a continuous trait rather than treatinghypertension status as a qualitative trait. A secondaryconclusion from this report is that the new method Stauperforms similar to its predecessors (Scomb and Smax). Itis worth further investigation to more clearly determineany advantages for using Stau.Competing interestsThe authors declare that they have no competing interests.Authors’ contributionsMDS designed the overall study, conceived of Stau, performed someanalyses, and wrote the manuscript. TK refined Stau; wrote computer code tocompute Stau, Smax, Scomb, and a script to perform all analyses; and providededitorial input on the manuscript. JN wrote computer code to compute VTand C-alpha and provided editorial input on the manuscript. RKY performeddata cleaning and preliminary analyses and provided editorial input on themanuscript. SS consulted on the analysis and interpretation of results andprovided editorial input on the manuscript II-L provided consultation for thecomputer code for Stau, Smax, and Scomb; analysis and interpretation ofresults; and editorial input on the manuscript. All authors read and approvedthe final manuscript.AcknowledgementsWe would like to acknowledge the funding sources. This work was fundedin part by National Cancer Institute (NCI) award K07-CA 123109 and R03CA141998 (MDS PI). The content is solely the responsibility of the authorsand does not necessarily represent the official views of the NCI or theNational Institutes of Health (NIH). The funders had no role in study design,data collection and analysis, decision to publish, or preparation of themanuscript. The GAW18 whole genome sequence data were provided bythe T2D-GENES (Type 2 Diabetes Genetic Exploration by Next-generationsequencing in Ethnic Samples) Consortium, which is supported by NIHgrants U01 DK085524, U01 DK085584, U01 DK085501, U01 DK085526, andU01 DK085545. The other genetic and phenotypic data for GAW18 wereprovided by the San Antonio Family Heart Study and San Antonio FamilyDiabetes/Gallbladder Study, which are supported by NIH grants P01HL045222, R01 DK047482, and R01 DK053889. The Genetic AnalysisWorkshop is supported by NIH grant R01 GM031575.This article has been published as part of BMC Proceedings Volume 8Supplement 1, 2014: Genetic Analysis Workshop 18. The full contents of thesupplement are available online at http://www.biomedcentral.com/bmcproc/supplements/8/S1. Publication charges for this supplement were funded bythe Texas Biomedical Research Institute.Authors’ details1Division of Biostatistics, University of Texas School of Public Health,Houston, TX 77025, USA. 2Department of Breast Medical Oncology,University of Texas MD Anderson Cancer Center, Houston, TX 77030, USA.3Department of Biostatistics, University of Texas MD Anderson CancerCenter, Houston, TX 77030, USA. 4Department of Biostatistics, ColumbiaUniversity, Mailman School of Public Health, New York, NY 10032, USA.Published: 17 June 2014References1. Morris AP, Zeggini E: An evaluation of statistical approaches to rarevariant analysis in genetic association studies. Genet Epidemiol 2010,34:188-193.2. Wu MC, Lee S, Cai T, Li Y, Boehnke M, Lin X: Rare-variant associationtesting for sequencing data with the sequence kernel association test.Am J Hum Genet 2011, 89(1):82-93.3. Price AL, Kryukov GV, de Bakker PI, Purcell SM, Staples J, Wei LJ, Sunyaev SR:Pooled association tests for rare variants in exon-resequencing studies.Am J Hum Genet 2010, 86:832-838.4. Ionita-Laza I, Buxbaum JD, Laird NM, Lange C: A new testing strategy toidentify rare variants with either risk or protective effect on disease.PLoS Genet 2011, 7:e1001289.5. Neale BM, Rivas MA, Voight BF, Altshuler D, Devlin B, Orho-Melander M,Kathiresan S, Purcell SM, Roeder K, Daly MJ: Testing for an unusualdistribution of rare variants. PLoS Genet 2011, 7:e1001322.6. Audo I, Bujakowska K, Orhan E, Poloschek CM, Defoort-Dhellemmes S,Drumare I, Kohl S, Luu TD, Lecompte O, Zrenner E, et al: Whole-exomesequencing identifies mutations in GPR179 leading to autosomal-recessive complete congenital stationary night blindness. Am J HumGenet 2012, 90:321-330.7. Almasy L, Dyer T, Peralta J, Jun G, Fuchsberger C, Almeida M, Kent JW Jr,Fowler S, Duggirala R, Blangero J: Data for Genetic Analysis Workshop 18:human whole genome sequence, blood pressure, and simulatedphenotypes in extended pedigrees. BMC Proc 2014, 8(suppl 2):S2.Swartz et al. BMC Proceedings 2014, 8(Suppl 1):S13http://www.biomedcentral.com/1753-6561/8/S1/S13Page 4 of 58. Swartz MD, Yu RK, Shete S: Finding factors influencing risk: comparingBayesian stochastic search and standard variable selection methodsapplied to logistic regression models of cases and controls. Stat Med2008, 27:6158-6174.doi:10.1186/1753-6561-8-S1-S13Cite this article as: Swartz et al.: Small sample properties of rare variantanalysis methods. BMC Proceedings 2014 8(Suppl 1):S13.Submit your next manuscript to BioMed Centraland take full advantage of: • Convenient online submission• Thorough peer review• No space constraints or color figure charges• Immediate publication on acceptance• Inclusion in PubMed, CAS, Scopus and Google Scholar• Research which is freely available for redistributionSubmit your manuscript at www.biomedcentral.com/submitSwartz et al. BMC Proceedings 2014, 8(Suppl 1):S13http://www.biomedcentral.com/1753-6561/8/S1/S13Page 5 of 5

Small sample properties of rare variant analysis methods

Michael D Swartz

Taebeom Kim

Jiangong Niu

Robert K Yu

Sanjay Shete

Iuliana Ionita-Laza

Crossref

Springer - Publisher Connector

PROCEEDINGS Open Access
Small sample properties of rare variant analysis
methods
Michael D Swartz1,3*, Taebeom Kim1, Jiangong Niu2, Robert K Yu3, Sanjay Shete3, Iuliana Ionita-Laza4
From Genetic Analysis Workshop 18
Stevenson, WA, USA. 13-17 October 2012
Abstract
We are now well into the sequencing era of genetic analysis, and methods to investigate rare variants associated
with disease remain in high demand. Currently, the more common rare variant analysis methods are burden tests
and variance component tests. This report introduces a burden test known as the modified replication based sum
statistic and evaluates its performance, and the performance of other common burden and variance component
tests under the setting of a small sample size (103 total cases and controls) using the Genetic Analysis Workshop
18 simulated data with complete knowledge of the simulation model. Specifically we look at the variable threshold
sum statistic, replication-based sum statistics, the C-alpha, and sequence kernel association test. Using minor allele
frequency thresholds of less than 0.05, we find that the modified replication based sum statistic is competitive with
all methods and that using 103 individuals leads to all methods being vastly underpowered. Much larger sample
sizes are needed to confidently find truly associated genes.
Background
We are now well into the sequencing era of genetic analy-
sis, and methods to investigate rare variants associated
with disease remain in high demand. The typical methods
for detecting single variants associated with disease are not
suited for the rare variants, owing to the low minor allele
frequency (MAF) of rare variants. Most rare variant analy-
sis methods fall into two categories, burden tests [1] and
variance component tests [2]. Both of these classes of tests
find association between rare variants and disease by pool-
ing rare variants in a defined region in some sense.
Whereas burden tests pool the change in risk (positive or
negative) caused by total rare variants in a region, variance
component methods compare the distribution of rare
variant counts with that expected under the null.
We begin by briefly summarizing some of the more
common and more recent methods for rare variant ana-
lyses and give their primary references. We consider the
following burden tests: the variable threshold sum statistic
(VT) [3] and the replication-based weighted sum statistic
(RBS) [4] and the following variance component tests:
C-alpha [5] and the sequence kernel association test
(SKAT) for rare variants [2]. The variable threshold sum
statistic computes a score for each region or gene based
on a weighted sum of variant counts. The weights are
defined by a combination of variant importance score, or
frequency weight, and allow the minor allele threshold to
vary according to locus to define the rare variant [3]. The
VT method assumes the variants in each gene affect risk
in the same direction. RBS constructs a weighted sum of
rare variant scores, defining weights as a function of the
minor allele being more frequent in the cases or controls
and constructing the appropriate weighted sum score of
those variants overrepresented in the controls (S+) or
cases (S-) to accommodate both protective or deleterious
rare variants within the same gene [4]. Then the final sta-
tistic to assess the association of rare variants to disease
can be constructed in one of two ways. Take the maxi-
mum of the weighted sum of the variants more frequent
in the cases and the weighted sum in the controls (Smax =
max[S-,S+]) or sum the weighted sums (Scomb = S- + S+)
[4]. In this paper, we propose an additional RBS that com-
bines S- and S+ based on the data. The C-alpha statistic is
a variance component method that compares the
* Correspondence: Michael.D.Swartz@uth.tmc.edu
1Division of Biostatistics, University of Texas School of Public Health,
Houston, TX 77025, USA
Full list of author information is available at the end of the article
Swartz et al. BMC Proceedings 2014, 8(Suppl 1):S13
http://www.biomedcentral.com/1753-6561/8/S1/S13
© 2014 Swartz et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons
Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in
any medium, provided the original work is properly cited. The Creative Commons Public Domain Dedication waiver (http://
creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
distribution of the rare variants in the sample with that
expected under the null, assuming a binomial distribution
of each rare variant [5]. And finally, SKAT uses a linear
model type approach to construct an aggregate weighted
score test statistic for variants in a given region of interest
[2]. Both C-alpha and SKAT can accommodate both risk
and protective variants in the same gene.
Additionally, a major issue faced with researchers
investigating rare variants, beyond the low MAF, is sam-
ple size considerations. Although some consortia exist,
providing larger sample sizes for rare variant analyses,
many researchers following up genome-wide association
studies (GWAS) do not have the resources to sequence
the full GWAS sample, leading to sample sizes on the
order of only a few hundred for rare variant analysis.
One recent study used 40 individuals in total [6].
As with all methods, larger samples give more power,
and limited sample size limits the power. With rare var-
iants, this is compounded because only a few individuals
in the sample will exhibit a rare variant. For smaller
samples, the MAF of the rare variant must be larger to
even have a chance of appearing in the sample. But
would a rare variant analysis still be useful for the rare
variants that do appear in the sample? And which
method would be effective in detecting the rare variants
with smaller sample size? In addition to examining our
new method, we investigate these questions using the
simulated data of the Genetic Analysis Workshop 18
(GAW18) (see [7] for full details).
Methods
In our study, we propose a modified RBS. We compare
its performance and that of the burden and variance
component tests mentioned earlier. The details of our
modified RBS follow.
The modified replication-based weighted sum statistic:
Stau
We follow the notation from [4]. Let nk
′
k+
denote the
number of variants in group (k’,k) where k’ denotes
copies of the minor allele that appear in the cases and k
denotes copies of the minor alleles that appear in the
controls, and k’ > k. Let nk
′
k− be defined as a similar
count, except k’ < k. We define S+ and S- as in [4],
which are the two statistics for variants with k’ > k and
k’ < k, respectively. Then we can define Sτ as the
weighted sum of weighted sums: Stau = τS+ + (1 − τ )S− ,
where τ =
∑
k′>k
nk
′
k+
∑
k′>k
nk
′
k+
+
∑
k′<k
nk
′
k−
. Defining τ in this way com-
bines the concepts touched on in the Smax and Scomb.
Smax models the scenario of extremes, either all the rare
variants are risk or all are protective, while the Scomb
assumes equal risk and protective rare variants. Stau
allows the data to weight the impact of risk and protec-
tive variants according to the data, modeling unequal
protective and risk variants in the combined statistic.
Data preparation and phenotype definition
Because the methods detailed are designed for case-
control data, we focus on the unrelated individuals.
Using all of the longitudinal data, we define as a case
any individual who became hypertensive over the course
of observation, and we define as a control any individual
who did not exhibit signs of hypertension. Across the
200 replicates, we have an average of 48 controls and 55
cases for a total of 103 unrelated individuals. (Of the
total 159 unrelated individuals, 103 had all the informa-
tion needed for this analysis.) Because we are comparing
the performance of methods on the same data, we
focused on genes on chromosome 3 that were used in
the simulation model. We used NCBI dbSNP to identify
all the single-nucleotide polymorphisms (SNPs) belong-
ing to each gene on chromosome 3 involved in the
simulation model.
We evaluated type 1 error, based on a resampling
approach, specifically, we simulated a dichotomous phe-
notype to be independent of the genotype following the
method in [8]. We simulate a Bernoulli random variable
with event probability 0.5. If the variable = 1, we change
the original phenotype to the alternate group, and if the
variable =0, we keep the original phenotype status.
Data analysis
For the burden tests, we computed a burden statistic for
each gene. We computed Smax, Scomb, Stau, C-alpha, and
VT for chromosome 3 for each replicate using the simu-
lated phenotype. We also used the freely available
R-package for computing SKAT using the default values
with the small sample size option [2]. For each replicate,
we recorded whether each method declared one of the
solution genes as significant and reported the power for
each method as the proportion of the 200 replicates
where each method identified the gene as associated
with the disease. We focused our analysis on rarer SNPs
with MAFs less than 5%.
Results
The signal strength for each gene on chromosome
3 ranged from 0% to 4.5% (the number of causal rare var-
iants divided by the total rare variants). The actual raw
number of causal variants per gene ranged from 0 to 9,
and the number of total variants per gene ranged from 5
to 2531. Thus, the total number of rare variants (denomi-
nator) controlled the signal to noise ratio (see Figure 1,
horizontal axis). To compare the performance of each
Swartz et al. BMC Proceedings 2014, 8(Suppl 1):S13
http://www.biomedcentral.com/1753-6561/8/S1/S13
Page 2 of 5
method on the nongenetic dichotomous phenotype, the
type 1 error was controlled at the 0.05 level. Across all 200
simulated replicates, the average proportion of significance
for all genes for each method was 0.05.
The modified replication based sum statistic, Stau
performs competitively, especially compared with the
other replication based statistics, Smax and Scomb,
within 5% power of each other. In fact, most of the 6
methods performed similarly for each gene, (within
5%). Power exceeded 10% for only 9 of the 30 causal
genes: ARHGEF3, FBLN2, GPR160, PAK2, PTPLB
(SKAT only), RAD18 (VT only), RYBP (VT only)
SERP1, TFDP2, and TUSC2 (Figure 1). Additionally,
for some genes (GRP160, SERP1, TFDP2, TUSC2), the
VT method is substantially more powerful than the
other methods, but for others (e.g., SEMA3F), it is less
powerful. Regardless of the variance in power across
methods across genes, when averaging each method’s
power across all genes, the averages were close to 0.05
(Table 1).
Figure 1 Power for analysis with minor allele frequency (MAF) less than 0.05. This figure shows the power of each method to detect each
gene as causal for the 200 replicates of the simulation data when including only rarer variants with empirical MAF less than 0.05. The methods
perform similarly within each gene. Especially note the performance of the modified replication based sum statistic Stau (solid black bar) relative
to its predecessors Scomb and Smax. For these data, all 3 methods are very competitive. Below the horizontal axis is the percent signal for each
gene (percent of causal rare variants divided by the total number of rare variants multiplied by 100).
Table 1 Mean power for each method across all genes
Method Hypertension Nongenetic
Stau 0.056 0.048
Scomb 0.083 0.049
Smax 0.055 0.046
C-alpha 0.043 0.050
VT 0.069 0.050
SKAT 0.051 0.055
This table reports the mean power for each of the 6 methods investigated for
those variants with minor allele frequencies less than 0.05. We report the
power for the simulated hypertension phenotype as well as for a nongenetic
phenotype for type 1 error.
SKAT, sequence kernel association test; VT, variable threshold.
Swartz et al. BMC Proceedings 2014, 8(Suppl 1):S13
http://www.biomedcentral.com/1753-6561/8/S1/S13
Page 3 of 5
Discussion
The new method we introduced in this paper, Stau, per-
forms competitively with other popular methods for
rare variant analysis. It does not exhibit false positives
much higher than the other methods, nor does it stand
alone in its detecting or failing to detect the causal var-
iants. However, given the small sample size, it is difficult
to identify any new advantages or disadvantages of this
modified replication-based method.
In fact, all methods investigated here are underpow-
ered for this case-control study with a sample size of
103 individuals. These methods fail to detect the causal
rare variants most of the time (Figure 1). Even more
complex, there is not an easily identified pattern or
rationale to when these methods do well for this sample
size. For instance, all methods exhibited poor power
(<0.05) to detect MAP4, which was simulated to be one
of the top 15 genes with the largest effect sizes for any
gene on chromosome 3. And yet TUSC2 has only one
causal variant according to the simulation model para-
meters, whose minor allele was not present in our sam-
ple. As a gene, it is much smaller with only 19 rare
variants compared with MAP4 with more than 400 var-
iants. Yet it only has a few variants analyzed, and 2 of
them are in linkage disequilibrium (LD) (R2 >0.25) with
a causal variant in MAP4 that has a large effect size for
both diastolic blood pressure (DBP) and systolic blood
pressure (SBP). Because this is a gene detected by all
methods with power greater than 0.2, the power could
be driven through the LD with the large effect size com-
bined with the small number of total variants included
in the weighted sum. Beyond gene size and LD, using a
dichotomous hypertension variable rather than the con-
tinuous trait could have cost power and added complex-
ity to determine the efficacies of these methods with
such a small sample size versus using SBP or DBP as a
continuous trait.
Conclusions
The main conclusion of this paper is that for all meth-
ods, 103 individuals are not enough for a rare variant
analysis of a complex qualitative disease such as hyper-
tension. Perhaps power can be increased using blood
pressure as a continuous trait rather than treating
hypertension status as a qualitative trait. A secondary
conclusion from this report is that the new method Stau
performs similar to its predecessors (Scomb and Smax). It
is worth further investigation to more clearly determine
any advantages for using Stau.
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
MDS designed the overall study, conceived of Stau, performed some
analyses, and wrote the manuscript. TK refined Stau; wrote computer code to
compute Stau, Smax, Scomb, and a script to perform all analyses; and provided
editorial input on the manuscript. JN wrote computer code to compute VT
and C-alpha and provided editorial input on the manuscript. RKY performed
data cleaning and preliminary analyses and provided editorial input on the
manuscript. SS consulted on the analysis and interpretation of results and
provided editorial input on the manuscript II-L provided consultation for the
computer code for Stau, Smax, and Scomb; analysis and interpretation of
results; and editorial input on the manuscript. All authors read and approved
the final manuscript.
Acknowledgements
We would like to acknowledge the funding sources. This work was funded
in part by National Cancer Institute (NCI) award K07-CA 123109 and R03
CA141998 (MDS PI). The content is solely the responsibility of the authors
and does not necessarily represent the official views of the NCI or the
National Institutes of Health (NIH). The funders had no role in study design,
data collection and analysis, decision to publish, or preparation of the
manuscript. The GAW18 whole genome sequence data were provided by
the T2D-GENES (Type 2 Diabetes Genetic Exploration by Next-generation
sequencing in Ethnic Samples) Consortium, which is supported by NIH
grants U01 DK085524, U01 DK085584, U01 DK085501, U01 DK085526, and
U01 DK085545. The other genetic and phenotypic data for GAW18 were
provided by the San Antonio Family Heart Study and San Antonio Family
Diabetes/Gallbladder Study, which are supported by NIH grants P01
HL045222, R01 DK047482, and R01 DK053889. The Genetic Analysis
Workshop is supported by NIH grant R01 GM031575.
This article has been published as part of BMC Proceedings Volume 8
Supplement 1, 2014: Genetic Analysis Workshop 18. The full contents of the
supplement are available online at http://www.biomedcentral.com/bmcproc/
supplements/8/S1. Publication charges for this supplement were funded by
the Texas Biomedical Research Institute.
Authors’ details
1Division of Biostatistics, University of Texas School of Public Health,
Houston, TX 77025, USA. 2Department of Breast Medical Oncology,
University of Texas MD Anderson Cancer Center, Houston, TX 77030, USA.
3Department of Biostatistics, University of Texas MD Anderson Cancer
Center, Houston, TX 77030, USA. 4Department of Biostatistics, Columbia
University, Mailman School of Public Health, New York, NY 10032, USA.
Published: 17 June 2014
References
1. Morris AP, Zeggini E: An evaluation of statistical approaches to rare
variant analysis in genetic association studies. Genet Epidemiol 2010,
34:188-193.
2. Wu MC, Lee S, Cai T, Li Y, Boehnke M, Lin X: Rare-variant association
testing for sequencing data with the sequence kernel association test.
Am J Hum Genet 2011, 89(1):82-93.
3. Price AL, Kryukov GV, de Bakker PI, Purcell SM, Staples J, Wei LJ, Sunyaev SR:
Pooled association tests for rare variants in exon-resequencing studies.
Am J Hum Genet 2010, 86:832-838.
4. Ionita-Laza I, Buxbaum JD, Laird NM, Lange C: A new testing strategy to
identify rare variants with either risk or protective effect on disease.
PLoS Genet 2011, 7:e1001289.
5. Neale BM, Rivas MA, Voight BF, Altshuler D, Devlin B, Orho-Melander M,
Kathiresan S, Purcell SM, Roeder K, Daly MJ: Testing for an unusual
distribution of rare variants. PLoS Genet 2011, 7:e1001322.
6. Audo I, Bujakowska K, Orhan E, Poloschek CM, Defoort-Dhellemmes S,
Drumare I, Kohl S, Luu TD, Lecompte O, Zrenner E, et al: Whole-exome
sequencing identifies mutations in GPR179 leading to autosomal-
recessive complete congenital stationary night blindness. Am J Hum
Genet 2012, 90:321-330.
7. Almasy L, Dyer T, Peralta J, Jun G, Fuchsberger C, Almeida M, Kent JW Jr,
Fowler S, Duggirala R, Blangero J: Data for Genetic Analysis Workshop 18:
human whole genome sequence, blood pressure, and simulated
phenotypes in extended pedigrees. BMC Proc 2014, 8(suppl 2):S2.
Swartz et al. BMC Proceedings 2014, 8(Suppl 1):S13
http://www.biomedcentral.com/1753-6561/8/S1/S13
Page 4 of 5
8. Swartz MD, Yu RK, Shete S: Finding factors influencing risk: comparing
Bayesian stochastic search and standard variable selection methods
applied to logistic regression models of cases and controls. Stat Med
2008, 27:6158-6174.
doi:10.1186/1753-6561-8-S1-S13
Cite this article as: Swartz et al.: Small sample properties of rare variant
analysis methods. BMC Proceedings 2014 8(Suppl 1):S13.
Submit your next manuscript to BioMed Central
and take full advantage of: 
• Convenient online submission
• Thorough peer review
• No space constraints or color figure charges
• Immediate publication on acceptance
• Inclusion in PubMed, CAS, Scopus and Google Scholar
• Research which is freely available for redistribution
Submit your manuscript at 
www.biomedcentral.com/submit
Swartz et al. BMC Proceedings 2014, 8(Suppl 1):S13
http://www.biomedcentral.com/1753-6561/8/S1/S13
Page 5 of 5


http://dx.doi.org/10.1186/1753-6561-8-S1-S13

Small sample properties of rare variant analysis methods

Abstract

Similar works

Full text

Available Versions

Columbia University Academic Commons

Crossref

Springer - Publisher Connector

Springer - Publisher Connector