Motivation: Resampling methods, such as permutation and bootstrap, have been widely used to generate an empirical distribution for assessing the statistical significance of a measurement. However, to obtain a very low P-value, a large size of resampling is required, where computing speed, memory and storage consumption become bottlenecks, and sometimes become impossible, even on a computer cluster. Results: We have developed a multiple stage P-value calculating program called FastPval that can efficiently calculate very low (up to 10-9) P-values from a large number of resampled measurements. With only two input files and a few parameter settings from the users, the program can compute P-values from empirical distribution very efficiently, even on a personal computer. When tested on the order of 109 resampled data, our method only uses 52.94% the time used by the conventional method, implemented by standard quicksort and binary search algorithms, and consumes only 0.11% of the memory and storage. Furthermore, our method can be applied to extra large datasets that the conventional method fails to calculate. The accuracy of the method was tested on data generated from Normal, Poison and Gumbel distributions and was found to be no different from the exact ranking approach. © The Author(s) 2010. Published by Oxford University Press.published_or_final_versio

Li, MJ

Sham, PC

Wang, J

Bioinformatics

PubMed

HKU Scholars Hub

Title FastPval: A fast and memory efficient program to calculate verylow P-values from empirical distributionAuthor(s) Li, MJ; Sham, PC; Wang, JCitation Bioinformatics, 2010, v. 26 n. 22, p. 2897-2899Issued Date 2010URL http://hdl.handle.net/10722/137123Rights[12:11 19/10/2010 Bioinformatics-btq540.tex] Page: 2897 2897–2899BIOINFORMATICS APPLICATIONS NOTE Vol. 26 no. 22 2010, pages 2897–2899doi:10.1093/bioinformatics/btq540Genome analysis Advance Access publication September 21, 2010FastPval: a fast and memory efficient program to calculate verylow P-values from empirical distributionMulin Jun Li1, Pak Chung Sham2 and Junwen Wang1,∗1Department of Biochemistry and 2Department of Psychiatry and State Key Laboratory of Cognitive and BrainSciences, LKS Faculty of Medicine, The University of Hong Kong, 21 Sassoon Rd, Pokfulam, Hong Kong SAR, ChinaAssociate Editor: Alfonso ValenciaABSTRACTMotivation: Resampling methods, such as permutation andbootstrap, have been widely used to generate an empirical dis-tribution for assessing the statistical significance of a measur-ement. However, to obtain a very low P-value, a large size ofresampling is required, where computing speed, memory andstorage consumption become bottlenecks, and sometimes becomeimpossible, even on a computer cluster.Results: We have developed a multiple stage P-value calculatingprogram called FastPval that can efficiently calculate very low (up to10−9) P-values from a large number of resampled measurements.With only two input files and a few parameter settings from theusers, the program can compute P-values from empirical distributionvery efficiently, even on a personal computer. When tested on theorder of 109 resampled data, our method only uses 52.94% thetime used by the conventional method, implemented by standardquicksort and binary search algorithms, and consumes only 0.11%of the memory and storage. Furthermore, our method can be appliedto extra large datasets that the conventional method fails to calculate.The accuracy of the method was tested on data generated fromNormal, Poison and Gumbel distributions and was found to be nodifferent from the exact ranking approach.Availability: The FastPval executable file, the java GUI and sourcecode, and the java web start server with example data andintroduction, are available at http://wanglab.hku.hk/pvalueContact: junwen@hku.hkSupplementary information: Supplementary data are available atBioinformatics online and http://wanglab.hku.hk/pvalue/.Received on August 2, 2010; revised on September 7, 2010;accepted on September 16, 20101 INTRODUCTIONPermutation and bootstrap are resampling procedures to assess thestatistical significance of a measurement. They are non-parametricstatistical tests that can convert a measurement (score) into anempirical P-value, even when the distribution of the measurements isunknown. Since resampling does not assume known distribution ofthe data, and biological data are usually complex, it has been widelyused in the bioinformatics field, such as transcription factor bindingsite searching, pathway analysis and genome-wide associationstudies.∗To whom correspondence should be addressed.Finding transcription factor binding sites (TFBSs) in the promoterregion of a gene is important to understand gene regulation (Zhanget al., 2007). TFBS of a particular transcription factor are usuallyrepresented by a computational model known as the position weightmatrix (PWM) (Pape et al., 2008). To search for a putative bindingsite, we use the PWM to score DNAsequences with a sliding windowapproach. For each window, we obtain a score. By comparing thisscore with the distribution of the scores from the background, we canobtain the statistical significance (empirical P-value) of this score.The empirical background score distribution is obtained by scoringa large set of random sequences from the intergenic regions in thegenome with the same PWM. We then sort the background scoresand save them for later usage. When we convert a new score intoa P-value, we load the background into the memory and search forthe score. The ranking of the score is then converted to a P-value(Hannenhalli, 2008).This empirical approach of calculating P-values is very powerfulsince it does not assume any distribution of the data. However, thesignificance of the P-value is limited by the size of the backgroundwe sample. To obtain a very low P-value, we have to sample avery large dataset from the background. The large dataset causesthree limitations: (i) sorting and searching in a large dataset are timeconsuming; (ii) storage of the sorted background scores requires alarge amount of hard disk space; and (iii) processing of the sortedarray requires a great deal of memory, which is not usually feasibleon a personal computer.Efficient methods have been developed to relieve thecomputational burden resulting from large-scale resampling. Forexample, Jensen et al. (2009) developed a Bayesian approach todynamically assign resamples for multiple testing problems. Formicroarray expression data, they assume that each gene has adifferent null distribution, and allocate more resamples to the geneswith P-values close to the classification threshold. But for theP-values that are far lower or far higher than the threshold and thedecisions that are easy to make, they allocate much fewer resamplesthan the traditional method. The dynamic resampling allocationstrategy has improved the computing efficiency, particularly whenthe number of tests is large.While the above mentioned method deals with the efficiency ofmultiple tests, assuming each test has a different null distribution,P-value calculation from resampling based on a single test, ormultiple tests assuming the same null distribution, is still hamperedby computing, memory and storage limitations.We have developed an efficient program to calculate the empiricalP-value for a single test, or multiple tests assuming the same null© The Author(s) 2010. Published by Oxford University Press.This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/2.5), which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.[12:11 19/10/2010 Bioinformatics-btq540.tex] Page: 2898 2897–2899M.J.Li et al.Table 1. Comparison of FastPval and Exact method in memory, storage consumptions and running timeResampling size(1 000 000)Memory(MB) Storage (MB) Running time (s)Model building P-value searchingaExact FastPval Exact FastPvalb Exact FastPval Exact FastPval1 4 0.39 12.1 1.3 + 0.013 1.10 1.05 0.74 + 2.33 0.08 + 1.5310 38 0.39 121.4 1.3 + 0.131 11.21 9.29 7.61 + 29.88 0.09 + 16.07100 373 0.78 1200 1.3 + 1.3 116.73 91.46 77.13 + 332.13 0.14 + 249.44500 1900 2 5900 1.3 + 6.0 677.58 455.23 380.47 + 1885.12 0.40 + 1297.441000 3700 4 11 900 1.3 + 11.9 1409.61 919.45 761.52 + 4019.77 0.72 + 2530.655000 N/Ac 19 59 900 1.3 + 59.8 N/Ac 5475.32 N/Ac 3.34 + 12885.87aModel loading + searching time.bSizes of first model + second model.cExact method failed to load due to large size of the dataset.distribution. This program separates the background distribution intomultiple parts, according to user specified cutoffs. The scores inthe less significant part are highly condensed into one table andthe P-values are calculated less exactly, while the scores in themore significant part are put into other tables and the P-valuesare calculated more accurately. Our experiments showed that thisalgorithm is more time efficient, and uses much less storageand memory. It can be used widely in resampling based P-valuecalculation, either as standalone software or as a plug-in module.2 METHODSFor simplicity, here we illustrate our method in a two-stage approach and usethe right tail of the distribution to calculate the statistics. In the first stage, werandomly sample a subset N from the original large dataset O. N is usuallyless than one-hundredth of the size of O, thus saving processing time. Wesort N and obtain a cutoff score Sc representing the top P portion of N . BothN and P are parameters specified by the users, and are set to N = 100 000 andP = 0.001 by the default. We then scan the original set and put scores greaterthan Sc into our second subset M, and we obtain the maximum score Sm inM. The two subsets N and M are sorted, saved, and serve as our two models(M1 and M2). To calculate the P-value for a new score S, we compare Swith Sc. If S ≤ Sc, we will find its P-value in M1. Otherwise we use M2. IfS > Sm, indicating S is out of our resampling score range, we use theoreticaldistribution to calculate its P-value or simply set the P-value to 0, at theuser’s preference. The parameters of two theoretical distributions, normaland extreme value distributions, were obtained from dataset N .To evaluate the performance of our method, we compared FastPval withthe traditional approach (named Exact) on a linux machine (Intel Xeon CPUE5410 2.33 GHz; 16 G of memory, SuSE linux 10.1). In the ‘Exact’approach,we used quicksort and binary search in c programming. FastPval used thesame sorting and searching algorithms for M1 and M2.To evaluate the accuracy of FastPval, we compared the calculatedP-values with the original P-values in three different distributions: Normal,Poisson and Gumbel. For each distribution, we tested the P-values in the−log10(P-value) ranged from 0 to 9 (corresponding P-value range from 1to 10−9, exclusively). We took 162 P-values (termed theoretical P-values)evenly distributed in each of the nine ranges (Supplementary Table S1).We converted these P-values into scores with the build-in functions in the Rplatform, using parameters for each distribution as shown in SupplementaryTable S2. Under the same parameter setting, we randomly sampled onebillion data points. We used FastPval to build models M1 and M2 from thesedata. Finally, we used FastPval to convert these 162 scores into P-values(termed FastPval P-values) with the models. The −log10 (FastPval P-values)were plotted against the −log10 (theoretical P-values) on a Log–Log QQplot. The Kolmogorov–Smirnov test was also used to compare the FastPvalP-values with theoretical P-values.3 RESULTS AND DISCUSSIONAs shown in Table 1, FastPval shows significant improvement overthe ‘Exact’ approach. Tested on 1 billion resampling data, FastPvalonly used 0.11% of the memory and storage and 52.94% of modelbuilding and searching times. When we increased the dataset size to5 billion, the ‘Exact’ method failed to load due to the large memoryrequirement, while our method was able to calculate P-valuessuccessfully. FastPval has speed, memory and storage consumptionapproximately linear to resembling size. The accuracies of FastPvalcalculated P-values from three different distributions were comparedwith the theoretical P-values. The results are shown in the form ofLog–Log QQ plots (Supplementary Fig. S1a–c). In all three testeddistributions, the FastPval calculated P-values and the theoreticalP-values are highly matched, forming a 45 degree line in theLog–Log QQ plots. The Kolmogorov–Smirnov tests showed theP-value = 1.000 for Normal and Poisson distributions, and 0.998 forGumbel distribution, indicating that the calculated P-values did notdeviate from the original distribution. We therefore conclude thatFastPval is accurate for calculating P-values for data from a varietyof distributions.The Java GUI interface of FastPval is shown in SupplementaryFigure S2a–c. In the ‘Method’ field, the user can either choose‘FastPval’ or the traditional ‘Exact’ method to calculate P-values.When the resembling size is greater than 10 million, FastPval is theonly suitable choice. In the ‘Step’ field, the user can either ‘Generatemodel’, or ‘Calculate P-value’ or ‘Do both’. The ‘Generate model’step allows the user to generate M1 and M2 models from thebackground dataset O; the ‘Calculate P-value’ step allows the userto calculate P-values for all the scores saved in a text file. Theinterface changes according to the user’s selection. In the ‘Generatemodel’ step, the user has to specify the background file and directoryto save two models, by clicking on ‘Background file’ and ‘Outputfolder’, respectively. The user also needs to select two parameters,the ‘Sampling size’ (N) and ‘P-value cutoff’ (P). The selections of Pand N are affected by the balance of accuracy and speed. Bigger Nsand Ps will give more accurate P-values but will be less efficient. We2898[12:11 19/10/2010 Bioinformatics-btq540.tex] Page: 2899 2897–2899FastPval: an efficient program for low P-value calculationrecommend N = 100 000 and P = 0.001. N∗P should be within therange of 10 to 1000, preferably 100. After the models are generated,the program will automatically change to the ‘Calculate P-value’step, with default model files selected. The user will need to specifythe file with scores for P-value calculation, by clicking on ‘Samplefile’. For scores that are out of the boundary of both models, the usercan choose either ‘Extreme distribution’ or ‘Normal distribution’ inthe ‘Assumed distribution’ field to calculate the P-value, or simplyselect ‘No distribution’ to assign the P-value to 0. The parameters ofboth distributions were calculated from the fitting of the dataset N .The program can be run in a command line mode, which is suitablefor large-scale batch processing. We provide both 32-bit and 64-bitexecutable GUI programs, in both linux and windows platforms. Thesource code of the program is provided in our companion website.Funding: Internal funds from the CRCG and the Genomic SRT ofthe University of Hong Kong; GRF 778609M and AoE M-04/04from the Research Grants Council of Hong Kong.Conflict of Interest: none declared.REFERENCESHannenhalli,S. (2008) Eukaryotic transcription factor binding sites—modeling andintegrative search methods. Bioinformatics, 24, 1325–1331.Jensen,S.T. et al. (2009) A Bayesian approach to efficient differential allocation forresampling-based significance testing. BMC Bioinformatics, 10, 198.Pape,U.J. et al. (2008) Natural similarity measures between position frequency matriceswith an application to clustering. Bioinformatics, 24, 350–357.Zhang,J. et al. (2007) Computing exact P-values for DNA motifs. Bioinformatics, 23,531–537.2899

FastPval: A fast and memory efficient program to calculate very low P-values from empirical distribution

A Bayesian approach to efﬁcient differential allocation for resampling-based signiﬁcance testing.

Computing exact P-values for DNA motifs.

Eukaryotic transcription factor binding sites—modeling and integrative search methods.

http://hub.hku.hk/bitstream/10722/137123/2/btq540.pdf

FastPval: A fast and memory efficient program to calculate very low P-values from empirical distribution

Abstract

Similar works

Full text

Available Versions

HKU Scholars Hub