Aberer, A. J.

Kobert, K.

Stamatakis, A.

English

KITopen

LetterFastTrackExaBayes: Massively Parallel Bayesian Tree Inference for theWhole-Genome EraAndre J. Aberer,*,1 Kassian Kobert,1 and Alexandros Stamatakis1,21Scientific Computing Group, Heidelberg Institute for Theoretical Studies, Heidelberg, Germany2Institute for Theoretical Informatics, Karlsruhe Institute of Technology, Karlsruhe, Germany*Corresponding author: E-mail: andre.aberer@h-its.org.Associate editor: Xun GuAbstractModern sequencing technology now allows biologists to collect the entirety of molecular evidence for reconstructingevolutionary trees. We introduce a novel, user-friendly software package engineered for conducting state-of-the-artBayesian tree inferences on data sets of arbitrary size. Our software introduces a nonblocking parallelization ofMetropolis-coupled chains, modifications for efficient analyses of data sets comprising thousands of partitions andmemory saving techniques. We report on first experiences with Bayesian inferences at the whole-genome level usingthe SuperMUC supercomputer and simulated data.Key words: software, Bayesian statistics, phylogenetic inference, whole-genome analyses, parallelization.The task of resolving the tree of life of extant species remainsone of the grand challenges in evolutionary biology. As thenumber of trees grows superexponentially with the number ofspecies for which an evolutionary tree is reconstructed, treeinference is considered a hard problem in computer science.The plethora of algorithmic challenges associated with phylo-genetic trees and their efficient computation gave rise to thediscipline of “phyloinformatics.” Likelihood-based statisticalmethods are highly popular because they can incorporatecomplex evolutionary models. Popular likelihood-basedtools comprise methods for maximum-likelihood (ML) esti-mation, such as RAxML (Stamatakis 2014) and PhyML(Guindon et al. 2010), as well as Bayesian inference packages,such as MrBayes (Ronquist et al. 2012) and BEAST(Drummond et al. 2012). Likelihood-based methods can, infact, unravel the true evolutionary tree given enough data andthe appropriate model (Yang 1994b). Although the ML ap-proach strives to optimize the likelihood of the tree and modelgiven the data, Bayesian inference uses Markov chain MonteCarlo (MCMC) (Hastings 1970) sampling to integrate over theentire parameter space (e.g., model parameters and tree to-pologies) given the data and subjective prior assumptions.Inexpensive wet-lab sequencing technologies allow amas-sing molecular evidence from hundreds of genes (DellAmpioet al. 2014). This amount of data is often required for resolvingancient radiations (Dunn et al. 2008). Hence, so-called phylo-genomic data sets are being increasingly used to disentangleevolution. Collaborative efforts such as the 1K InsectTranscriptome Evolution (http://1kite.org, last accessedAugust 13, 2014) project aim at assembling the entirety ofmolecular sequence data to infer accurate phylogenies.Conducting such phylogenomic analyses is challenging dueto exorbitant runtime and memory requirements. In a recentstudy (Rinke et al. 2013) with more than 2,000 microbialspecies and more than 8,000 amino acid alignment charac-ters, the authors did not complement ML trees by Bayesiantree inferences. The reason for this was the limited ability ofcurrent Bayesian inference software to leverage supercom-puter resources (often several CPU-months or CPU-yearsare required) and to accommodate the excessive mainmemory requirements which for this example are in theorder of 10–20 GB.Novel Approaches and Software FeaturesWe resolve these computational limitations of Bayesian infer-ence tools by introducing ExaBayes, a software packageengineered for efficient Bayesian tree inference on data setsof almost arbitrary size. ExaBayes can conduct Bayesiananalyses for the most widely used priors, models, and inputdata types. These comprise Dirichlet, exponential and uni-form priors, the general time-reversible (GTR) model of nu-cleotide substitution (Tavare 1986), the G model of rateheterogeneity (Yang 1994a), and unconstrained branchlength sampling. For protein data, we offer 18 commonlyused fixed-rate substitution matrices as well as a GTRmodel. We also implemented a comprehensive set of topo-logical proposals that assure rapid convergence and adaptedthese for massively parallel execution.ExaBayes is a self-contained software package, that is, itcomprises several postprocessing methods such as stand-alone tools for building consensus trees, for assessing topo-logical convergence among independent runs (Lakner et al.2008), and for extracting sample statistics. The output formatis compatible with popular visualization tools, such asFigTree (Rambaut 2014) or Tracer (Rambaut andDrummond 2007).Runtime as well as memory requirements of Bayesianinference is largely dominated by evaluating the The Author 2014. Published by Oxford University Press on behalf of the Society for Molecular Biology and Evolution.This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work isproperly cited. Open AccessMol. Biol. Evol. 31(10):2553–2556 doi:10.1093/molbev/msu236 Advance Access publication August 18, 2014 2553Downloaded from https://academic.oup.com/mbe/article-abstract/31/10/2553/1016562 by Karlsruher Institut fur Technologie - KIT user on 07 November 2018phylogenetic-likelihood function. In ExaBayes, we deploythe efficient-likelihood kernel developed for RAxML(Stamatakis 2014) that allows fully leveraging the computa-tional power of SSE and AVX vector units on modern CPUs.The sequential AVX version of ExaBayes outperformsMrBayes for DNA data, whereas for protein dataMrBayes is faster (see Supplementary Material online, fordetailed discussions on this and further results). The centralcontribution of ExaBayes lies in its scalability.ExaBayes implements three layers of parallelism (seefig. 1). For instance, the parallel version of ExaBayesallows to distribute the likelihood calculations across severalprocessors and across several computing nodes in a cluster.Thus, the resources of an entire cluster or supercomputer canbe used to accommodate the runtime and memory require-ments of large-scale alignments. Parallel execution reducesruntimes almost linearly to the number of CPUs used. For asingle chain and on a simulated data set with 200 species and500,000 DNA characters, using the parallel version ofExaBayes reduces runtimes from 1 day and 4 h on onecore to only 43.5 s on 8,192 cores, achieving a speedup of2,368-fold. If we increase the number of characters by afactor of 10, ExaBayes scales from 256 up until at least32,768 cores and improves runtime even faster than theoret-ically expected for up to 4,096 cores because of increasedcache efficiency (see fig. 2).ExaBayes implements Metropolis-coupling (Geyer1992), a fundamental mechanism to accelerate convergenceon “difficult” data sets. Heated chains with increased accep-tance probabilities are coupled to the chain that is beingsampled. Swaps between chains are accepted proportionalto the posterior probability of chain states. As memory re-quirements increase linearly with each additional chain, largedata sets require chains to be distributed across several CPUsor computing nodes in a cluster (second layer of parallelism).To this end, we modified the state-of-the-art parallel algo-rithm for Metropolis-coupled MCMC (Altekar et al. 2004) touse faster, nonblocking communication among processes (seesupplementary material, Supplementary Material online, for adetailed description). Thus, when a set of processors proposea swap through a nonblocking message, they can immediatelycontinue doing useful work by conducting calculations on adifferent chain that will not be swapped. This reduces run-times by up to 19% and thus avoids wasting hundreds of CPUhours for large-scale analyses. Finally, independent analysescan be executed concurrently using a third layer of parallelism(see fig. 1).Moreover, ExaBayes adapts two orthogonal memorysaving techniques. The so-called subtree equality vector ap-proach allows to save memory proportional to the fraction ofunknown or missing data. At the same time, it can also accel-erate likelihood calculations on data sets with a large propor-tion of missing data. This strategy was designed forphylogenomic data sets (Izquierdo-Carrasco et al. 2011),where unknown orthologs can lead to a large fraction of miss-ing data exceeding for instance 75% (DellAmpio et al. 2014).Our second technique allows discarding memory-intensivepartial results by recomputing these on demand. Three set-tings trade varying amounts of runtime for additional memorysavings by means of recomputation. Specifically, when cou-pled chains are distributed, this reduces memory requirementsby a factor of up to 2, while we observe a slow-down of lessthan 1.5-fold. As the size of data sets that can be computedby ExaBayes is merely constrained by available hardware,our memory saving techniques allow conducting ambitiousanalyses on small and less expensive computer clusters.For improved model fit, it is often desirable to sampledistinct model parameters (e.g., substitution rates) for eachgene or partition of an alignment. We have tested ExaBayesusing simulated alignments with 1,000–10,000 partitions. Allparameters (including branch lengths, but excluding the to-pology) can be flexibly linked or unlinked across partitions.We observed a performance decrease with an increasingnumber of partitions. To alleviate this issue, we modifiedour MCMC algorithm and data-to-processor assignmentscheme. This modification induces a runtime improvementof up to 22 times for runs in which branch lengths are linkedacross all partitions. ExaBayes runs up to 87 times faster,when each partition has distinct branch lengths (i.e., branchlengths are unlinked across partitions).Inference from a Whole-Genome Data setTo demonstrate the capabilities of ExaBayes, we executeda Bayesian inference on a “difficult” complete simulated#processes500 1000 2000 5000 10000 20000125102050●●●●●●●●020406080100120scaling performance [%]scaling factorFIG. 2. Scaling factor (sequential runtime divided by parallel runtime)and efficiency (scaling factor divided by number of processes) for exe-cuting ExaBayes on 256 cores up to 32,768 cores on a 200 speciesalignment with 500,000 characters.41node B23node D14node A23node Cnode E67node F58independent runscoupledchainslikelihood evaluationFIG. 1. The three layers of parallelism employed by ExaBayes(distributed likelihood evaluation, distributed Metropolis-coupledchains, and distributed independent analyses).2554Aberer et al. . doi:10.1093/molbev/msu236 MBEDownloaded from https://academic.oup.com/mbe/article-abstract/31/10/2553/1016562 by Karlsruher Institut fur Technologie - KIT user on 07 November 2018genome (for which the “true” evolutionary history is known)comprising 100 partitions with 1,000,000 characters each(inspired by a per-chromosome partitioning). To simulatethe alignment, we used a tree with 200 species. The treeis bush-like and contains many long, outer branches andshort, hard-to-resolve, inner branches. As the RAM require-ments of this data set exceed 5 TB, we employed theSuperMUC supercomputer, which is currently among theten fastest supercomputers in the world. A total of fourindependent runs with one chain each seeded by reasonable(i.e., nonrandom) parsimony starting trees (Fitch andMargoliash 1967) rapidly converged to the true tree topologywithin less than 20,000 generations (total chain length:100,000 generations; a sample was extracted every 500 gen-erations). All branches showed 100% certainty (posteriorprobability). Using over 4,096 CPU cores, the slowest runtook 1 h 40 min. Thus, the accumulated CPU-time over allfour runs is 3 years and 45 days. To avoid potential biasesinduced by the parsimony starting trees, we also ran twoindependent chains starting from random trees. Here,the chains converged to the true tree topology after&60,000 generations. With random starting trees, wehave to discard substantially longer burn-ins, before weattain an accurate sampling of the posterior probability ofthe trees.We examined how the posterior probability of the treeschanges as we reduce the amount of data in steps of 2 ordersof magnitude. If we discard 33% of samples as burn-in, weobtain 1) 100% confidence for all splits in the tree inferredfrom 100,000,000 characters, 2) only one split with 98.5%certainty for 1,000,000 characters, and 3) nine splits with aposterior probability between 61.2% and 99.2% for10,000 characters. For 10,000 characters we do not attainmaximum confidence in the true tree, even if the burn-inphase is substantially extended. Thus, we conclude that thisamount of data is insufficient for inferring a reliable tree onthis data set. As illustrated in figure 3, the chain instead jumpsbetween several trees of high posterior probability afterburn-in.Finally, we examined how confidence intervals for branchlengths change as we increase the amount of data. We findthat, with only 10,000 characters, we mostly do not obtainaccurate branch length estimates and that confidenceintervals cover a wide spectrum specifically for shortbranch lengths. Branch lengths can be determined moreaccurately with more data; however, increasing the data setsize from 1,000,000 characters to 100,000,000 charactersdoes not substantially decrease the standard deviationin branch length samples (see supplementary material,Supplementary Material online). We observed thatgenome-scale data allow for high certainty about the “true”(here simulated) branch length; however, even this amountof data does not guarantee small confidence intervals forbranch lengths in the range of 103  104. Nevertheless,we expect that with chromosome- and genome-size data,extremely precise estimations of divergence times can beperformed.Supplementary MaterialSupplementary material is available at Molecular Biology andEvolution online (http://www.mbe.oxfordjournals.org/).AcknowledgmentsThe authors are extremely grateful to Fredrik Ronquist forsharing his knowledge and expertise about Bayesian phyloge-netic analysis with them. They thank Paschalia Kapli for-testing the program and helping to improve the user-friendliness of ExaBayes. Experiments for demonstratingscalability and the whole-genome inference were coveredby a CPU-time grant for the SuperMUC system at theLeibniz Supercomputing Centre (LRZ). All authors arefunded through institutional funding provided by theHeidelberg Institute for Theoretical Studies.ReferencesAltekar G, Dwarkadas S, Huelsenbeck JP, Ronquist F. 2004. ParallelMetropolis coupled Markov chain Monte Carlo for Bayesian phylo-genetic inference. Bioinformatics 20(3):407–415.DellAmpio E, Meusemann K, Szucsich NU, Peters RS, Meyer B, Borner J,Petersen M, Aberer AJ, Stamatakis A, Walzl MG, et al. 2014.Decisive data sets in phylogenomics: lessons from studies on thephylogenetic relationships of primarily wingless insects. Mol Biol Evol.31(1):239–249.Drummond AJ, Suchard MA, Xie D, Rambaut A. 2012. Bayesian phylo-genetics with BEAUti and the BEAST 1.7. Mol Biol Evol.29(8):1969–1973.Dunn CW, Hejnol A, Matus DQ, Pang K, Browne WE, Smith SA, SeaverE, Rouse GW, Obst M, Edgecombe GD, et al. 2008. Broaddimension 1dimension 2−3−2−101−4 −2 0 210,000 1,000,000 100,000,000sim. treeFIG. 3. Inference from genome-sized data: 2D-rescaled representationemploying multidimensional scaling (MDS) of Robinson–Foulds dis-tances among sampled trees for chains with varying amount of data.The position of the simulated true tree is shown in red. The appliedMDS algorithm maps identical trees to adjacent nonidentical positions(i.e., overlapping squares represent identical trees).2555Parallel Bayesian Tree Inference . doi:10.1093/molbev/msu236 MBEDownloaded from https://academic.oup.com/mbe/article-abstract/31/10/2553/1016562 by Karlsruher Institut fur Technologie - KIT user on 07 November 2018phylogenomic sampling improves resolution of the animal tree oflife. Nature 452(7188):745–749.Fitch WM, Margoliash E. 1967. Construction of phylogenetic trees.Science 155(760):279–284.Geyer CJ. 1991. Markov chain Monte Carlo maximum likelihood.Computing Science and Statistics, Proceedings of the 23rdSymposium on the Interface. p. 156–163.Guindon S, Dufayard J-F, Lefort V, Anisimova M, Hordijk W, Gascuel O.2010. New algorithms and methods to estimate maximum-likeli-hood phylogenies: assessing the performance of phyml 3.0. Syst. Biol.59(3):307–321.Hastings WK. 1970. Monte Carlo sampling methods usingMarkov chains and their applications. Biometrika 57(1):97–109.Izquierdo-Carrasco F, Smith SA, Stamatakis A. 2011. Algorithms, datastructures, and numerics for likelihood-based phylogenetic inferenceof huge trees. BMC Bioinformatics 12(1):470.Lakner C, van der Mark P, Huelsenbeck JP, Larget B, Ronquist F. 2008.Efficiency of Markov chain Monte Carlo tree proposals in Bayesianphylogenetics. Syst. Biol. 57(1):86–103.Rambaut A. 2014. Figtree, a graphical viewer of phylogenetic trees[Internet]. Available from: http://tree.bio.ed.ac.uk/software/figtree.Rambaut A, Drummond A. 2007. Tracer v1.4 [Internet]. Available fromhttp://tree.bio.ed.ac.uk/software/tracer/.Rinke C, Schwientek P, Sczyrba A, Ivanova NN, Anderson IJ, Cheng J-F,Darling A, Malfatti S, Swan BK, Gies EA, et al. 2013. Insights into thephylogeny and coding potential of microbial dark matter. Nature499:431–437.Ronquist F, Teslenko M, van der Mark P, Ayres DL, Darling A,H€ohna S, Larget B, Liu L, Suchard MA, Huelsenbeck JP. 2012.MrBayes 3.2: efficient Bayesian phylogenetic inference andmodel choice across a large model space. Syst. Biol.61(3):539–542.Stamatakis A. 2014. Raxml version 8: a tool for phylogenetic analysis andpost-analysis of large phylogenies. Bioinformatics 30(9):1312–1313.Tavare S. 1986. Some probabilistic and statistical problems in the anal-ysis of DNA sequences. Lect Math Life Sci. 17:57–86.Yang Z. 1994a. Maximum likelihood phylogenetic estimation from DNAsequences with variable rates over sites: approximate methods. J MolEvol. 39(3):306–314.Yang Z. 1994b. Statistical properties of the maximum likelihood methodof phylogenetic estimation and comparison with distance matrixmethods. Syst Biol. 43(3):329–342.2556Aberer et al. . doi:10.1093/molbev/msu236 MBEDownloaded from https://academic.oup.com/mbe/article-abstract/31/10/2553/1016562 by Karlsruher Institut fur Technologie - KIT user on 07 November 2018

ExaBayes: Massively Parallel Bayesian Tree Inference for the Whole-Genome Era

https://publikationen.bibliothek.kit.edu/1000045631/19159275

ExaBayes: Massively Parallel Bayesian Tree Inference for the Whole-Genome Era

Abstract

Similar works

Full text

Available Versions

KITopen