Motivation: Previously we presented swarm, an open-source amplicon clustering programme that produces fine-scale molecular operational taxonomic units (OTUs) that are free of arbitrary global clustering thresholds. Here, we present swarm v3 to address issues of contemporary datasets that are growing towards tera-byte sizes.
					Results: When compared with previous swarm versions, swarm v3 has modernized C++ source code, reduced memory footprint by up to 50%, optimized CPU-usage and multithreading (more than 7 times faster with default parameters), and it has been extensively tested for its robustness and logic

Birol, Inanc

Czech, Lucas

de Vargas, Colomban

Dunthorn, Micah

Mahé, Frédéric

Quince, Christopher

Rognes, Torbjørn

Stamatakis, Alexandros

English

KITopen

Sequence analysisSwarm v3: towards tera-scale amplicon clusteringFrédéric Mahé1,2,*, Lucas Czech 3,4, Alexandros Stamatakis3,5,Christopher Quince6,7,8, Colomban de Vargas9,10, Micah Dunthorn 11,12 andTorbjørn Rognes13,141UMR PHIM, CIRAD, Montpellier, France, 2PHIM Plant Health Institute, Univ Montpellier, CIRAD, INRAE, Institut Agro, IRD, Montpellier,France, 3Computational Molecular Evolution Group, Heidelberg Institute for Theoretical Studies, Heidelberg, Germany, 4Department ofPlant Biology, Carnegie Institution for Science, Stanford, CA, USA, 5Institute of Theoretical Informatics, Karlsruhe Institute ofTechnology, Karlsruhe, Germany, 6Organisms and Ecosystems, Earlham Institute, Norwich, UK, 7Gut Microbes and Health, QuadramInstitute, Norwich, UK, 8Warwick Medical School, University of Warwick, Coventry, UK, 9Sorbonne Université, CNRS, StationBiologique de Roscoff, UMR7144, ECOMAP, Roscoff, France, 10Research Federation for the study of Global Ocean Systems Ecologyand Evolution, FR2022/Tara GOSEE, Paris, France, 11Natural History Museum, University of Oslo, Oslo, Norway, 12EukaryoticMicrobiology, University of Duisburg-Essen, Essen, Germany, 13Department of Informatics, University of Oslo, Oslo, Norway and14Department of Microbiology, Oslo University Hospital, Rikshospitalet, Oslo, Norway*To whom correspondence should be addressed.Associate Editor: Inanc BirolReceived on January 6, 2021; revised on May 24, 2021; editorial decision on June 13, 2021; accepted on July 1, 2021AbstractMotivation: Previously we presented swarm, an open-source amplicon clustering programme that produces fine-scale molecular operational taxonomic units (OTUs) that are free of arbitrary global clustering thresholds. Here, wepresent swarm v3 to address issues of contemporary datasets that are growing towards tera-byte sizes.Results: When compared with previous swarm versions, swarm v3 has modernized Cþþ source code, reducedmemory footprint by up to 50%, optimized CPU-usage and multithreading (more than 7 times faster with defaultparameters), and it has been extensively tested for its robustness and logic.Availability and implementation: Source code and binaries are available at https://github.com/torognes/swarm.Contact: frederic.mahe@cirad.frSupplementary information: Supplementary data are available at Bioinformatics online.1 IntroductionIn emerging planetary biology, large-scale amplicon sequencingdatasets are used to unravel global ecological and evolutionary pat-terns within and across biomes and biota (de Vargas et al., 2015;Mahé et al., 2017; Giner et al., 2020). With today’s sequencing plat-forms, such as Illumina and PacBio, single environmental diversitystudies can produce massive amounts of data. A critical bioinformat-ics step in the handling of these massive metabarcoding datasets is tocluster the sequencing reads into operational taxonomic units(OTUs). OTUs are often used as units of comparison in downstreamstatistical analyses and are often interpreted as proxies for speciesand other taxa (Santoferrara et al., 2020).Swarm v1 (Mahé et al., 2014) was introduced as a novel ap-proach to cluster amplicons into OTUs, inspired by previous single-linkage methods, such as DOTUR (Schloss and Handelsman, 2005).The key underlying idea of swarm was to use a local, iterative,single-linkage clustering process to group closely related sequences(by default with one difference in their nucleotide sequences, i.e.d¼1). Swarm’s clustering process differs from global clusteringthreshold approaches that apply an arbitrary fixed minimal similaritybetween the OTU seed and other OTU members; often set at 97% or98% (Edgar et al., 2010), or from model-based noise-filtering meth-ods, such as DADA2 (Callahan et al., 2016) and Deblur (Amir et al.,2017). The recommended usage of these methods is to process sam-ples or sequencing runs independently and then to merge the results.Swarm offers a fast alternative allowing users to (re-)process entiredatasets at once. Swarm v2 (Mahé et al., 2015) implemented in Cþþtwo additional features to refine clustering: OTU-breaking that splitsOTUs that are only linked via low-abundant sequences (–no-otu-breaking to disable); and the merging that grafts low-abundant OTUsonto higher-abundant OTUs (–fastidious to enable).Swarm v2 was completely implemented in Cþþ and was sub-stantially faster due to algorithmic advances when used with defaultparameters (d¼1). There was still room for improvement. Therewere issues with code standardization that could limit compile-timeoptimization and raise warnings or errors with future compilers(Darriba et al., 2018; Wilson et al., 2014). The code could only beVC The Author(s) 2021. Published by Oxford University Press. 267This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unre-stricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.Bioinformatics, 38(1), 2022, 267–269doi: 10.1093/bioinformatics/btab493Advance Access Publication Date: 9 July 2021Applications NoteDownloaded from https://academic.oup.com/bioinformatics/article/38/1/267/6318385 by KIT Library user on 16 December 2022executed on GNU/Linux and macOS on x86-64 CPUs. And althoughswarm v2 was multithreaded and fast, its time and memory require-ments could become a limiting factor on very large current and fu-ture datasets, especially as amplicon sequences become longer.Swarm v3 addresses these issues.2 Code quality and portabilityFollowing the recommendations of Darriba et al. (2018), swarm v3features a substantially revised and improved documentation (e.g. helpand man page), as well as clearer and more helpful warnings and errormessages. Swarm’s logic and behavior have been tested extensively viaautomatically generated input (afl-fuzz; https://lcamtuf.coredump.cx/afl/) and 669 hand-crafted functional software tests (https://github.com/frederic-mahe/swarm-tests/), covering more than 95% of swarm’scode (the remaining code is CPU architecture-specific). The Codecov(https://codecov.io) tool tracks code coverage evolution, and theTravis-CI (https://travis-ci.org) suite automatically executes the testsuite on each new code modification to prevent regressions.To facilitate swarm’s long-term maintenance and portability,advanced compiler options [gcc (https://gcc.gnu.org) and clang(https://clang.llvm.org)] as well as state-of-the-art static [cppcheck(http://cppcheck.sourceforge.net) and clang-tidy (https://clang.llvm.org/extra/clang-tidy/)] and dynamic Cþþ analyzers [valgrind(https://www.valgrind.org)] were used to detect unsafe or deprecatedcode not reported by commonly used compiler options. More than1600 warnings were fixed so far, improving swarm’s global codequality score as assessed by SoftWipe (Zapletal et al., 2020) from5.2 to 6.6 out of 10. Swarm has now been ported to new combina-tions of CPU architectures and operating systems: Microsoft’sWindows on x86-64, GNU/Linux and macOS on ARM 64 andGNU/Linux on POWER8, in addition to the already available ver-sions for GNU/Linux and macOS on x86-64.3 Time and space optimization, real-world resultsDNA sequences are stored in silico as strings of the four characters A,C, G and T. Rather than using a byte of memory for storing each nu-cleotide, it is possible to only use two bits. Thereby, four nucleotidescan be stored per byte. This compression reduces the global memory-footprint but also requires some storage overhead and additionalencoding-decoding operations as CPUs cannot operate directly onanything smaller than a byte. To alleviate this, swarm v3 deploys afaster hash function (Zobrist, 1970) and an efficient Bloom filter(Putze et al., 2009), and was re-written to operate on fixed-lengthchunks of compressed sequences, rather than on individual nucleoti-des (see Supplementary File). It should be noted that this new algo-rithm only applies to the default value for swarm’s d parameter(d¼1). Higher d values use the same algorithm as in swarm v2.On a dataset of 10.6 million unique SSU-rRNA V4 sequences(representing 31.6 million reads, 380 bp on average, Mahé et al.,2017), and a series of subsamplings (1% and 10–90% steps), swarmv3 outperformed swarm v2 in every performance metric, while yield-ing exactly identical clustering results. With both versions runningon 1 core, v3 was more than 7 times faster than v2. When both wererunning on 16 cores, v3 was about 10 times faster than v2. Thememory requirement of v3 was about half that of v2(Supplementary Fig. S1). Comparable results were obtained on a se-cond dataset of 10.6 million unique SSU-rRNA V9 sequences(130 bp on average, de Vargas et al., 2015), but with a less pro-nounced memory-footprint reduction as the storage overhead oftwo-bit compressed sequences has a larger impact with shortersequences (see Supplementary Figs. S2, S3 and Supplementary Filefor a detailed benchmark description).When using the merging option (named fastidious), swarm v3 ismore than 5 times faster for SSU-rRNA V9 (130 bp), and more than9 times faster for SSU-rRNA V4 (380 bp) (Supplementary Fig. S2).The memory-footprint is only reduced by 5–10% due to the fact thatthe fastidious algorithm relies on a Bloom filter to store hash valuesinstead of DNA sequences, and therefore does not profit from thetwo-bit sequence compression.4 ConclusionSwarm v3 is a clustering method designed to maximize taxonomicresolution, sensitivity and speed. If coupled with ‘lossy’ post-clustering filtering steps, such as chimera detection, quality filteringand multi-sample co-occurrence patterns (e.g. Frøslev et al., 2017),swarm has the potential to yield robust, single-nucleotide resolutionresults. Swarm v3 can be used on short and long read metabarcodingdata (with sequences up to 10 Mbp when using d¼ 1), or on meta-transcriptomic/genomic data that has been subsampled from thesame locus. It offers a comprehensive set of options that gives usersfull-control and access to intermediate internal data, such as thecomplete pairwise sequence network (see Forster et al., 2020, for ausage example). Swarm v3 is open-source, actively maintained, port-able and efficient, thus reducing the need for expensive computation-al resources. As an example, the UniEuk project (Berney et al., 2017)gathered from the global research community an SSU-rRNA V4dataset with nearly 324 million unique sequences (123 billionnucleotides), more than three times the volume of the recently pub-lished Earth Microbiome Project (Thompson et al., 2017). Using de-fault parameters, swarm v3 required 50 min to cluster the UniEukdataset on a 16-core system. We estimate that it would take less thansix hours on the same machine to process a one trillion nucleotide,or one tera-byte dataset.AcknowledgementsThe authors thank Étienne Platini and Milena Königshoffen for writing unittests, and Claude Monet for providing the impressionist background. The bio-informatics analyses were performed on the Core Cluster of the InstitutFrançais de Bioinformatique (IFB) (ANR-11-INBS-0013). We are also gratefulfor access to computational resources provided by UNINETT Sigma2—theNational Infrastructure for High Performance Computing and Data Storagein Norway (project NN9383K), the University of Oslo and the Oregon StateUniversity.FundingThis work was supported by the Gordon and Betty Moore Foundationthrough the UniEuk grant GBMF5275, the Klaus Tschira Foundation, and theDeutsche Forschungsgemeinschaft (#DU1319/5-1).Conflict of Interest: none declared.ReferencesAmir,A. et al. (2017) Deblur rapidly resolves single-nucleotide community se-quence patterns. mSystems, 2, e00191-16.Berney,C. et al. (2017) UniEuk: time to speak a common language in protist-ology! J. Euk. Microbiol., 64, 407–411.Callahan,B. et al. (2016) DADA2: high-resolution sample inference fromIllumina amplicon data. Nat. Meth., 13, 581–583.Darriba,D. et al. (2018) The state of software for evolutionary biology. Mol.Biol. Evol., 35, 1037–1046.Edgar,R.C. (2010) Search and clustering orders of magnitude faster thanBLAST. Bioinformatics, 26, 2460–2461.Forster,D. et al. (2020) Evaluating geographic variation within molecular op-erational taxonomic units (OTUs) using network analyses in Scandinavianlakes. BioRxiv, 2020.08.06.240267.Frøslev,T.G. et al. (2017) Algorithm for post-clustering curation of DNA ampli-con data yields reliable biodiversity estimates. Nat. Commun., 8, 1188.Giner,C.R. et al. (2020) Marked changes in diversity and relative activity ofpicoeukaryotes with depth in the world ocean. ISME J., 14, 437–449.Mahé,F. et al. (2014) Swarm: robust and fast clustering method foramplicon-based studies. PeerJ, 2, e593.268 F.Mahé et al.Downloaded from https://academic.oup.com/bioinformatics/article/38/1/267/6318385 by KIT Library user on 16 December 2022Mahé,F. et al. (2015) Swarm v2: highly-scalable and high-resolution ampliconclustering. PeerJ, 3, e1420.Mahé,F. et al. (2017) Parasites dominate hyperdiverse soil protist communitiesin Neotropical rainforests. Nat. Ecol. Evol., 1, 0091.Putze,F. et al. (2009) Cache-, hash- and space-efficient bloom filters. J. Experi.Algorithm., 14, 4.Santoferrara,L. et al. (2020) Perspectives from ten years of protist studies byhigh-throughput metabarcoding. J. Eukaryot. Microbiol., 67, 612–622.Schloss,P.D. and Handelsman,J. (2005) Introducing DOTUR, a computer pro-gram for defining operational taxonomic units and estimating species rich-ness. Appli. Environ. Microbiol., 71, 1501–1506.Thompson,L. et al.; Earth Microbiome Project Consortium (2017) A communal cata-logue reveals Earth’s multiscale microbial diversity. Nature, 551, 457–463.de Vargas,C. et al.; Tara Oceans Coordinators (2015) Eukaryotic plankton di-versity in the sunlit global ocean. Science, 348, 1261605.Wilson,G. et al. (2014) Best practices for scientific computing. PLoS Biol., 12,e1001745.Zapletal,A. et al. (2020) SoftWipe—a tool and benchmark to assess scientificsoftware quality. BioRxiv, 2020.10.07.330621.Zobrist,A.L. (1970) A New Hashing Method with Application for GamePlaying. Tech. Rep., 88, Computer Sciences Department, University ofWisconsin, Madison, WI, USA.Swarm v3 269Downloaded from https://academic.oup.com/bioinformatics/article/38/1/267/6318385 by KIT Library user on 16 December 2022

Swarm v3: towards tera-scale amplicon clustering

https://publikationen.bibliothek.kit.edu/1000153795/149843013

Swarm v3: towards tera-scale amplicon clustering

Abstract

Similar works

Full text

Available Versions

KITopen