Contributed presentation to the 14th Bioinformatics Open Source Conference.2013-07Berlin14th Bioinformatics Open Source Conferenc

Leo, Simone

Pireddu, Luca

Zanetti, Gianluigi

P-arch

An Update onThe Seal Hadoop-based Sequence Processing ToolboxLuca Pireddu, Simone Leo, Gianluigi ZanettiCRS4Distributed Computing GroupJuly 20, 2013luca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 1 / 23Outline1 Motivation2 Introducing Seal3 New additions4 Conclusionluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 2 / 23Advances in sequencingRegular advances to sequencing technologies arelowering sequencing costsincreasing acquisition speedData production rate is growing exponentiallyE.g., 1 Illumina HT machine can now produce about 9 TB of raw dataper monthProcessing capacity is not growing this fast!luca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 3 / 23Growth of dataIn recent years there has been a steadyincrease in the amount of digitized dataavailableRise of data-driven businessesGoogle apparently processed 24 PB/day in 2009That's about 20000 Illumina run directories. . . per day!Relative to theirs, our problem doesn't seem so bigluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 4 / 23How can do they do that?luca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 5 / 23ScalingChange!Adopt new a computational paradigmScale horizontally, using lots of machinesWrite software that accepts and handles hardware failureSpread the datasplit it into partsdistribute them on the processing nodesMove the computation to the dataluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 6 / 23HadoopThose ideas are already implemented in an open source solutionHadoopRefactors distribution and robustness into a reusabe frameworkNot a second-class citizen: this is the system used by Twitter,Facebook, Yahoo, LinkedIn, and othersMaybe those processing a lot of sequencing data should try it. . .luca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 7 / 23CRS4CRS4 Sequencing and Genotyping PlatformCurrently the largest sequencing center in ItalySequencing Equipment: 3 Illumina HiSeq2000, plus older sequencersSequencing Capacity: about 5 Tbases/monthSince Sept. 2010 we've sequenced about. . .over 2000 whole-genome samples (mostly low-pass, somehigh-coverage)800 RNA samples100 exomesa handful (≈ 30) of ChIP-Seq samplesluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 8 / 23HadoopWe needed to scaleDecided to trying doing so with HadoopBut. . .. . . software has to be written speciﬁcally for Hadoopluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 9 / 23Outline1 Motivation2 Introducing Seal3 New additions4 Conclusionluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 10 / 23SealSeal is:a suite of distributed tools for processing HT sequencing dataruns on the Hadoop MapReduce frameworkGoalsScalable In cluster sizeIn data sizeRobust Resilient to node failure and transient cluster problemsSuﬃcient Implement all data-intensive steps of our sequencingpipelineluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 11 / 23At time of last publication. . .At the time of the last Seal publication we only had two tools:SeqalHadoop-based read alignerIncorporates BWA's alignment codeSimultaneously identiﬁes PCR duplicatesPrqRequired for SeqalReformat read and mate into the same recordCRE_242:1:2204:1453;1918#0 READ1 QUAL1 READ2 QUAL2luca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 12 / 23Outline1 Motivation2 Introducing Seal3 New additions4 Conclusionluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 13 / 23Bcl to QseqDescriptionExtract reads in qseq format from Illumina bcl ﬁles, using HadoopWraps Illumina's own bclToQseq utilityAutomatically runs many instances in parallel on Hadoop clusterBased on PydoopSupports all the original utility's features; adds Hadoop beneﬁtsluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 14 / 23DemuxDescriptionSeparate (demultiplex) samples in multiplexed runs.Analogous to functionality provided by Illumina's tools, but scalableSeparates samples into their own directoryCan, optionally, also separate reads by number (i.e. 1, 2)Can allow for substitution errors in barcodesluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 15 / 23RecabTableDescriptionCollect base quality statistics for recalibration using HadoopEquivalent to GATK CountCovariatesWalkerSupported factors (hard-coded):Read groupBase quality scoreSequencing cycleDinucleotideGenerates a table that can be fed to the GATK base qualityrecalibratorluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 16 / 23ReadSortDescriptionDistributed sorting of read alignmentsSorting required to create ﬁles usable by downstream softwareReadSort uses an algorithm based on TeraSortDivides work among all nodesGetting data out of the Seal environment:ReadSort leaves data in n sorted ﬁlesmerge_alignments program provided to concatenate them allluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 17 / 23File format supportSeal now uses HadoopBAM (I/O library for sequencing ﬁle formats onHadoop)Introduced support for data in multiple ﬁle formatsQseqFastqSAMBoth input and outputAlso supports transparent distributed compression and decompressionCodecs: snappy, bzip2, gzip (gzip input ﬁles not splittable)luca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 18 / 23Galaxy integrationWe have implemented a Galaxy wrappers for the Seal toolsA bit tricky since Hadoop doesn't follow Galaxy's modelNot directly in the Seal projectPlan to release them later this yearluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 19 / 23Outline1 Motivation2 Introducing Seal3 New additions4 Conclusionluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 20 / 23Production useSeal-based pipeline has been in use for over a yearOur experience has been positiveScales wellSigniﬁcantly improved processing throughputSigniﬁcantly lowered operational eﬀortJobs fail much less frequently, and they are relatively easy to monitorRobustness is important for automationluca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 21 / 23Future developmentComplete Hadoop 2 compatibilityBase quality recalibration (complete the workﬂow)OptimizationSupport RNA expression analysisSupport for eﬃcient columnar ﬁle formatsSupport for sequencing platforms other than IlluminaToo bad we won't have the time to do all this. Pull requests welcome!luca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 22 / 23Try it and contribute!Repositoryhttps://github.com/crs4/sealMight see more frequent activity athttps://github.com/ilveroluca/sealNext release should arrive soonCurrently updating documentationWeb site: http://biodoop-seal.sf.netThank you!luca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 23 / 23Try it and contribute!Repositoryhttps://github.com/crs4/sealMight see more frequent activity athttps://github.com/ilveroluca/sealNext release should arrive soonCurrently updating documentationWeb site: http://biodoop-seal.sf.netThank you!luca.pireddu@crs4.it (CRS4) Seal update July 20, 2013 23 / 23

An Update on the Seal Hadoop-based Sequencing Processing Toolbox

http://www.p-arch.it/bitstream/handle/11050/903/presentation.pdf?sequence=1&isAllowed=y

An Update on the Seal Hadoop-based Sequencing Processing Toolbox

Abstract

Similar works

Full text

Available Versions

P-arch