Motivation: Computational methods are essential to extract actionable
information from raw sequencing data, and to thus fulfill the promise of
next-generation sequencing technology. Unfortunately, computational tools
developed to call variants from human sequencing data disagree on many of their
predictions, and current methods to evaluate accuracy and computational
performance are ad-hoc and incomplete. Agreement on benchmarking variant
calling methods would stimulate development of genomic processing tools and
facilitate communication among researchers.
Results: We propose SMaSH, a benchmarking methodology for evaluating human
genome variant calling algorithms. We generate synthetic datasets, organize and
interpret a wide range of existing benchmarking data for real genomes, and
propose a set of accuracy and computational performance metrics for evaluating
variant calling methods on this benchmarking data. Moreover, we illustrate the
utility of SMaSH to evaluate the performance of some leading single nucleotide
polymorphism (SNP), indel, and structural variant calling algorithms.
Availability: We provide free and open access online to the SMaSH toolkit,
along with detailed documentation, at smash.cs.berkeley.edu
