8 research outputs found
SMaSH: A Benchmarking Toolkit for Human Genome Variant Calling
Motivation: Computational methods are essential to extract actionable
information from raw sequencing data, and to thus fulfill the promise of
next-generation sequencing technology. Unfortunately, computational tools
developed to call variants from human sequencing data disagree on many of their
predictions, and current methods to evaluate accuracy and computational
performance are ad-hoc and incomplete. Agreement on benchmarking variant
calling methods would stimulate development of genomic processing tools and
facilitate communication among researchers.
Results: We propose SMaSH, a benchmarking methodology for evaluating human
genome variant calling algorithms. We generate synthetic datasets, organize and
interpret a wide range of existing benchmarking data for real genomes, and
propose a set of accuracy and computational performance metrics for evaluating
variant calling methods on this benchmarking data. Moreover, we illustrate the
utility of SMaSH to evaluate the performance of some leading single nucleotide
polymorphism (SNP), indel, and structural variant calling algorithms.
Availability: We provide free and open access online to the SMaSH toolkit,
along with detailed documentation, at smash.cs.berkeley.edu
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption. However, designing and implementing the systems that support ML models in real-world deployments remains a significant obstacle, in large part due to the radically different development and deployment profile of modern ML methods, and the range of practical concerns that come with broader adoption. We propose to foster a new systems machine learning research community at the intersection of the traditional systems and ML communities, focused on topics such as hardware systems for ML, software systems for ML, and ML optimized for metrics beyond predictive accuracy. To do this, we describe a new conference, MLSys, that explicitly targets research at the intersection of systems and machine learning with a program committee split evenly between experts in systems and ML, and an explicit focus on topics at the intersection of the two
Recommended from our members
SM a SH: a benchmarking toolkit for human genome variant calling
MotivationComputational methods are essential to extract actionable information from raw sequencing data, and to thus fulfill the promise of next-generation sequencing technology. Unfortunately, computational tools developed to call variants from human sequencing data disagree on many of their predictions, and current methods to evaluate accuracy and computational performance are ad hoc and incomplete. Agreement on benchmarking variant calling methods would stimulate development of genomic processing tools and facilitate communication among researchers.ResultsWe propose SMaSH, a benchmarking methodology for evaluating germline variant calling algorithms. We generate synthetic datasets, organize and interpret a wide range of existing benchmarking data for real genomes and propose a set of accuracy and computational performance metrics for evaluating variant calling methods on these benchmarking data. Moreover, we illustrate the utility of SMaSH to evaluate the performance of some leading single-nucleotide polymorphism, indel and structural variant calling algorithms.Availability and implementationWe provide free and open access online to the SMaSH tool kit, along with detailed documentation, at smash.cs.berkeley.ed
A large-scale evaluation of computational protein function prediction.
Automated annotation of protein function is challenging. As the number of sequenced genomes rapidly grows, the overwhelming majority of protein products can only be annotated computationally. If computational predictions are to be relied upon, it is crucial that the accuracy of these methods be high. Here we report the results from the first large-scale community-based critical assessment of protein function annotation (CAFA) experiment. Fifty-four methods representing the state of the art for protein function prediction were evaluated on a target set of 866 proteins from 11 organisms. Two findings stand out: (i) today's best protein function prediction algorithms substantially outperform widely used first-generation methods, with large gains on all types of targets; and (ii) although the top methods perform well enough to guide experiments, there is considerable need for improvement of currently available tools