Article thumbnail

Threshold Average Precision (TAP-k): a measure of retrieval designed for bioinformatics

By Hyrum D. Carroll, Maricel G. Kann, Sergey L. Sheetlin and John L. Spouge

Abstract

Motivation: Since database retrieval is a fundamental operation, the measurement of retrieval efficacy is critical to progress in bioinformatics. This article points out some issues with current methods of measuring retrieval efficacy and suggests some improvements. In particular, many studies have used the pooled receiver operating characteristic for n irrelevant records (ROCn) score, the area under the ROC curve (AUC) of a ‘pooled’ ROC curve, truncated at n irrelevant records. Unfortunately, the pooled ROCn score does not faithfully reflect actual usage of retrieval algorithms. Additionally, a pooled ROCn score can be very sensitive to retrieval results from as little as a single query

Topics: Original Papers
Publisher: Oxford University Press
OAI identifier: oai:pubmedcentral.nih.gov:2894514
Provided by: PubMed Central

To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.

Suggested articles

Citations

  1. (1992). An information measure of retrieval performance.
  2. (2006). An introduction to ROC analysis.
  3. (1975). Area above ordinal dominance graph and area below receiver operating characteristic graph.
  4. (1998). Assessing sequence comparison methods with reliable structurally identified distant evolutionary relationships.
  5. (2003). Assessing sequence comparison methods with the average precision criterion.
  6. (2002). Associating genes with gene ontology codes using a maximum entropy analysis of biomedical literature.
  7. (2005). Automated methods of predicting the function of biological sequences using GO and BLAST.
  8. (2002). Bootstrapping and normalization for enhanced evaluations of pairwise sequence comparison.
  9. (2007). CDD: a conserved domain database for interactive domain family analysis.
  10. (2007). Comparing valuation metrics for sentence boundary detection.
  11. (2008). ConFunc - functional annotation in the twilight zone.
  12. (1967). Effectiveness of Information Retrieval Methods.
  13. (2007). FlowerPower: clustering proteins into domain architecture classes for phylogenomic inference of protein function,
  14. (2010). Homologous over-extension: a challenge for iterative similarity searches.
  15. (1999). IMPALA: matching a protein sequence against a collection of PSI-BLAST-constructed position-specific score matrices.
  16. (2001). Improving the accuracy of PSI-BLAST protein database searches with composition-based statistics and other refinements.
  17. Landgrebe,T.C.W.etal.(2006)Precision-recalloperatingcharacteristic(P-ROC)curves in imprecise environments.
  18. (2009). Measuring classifier performance: a coherent alternative to the area under the ROC curve.
  19. (1988). Measuring the accuracy of diagnostic systems.
  20. (1998). Profile hidden Markov models.
  21. (2004). Sensitivity and selectivity in protein structure comparison.
  22. (2007). The identification of complete domains within protein sequences using accurate E-values for semi-global alignment.
  23. (2005). The limits of protein sequence comparison?
  24. (2008). The Pfam protein families database.
  25. (2006). The Relationship Between Precision-Recall and ROC Curves.
  26. (1996). Use of receiver operating characteristic (ROC) analysis to evaluate sequence matching.