Data preparation and interannotator agreement: BioCreAtIvE Task 1B

<p>Abstract</p> <p>Background</p> <p>We prepared and evaluated training and test materials for an assessment of text mining methods in molecular biology. The goal of the assessment was to evaluate the ability of automated systems to generate a list of unique gene identifiers from PubMed abstracts for the three model organisms Fly, Mouse, and Yeast. This paper describes the preparation and evaluation of answer keys for training and testing. These consisted of lists of normalized gene names found in the abstracts, generated by adapting the gene list for the full journal articles found in the model organism databases. For the training dataset, the gene list was pruned automatically to remove gene names not found in the abstract; for the testing dataset, it was further refined by manual annotation by annotators provided with guidelines. A critical step in interpreting the results of an assessment is to evaluate the quality of the data preparation. We did this by careful assessment of interannotator agreement and the use of answer pooling of participant results to improve the quality of the final testing dataset.</p> <p>Results</p> <p>Interannotator analysis on a small dataset showed that our gene lists for Fly and Yeast were good (87% and 91% three-way agreement) but the Mouse gene list had many conflicts (mostly omissions), which resulted in errors (69% interannotator agreement). By comparing and pooling answers from the participant systems, we were able to add an additional check on the test data; this allowed us to find additional errors, especially in Mouse. This led to 1% change in the Yeast and Fly "gold standard" answer keys, but to an 8% change in the mouse answer key.</p> <p>Conclusion</p> <p>We found that clear annotation guidelines are important, along with careful interannotator experiments, to validate the generated gene lists. Also, abstracts alone are a poor resource for identifying genes in paper, containing only a fraction of genes mentioned in the full text (25% for Fly, 36% for Mouse). We found that there are intrinsic differences between the model organism databases related to the number of synonymous terms and also to curation criteria. Finally, we found that answer pooling was much faster and allowed us to identify more conflicting genes than interannotator analysis.</p

Pushing the Acquisition Innovation Envelope at the Office of Naval Research

Developing prototypes may require performers, all with different areas of expertise, working together to address the complexity required for a successful development effort. Current Federal Acquisition Regulation (FAR) policy makes it difficult for these collaborations to assemble efficiently. Complex research projects, such as the Office of Naval Research's Incapacitation Prediction in Expeditionary Domains: An Integrated Software Tool (I-PREDICT) project, which seeks to develop a computational model to predict human injury and functional incapacitation as a result of military hazards, often face difficulty when attempting to transition across the "valley of death"from development to adoption. A decision framework was developed and implemented for I-PREDICT to select the appropriate acquisition strategy aligned with the technical needs of the program. A three-phase implementation strategy was also designed, which included the use of an Other Transaction Authority (OTA) and the use of a Technical Committee to promote communication between performers. The resulting decision framework and implementation strategy may be used Navy-wide or across other military Services for R&D programs requiring acquisition flexibility coupled with collaborative technology development. Additionally, the research produced a customizable method for leveraging OTAs as a mechanism for development of complex prototypes depending on disparate kinds and sources of expertise.Naval Postgraduate School Acquisition Research Progra

Graph showing the differences between the participant's original F-measure and their final F-measure

<p><b>Copyright information:</b></p><p>Taken from "Data preparation and interannotator agreement: BioCreAtIvE Task 1B"</p><p></p><p>BMC Bioinformatics 2005;6(Suppl 1):S12-S12.</p><p>Published online 24 May 2005</p><p>PMCID:PMC1869005.</p><p></p