Improving a gold standard: treating human relevance judgments of MEDLINE document pairs

Given prior human judgments of the condition of an object it is possible to use these judgments to make a maximal likelihood estimate of what future human judgments of the condition of that object will be. However, if one has a reasonably large collection of similar objects and the prior human judgments of a number of judges regarding the condition of each object in the collection, then it is possible to make predictions of future human judgments for the whole collection that are superior to the simple maximal likelihood estimate for each object in isolation. This is possible because the multiple judgments over the collection allow an analysis to determine the relative value of a judge as compared with the other judges in the group and this value can be used to augment or diminish a particular judge’s influence in predicting future judgments. Here we study and compare five different methods for making such improved predictions and show that each is superior to simple maximal likelihood estimates

Java table browser: transportation and presentation of large statistical tables over Network

This study describes a novel design of Java table browser using XML (jTBX). A standard table-browsing environment is provided in the form of Java applet which can be opened in any web browser such as Netscape, or Internet Explore with Java 1.2 plug-in installed. Some manipulation functions to tables are supported for various level of objects in table (table, sub-table, column, row, cell). A hierarchical Java tree provides the table of content (TOC) of available tables. Three tier architecture is used in jTBX system design. Remote database tier provides raw data from distributed sites. Web server tier generates the response in the standard XML format to the requests from the client side tier (table browser). Metadata for tables are integrated into the XML files (or streams) before being used by client Java applet. Multiple threads are generated for a large table transporting

The Knowledge in Multiple Human Relevance Judgments

INTRODUCTION We report two principal observations. The first is that pooled human relevance judgments lead to retrieval rankings that are superior to those that may be based on the judgments of a single individual. The multiple human judgments were originally constructed as part of test sets used in developing our in-house retrieval system [NCBI 1992]. The retrieval system is partially based on a precomputed list of closely related documents for each document in the system. Each document is treated as a query in a "query-by-example" paradigm, and the most similar documents found are Author's address: National Library of Medicine and National Institutes of Health, National Center for Biotechnology Information (NCBI), Bldg. 38A, Rm. 8S806, 8600 Rockville Pike, Bethesda, MD 20894; email: [email protected]. Permission to make digital / hard copy of part or all of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed fo