Aslib Cranfield research project: report on the testing and analysis of an investigation into the comparative efficiency of indexing systems

This volume continues the account of the Aslib-Cranfield project as given in the "Final Report of the First Stage of an Investigation into the Comparative Efficiency of Indexing Systems". The major portion of the two years spent on this present stage has been involved with the analysis of the considerable amount of data which was obtained from the main test programme. A difficulty in this work was in deciding on the type of analysis which would be most likely to yield valuable information. In order to keep this volume within reasonable limits, it has been necessary to select from the analysis that was done, and even so in many cases only brief examples are given. The major emphasis has been placed on the reasons for failure to retrieve source documents, for this is considered to give some of the most interesting results of the project and has not, to our knowledge, been previously attempted. Of possible equal importance, but certainly more difficult to evaluate, is the reason for the retrieval of non-relevant references. This analysis has not been attempted within the present work, but will be one of the matters to be investigated in the continuation of the project.National Science FoundationAslibCollege of Aeronautics, Cranfiel

Aslib Cranfield research project - A report on a test of the index of metallurgical literature of Western Reserve University

For many years the index to metallurgical literature at the Center for Documentation at Western Reserve University has been recognised as one of the most advanced indexing systems in use. It has been developed over many years with the assistance of grants from the American Society of Metals, and was put on an operational basis in 1960, when it was made available to members of the Society by subscription. At Cranfield, a by-product of the test on the efficiency of indexing systems had been the development of a method for testing existing systems. The opportunity of trying-out this method on the W.R.U. index was greatly appreciated by the Cranfield group; that it was an experiment in the use of the technique, equally as much as a test of the W.R.U. index, was accepted by both groups from the start. However, as the work developed, it became clear that the test was also becoming a research investigation in its own right, and that it was making significant additions to our knowledge of the operation of indexing systems. The following report covers all three stages of the work.National Science FoundationAslibCollege of Aeronautics, Cranfiel

Aslib Cranfield research project - Factors determining the performance of indexing systems; Volume 2, Test results

The test results are presented for a number of different index languages using various devices which affect recall or precision. Within the environment of this test, it is shown that the best performance was obtained with the group of eight index languages which used single terms. The group of fifteen index languages which were based on concepts gave the worst performance, while a group of six index languages based on the Thesaurus of Engineering Terms of the Engineers Joint Council were intermediary. Of the single term index languages, the only method of improving performance was to group synonyms and word forms, and any broader groupings of terms depressed performance. The use of precision devices such as links gave no advantage as compared to the basic device of simple coordination. All results have to be considered within the context of the experimental environment, but they can be said to substantiate or clarify many of the findings of Cranfield I. It is conclusively shown that an inverse relationship exists between recall and precision, whatever the variable may be that is being changed. The two factors which appear most likely to affect performance are the level of exhaustivity of indexing and the level of specificity of the terms in the index language. For any given operational situation, the optimum levels cannot be categorically stated in advance, but can only be determined by an evaluation of the system, the main consideration probably being the subject field. It would be unusual if the characteristics of the subject field used for this test were such as to make it unique, so the high performance obtained with the single terms in natural language can be considered to be of some importance in regard to the use of natural language text as input to mechanised systems.National Science FoundationAslibCollege of Aeronautics, Cranfiel

A comparative evaluation of searching by controlled language and natural language in experimental N.A.S.A. data base

European Space Agency Contract Report, ESA contract number I/432An evaluation test was made of an experimental data-base prepared by the Space Documentation Service of the European Space Agency, consisting of some 44,000 items from NASA STAR for 1973 and 1974. With this data-base it was possible to search on natural language terms in the titles and abstracts, in addition to the normal searches on controlled language index terms. The on-line searches were carried out at four centres, each centre being responsible for ten questions, with two searches in the alternative search modes being made by different people for each question. Up to twenty-five documents retrieved in the two searches for each question were sent to the originator of the question for relevance assessment. The results are presented in a number of different ways, but in every case the natural language searches showed a significantly higher recall ratio than the controlled language, with little difference in the precision ratios. It is suggested that the main reason for the superiority of natural language searching is the greater exhaustivity of the abstracts as compared to the indexing

Proposals for an investigation into the efficiency of various retrieval systems

Research proposa

The effect of variations in relevance assessments in comparative experimental tests of index languages

It was desired to check whether the unexpected test results obtained in Cranfield II had been influenced by the relevance decisions. Three new sets of relevance decisions were therefore obtained, and the Cranfield II results were re-calculated for nineteen index languages on the basis of the new relevance judgements. A rank order of the index languages was found; in no case did the correlation co-efficient of any of the three new rank orders fall below 0.921 when compared with the original Cranfield II results. The findings of a similar type of test by Lesk and Salton are considered, and the conclusion is reached that the results of Cranfield II had showed that the superiority of the single term index languages were not significantly affected by relevance judgements.Great Britain, Department of Education and Science, Office of Scientific and Technical Informatio

The critical appraisal of information retrieval systems

Paper presented at The International Congress of the International Federation for Documentation, Moscow, September 1968The paper reviews one set of methods which can be used in the critical appraisal of various stages of an information retrieval system