5 research outputs found
Implications of Inter-Rater Agreement on a Student Information Retrieval Evaluation
This paper is about an information retrieval evaluation on three different
retrieval-supporting services. All three services were designed to compensate
typical problems that arise in metadata-driven Digital Libraries, which are not
adequately handled by a simple tf-idf based retrieval. The services are: (1) a
co-word analysis based query expansion mechanism and re-ranking via (2)
Bradfordizing and (3) author centrality. The services are evaluated with
relevance assessments conducted by 73 information science students. Since the
students are neither information professionals nor domain experts the question
of inter-rater agreement is taken into consideration. Two important
implications emerge: (1) the inter-rater agreement rates were mainly fair to
moderate and (2) after a data-cleaning step which erased the assessments with
poor agreement rates the evaluation data shows that the three retrieval
services returned disjoint but still relevant result sets.Comment: 7 pages, 3 figures, LWA 2010, Workshop I
Applying Science Models for Search
The paper proposes three different kinds of science models as value-added
services that are integrated in the retrieval process to enhance retrieval
quality. The paper discusses the approaches Search Term Recommendation,
Bradfordizing and Author Centrality on a general level and addresses
implementation issues of the models within a real-life retrieval environment.Comment: 14 pages, 3 figures, ISI 201
Science Models as Value-Added Services for Scholarly Information Systems
The paper introduces scholarly Information Retrieval (IR) as a further
dimension that should be considered in the science modeling debate. The IR use
case is seen as a validation model of the adequacy of science models in
representing and predicting structure and dynamics in science. Particular
conceptualizations of scholarly activity and structures in science are used as
value-added search services to improve retrieval quality: a co-word model
depicting the cognitive structure of a field (used for query expansion), the
Bradford law of information concentration, and a model of co-authorship
networks (both used for re-ranking search results). An evaluation of the
retrieval quality when science model driven services are used turned out that
the models proposed actually provide beneficial effects to retrieval quality.
From an IR perspective, the models studied are therefore verified as expressive
conceptualizations of central phenomena in science. Thus, it could be shown
that the IR perspective can significantly contribute to a better understanding
of scholarly structures and activities.Comment: 26 pages, to appear in Scientometric