342 research outputs found
Combining Text and Formula Queries in Math Information Retrieval: Evaluation of Query Results Merging Strategies
Specific to Math Information Retrieval is combining text with mathematical
formulae both in documents and in queries. Rigorous evaluation of query
expansion and merging strategies combining math and standard textual keyword
terms in a query are given. It is shown that techniques similar to those known
from textual query processing may be applied in math information retrieval as
well, and lead to a cutting edge performance. Striping and merging partial
results from subqueries is one technique that improves results measured by
information retrieval evaluation metrics like Bpref
Queensland University of Technology at TREC 2005
The Information Retrieval and Web Intelligence (IR-WI) research group is a research team at the Faculty of Information Technology, QUT, Brisbane, Australia. The IR-WI group participated in the Terabyte and Robust track at TREC 2005, both for the first time. For the Robust track we applied our existing information retrieval system that was originally designed for use with structured (XML) retrieval to the domain of document retrieval. For the Terabyte track we experimented with an open source IR system, Zettair and performed two types of experiments. First, we compared Zettairās performance on both a high-powered supercomputer and a distributed system across seven midrange personal computers. Second, we compared Zettairās performance when a standard TREC title is used, compared with a natural language query, and a query expanded with synonyms. We compare the systems both in terms of efficiency and retrieval performance. Our results indicate that the distributed system is faster than the supercomputer, while slightly decreasing retrieval performance, and that natural language queries also slightly decrease retrieval performance, while our query expansion technique significantly decreased performance
DCU@TRECMed 2012: Using ad-hoc baselines for domain-specific retrieval
This paper describes the first participation of DCU in the TREC Medical Records Track (TRECMed). We performed some initial experiments on the 2011 TRECMed data based on the BM25 retrieval model. Surprisingly, we found that the standard BM25 model with default parameters, performs comparable to the best automatic runs submitted to TRECMed 2011 and would have resulted in rank four out of 29 participating groups. We expected that some form of domain adaptation would increase performance. However, results on the 2011 data proved otherwise: concept-based query expansion decreased performance, and filtering and reranking by term proximity also decreased performance slightly. We submitted four runs based on the BM25 retrieval model to TRECMed 2012 using standard BM25, standard query expansion, result filtering, and concept-based query expansion. Official results for 2012 confirm that domain-specific knowledge does not increase performance compared to the BM25 baseline as applied by us
Users' effectiveness and satisfaction for image retrieval
This paper presents results from an initial user
study exploring the relationship between system
effectiveness as quantified by traditional
measures such as precision and recall, and usersā
effectiveness and satisfaction of the results. The
tasks involve finding images for recall-based
tasks. It was concluded that no direct relationship
between system effectiveness and usersā
performance could be proven (as shown by
previous research). People learn to adapt to a
system regardless of its effectiveness. This study
recommends that a combination of attributes
(e.g. system effectiveness, user performance and
satisfaction) is a more effective way to evaluate
interactive retrieval systems. Results of this
study also reveal that users are more concerned
with accuracy than coverage of the search
results
Evaluating epistemic uncertainty under incomplete assessments
The thesis of this study is to propose an extended methodology for laboratory based Information Retrieval evaluation under incomplete relevance assessments. This new methodology aims to identify potential uncertainty during system comparison that may result from incompleteness. The adoption of this methodology is advantageous, because the detection of epistemic uncertainty - the amount of knowledge (or ignorance) we have about the estimate of a system's performance - during the evaluation process can guide and direct researchers when evaluating new systems over existing and future test collections. Across a series of experiments we demonstrate how this methodology can lead towards a finer grained analysis of systems. In particular, we show through experimentation how the current practice in Information Retrieval evaluation of using a measurement depth larger than the pooling depth increases uncertainty during system comparison
The Relationship between IR Effectiveness Measures and User Satisfaction
This paper presents an experimental study of users assessing the quality of Google web search results. In particular we look at how users' satisfaction correlates with the effectiveness of Google as quantified by IR measures such as precision and the suite of Cumulative Gain measures (CG, DCG, NDCG). Results indicate strong correlation between users' satisfaction, CG and precision, moderate correlation with DCG, with perhaps surprisingly negligible correlation with NDCG. The reasons for the low correlation with NDCG are examined
Parsimonious Language Models for a Terabyte of Text
The aims of this paper are twofold. Our first aim\ud
is to compare results of the earlier Terabyte tracks\ud
to the Million Query track. We submitted a number\ud
of runs using different document representations\ud
(such as full-text, title-fields, or incoming\ud
anchor-texts) to increase pool diversity. The initial\ud
results show broad agreement in system rankings\ud
over various measures on topic sets judged at both\ud
Terabyte and Million Query tracks, with runs using\ud
the full-text index giving superior results on\ud
all measures, but also some noteworthy upsets.\ud
Our second aim is to explore the use of parsimonious\ud
language models for retrieval on terabyte-scale\ud
collections. These models are smaller thus\ud
more efficient than the standard language models\ud
when used at indexing time, and they may also improve\ud
retrieval performance. We have conducted\ud
initial experiments using parsimonious models in\ud
combination with pseudo-relevance feedback, for\ud
both the Terabyte and Million Query track topic\ud
sets, and obtained promising initial results
Rhetorical relations for information retrieval
Typically, every part in most coherent text has some plausible reason for its
presence, some function that it performs to the overall semantics of the text.
Rhetorical relations, e.g. contrast, cause, explanation, describe how the parts
of a text are linked to each other. Knowledge about this socalled discourse
structure has been applied successfully to several natural language processing
tasks. This work studies the use of rhetorical relations for Information
Retrieval (IR): Is there a correlation between certain rhetorical relations and
retrieval performance? Can knowledge about a document's rhetorical relations be
useful to IR? We present a language model modification that considers
rhetorical relations when estimating the relevance of a document to a query.
Empirical evaluation of different versions of our model on TREC settings shows
that certain rhetorical relations can benefit retrieval effectiveness notably
(> 10% in mean average precision over a state-of-the-art baseline)
- ā¦