Development and evaluation of a geographic information retrieval system using fine grained toponyms

Geographic information retrieval (GIR) is concerned with returning information in response to an information need, typically expressed in terms of a thematic and spatial component linked by a spatial relationship. However, evaluation initiatives have often failed to show significant differences between simple text baselines and more complex spatially enabled GIR approaches. We explore the effectiveness of three systems (a text baseline, spatial query expansion, and a full GIR system utilizing both text and spatial indexes) at retrieving documents from a corpus describing mountaineering expeditions, centred around fine grained toponyms. To allow evaluation, we use user generated content (UGC) in the form of metadata associated with individual articles to build a test collection of queries and judgments. The test collection allowed us to demonstrate that a GIR-based method significantly outperformed a text baseline for all but very specific queries associated with very small query radii. We argue that such approaches to test collection development have much to offer in the evaluation of GIR

Creating test collections from user generated content for GIR evaluation

Evaluation of the effectiveness of Geographic Information Retrieval (GIR) systems is challenging and time consuming. We describe an approach to such evaluations, where we use user generated content in the form of text and associated metadata to build a large test colletion automatically. We can thus show that the UGC test collection is useful for evaluating and exploring some of the critical aspects of a GIR, for instance by submitting large numbers of queries