Unbiased Comparative Evaluation of Ranking Functions

Eliciting relevance judgments for ranking evaluation is labor-intensive and costly, motivating careful selection of which documents to judge. Unlike traditional approaches that make this selection deterministically, probabilistic sampling has shown intriguing promise since it enables the design of estimators that are provably unbiased even when reusing data with missing judgments. In this paper, we first unify and extend these sampling approaches by viewing the evaluation problem as a Monte Carlo estimation task that applies to a large number of common IR metrics. Drawing on the theoretical clarity that this view offers, we tackle three practical evaluation scenarios: comparing two systems, comparing $k$ systems against a baseline, and ranking $k$ systems. For each scenario, we derive an estimator and a variance-optimizing sampling distribution while retaining the strengths of sampling-based evaluation, including unbiasedness, reusability despite missing data, and ease of use in practice. In addition to the theoretical contribution, we empirically evaluate our methods against previously used sampling heuristics and find that they generally cut the number of required relevance judgments at least in half.Comment: Under review; 10 page

Regeneration of Nardus stricta subalpine grasslands in the Giant Mountains [Krkonose]

The origin of Nardus stricta dominated subalpine grassland (Nardo-Caricion rigidae alliance) is a frequently discussed topic in the Giant Mountains (Karkonosze in the Czech Republic). Many researchers considered them, as secondary stands arisen after Pinus mugo removal and by consequent oligotrophization of original plant communities, caused by long-term rough grazing and hay making activities without manuring. On the contrary, they are recognized as natural due to inability of generative reproduction of N. stricta and the very slow vegetative spread there. The aim of this study was to find proofs for generative reproduction of mat grass in subalpine conditions of the Giant Mountains. We identified a Pinus mugo nursery with arable land abandoned in 1956 and compared its vegetation with that of the surrounding area. Dense and homogenous sward dominated by N. stricta developed during the succession on the old arable land for 48 years, and it is an indisputable proof of generative reproduction of N. stricta there. Synthesizing historical facts on human activities in the past and the results of our contemporary vegetation analysis, we conclude that the Nardo-Caricion rigidae grassland was capable to spread relatively quickly, when agricultural activities above the upper timber line were introduced

Farmer decision making and its effect on subalpine grassland succession in the Giant Mts., Czech Republic

Nitrogen deposition is generally considered as a main reason for many recent plant expansions, but management changes are often not taken into account. Understanding the effects of agriculture management in the past can be decisive in the explanation of plant expansions at present. In order to understand the spread of Molinia caerulea and Calamagrostis villosa into Nardus stricta dominated subalpine grassland in the Giant Mts. (Krkonoše, Karkonosze), we undertook an experiment to explain farmer decision making and we discussed its effect on grassland succession. We measured mowing productivity, yields, biomass quality and nutrient removal in N. stricta, M. caerulea, and C. villosa dominated swards. With regard to defoliation management performed on the subalpine grasslands for at least 500 years and cancelled after the Second World War, we found the following results and conclusions. 1. Mowing productivity, yield and forage quality were lowest in the N. stricta sward, therefore farmers preferred to harvest C. villosa and M. caerulea stands if they had the possibility to select a sward for mowing. 2. Removal of all nutrients was the lowest in the N. stricta sward. With respect to these facts, the competitive advantage of N. stricta is obvious under long-term scything without fertilization. Consequently, the recent increase of defoliation sensitive species M. caerulea and C. villosa above the timber line must be evaluated with respect to both: termination of agricultural activities and recent nitrogen deposition