Unbiased Comparative Evaluation of Ranking Functions

Eliciting relevance judgments for ranking evaluation is labor-intensive and costly, motivating careful selection of which documents to judge. Unlike traditional approaches that make this selection deterministically, probabilistic sampling has shown intriguing promise since it enables the design of estimators that are provably unbiased even when reusing data with missing judgments. In this paper, we first unify and extend these sampling approaches by viewing the evaluation problem as a Monte Carlo estimation task that applies to a large number of common IR metrics. Drawing on the theoretical clarity that this view offers, we tackle three practical evaluation scenarios: comparing two systems, comparing $k$ systems against a baseline, and ranking $k$ systems. For each scenario, we derive an estimator and a variance-optimizing sampling distribution while retaining the strengths of sampling-based evaluation, including unbiasedness, reusability despite missing data, and ease of use in practice. In addition to the theoretical contribution, we empirically evaluate our methods against previously used sampling heuristics and find that they generally cut the number of required relevance judgments at least in half.Comment: Under review; 10 page

Semantic concept detection in imbalanced datasets based on different under-sampling strategies

Semantic concept detection is a very useful technique for developing powerful retrieval or filtering systems for multimedia data. To date, the methods for concept detection have been converging on generic classification schemes. However, there is often imbalanced dataset or rare class problems in classification algorithms, which deteriorate the performance of many classifiers. In this paper, we adopt three “under-sampling” strategies to handle this imbalanced dataset issue in a SVM classification framework and evaluate their performances on the TRECVid 2007 dataset and additional positive samples from TRECVid 2010 development set. Experimental results show that our well-designed “under-sampling” methods (method SAK) increase the performance of concept detection about 9.6% overall. In cases of extreme imbalance in the collection the proposed methods worsen the performance than a baseline sampling method (method SI), however in the majority of cases, our proposed methods increase the performance of concept detection substantially. We also conclude that method SAK is a promising solution to address the SVM classification with not extremely imbalanced datasets

Size and shape constancy in consumer virtual reality

With the increase in popularity of consumer virtual reality headsets, for research and other applications, it is important to understand the accuracy of 3D perception in VR. We investigated the perceptual accuracy of near-field virtual distances using a size and shape constancy task, in two commercially available devices. Participants wore either the HTC Vive or the Oculus Rift and adjusted the size of a virtual stimulus to match the geometric qualities (size and depth) of a physical stimulus they were able to refer to haptically. The judgments participants made allowed for an indirect measure of their perception of the egocentric, virtual distance to the stimuli. The data show under-constancy and are consistent with research from carefully calibrated psychophysical techniques. There was no difference in the degree of constancy found in the two headsets. We conclude that consumer virtual reality headsets provide a sufficiently high degree of accuracy in distance perception, to allow them to be used confidently in future experimental vision science, and other research applications in psychology

The Life Satisfaction Approach to Environmental Valuation

In many countries environmental policies and regulations are implemented to improve environmental quality and thus individuals’ well-being. However, how do individuals value the environment? In this paper, we review the Life Satisfaction Approach (LSA) representing a new non-market valuation technique. The LSA builds on the recent development of subjective well-being research in economics and takes measures of reported life satisfaction as an empirical approximation to individual welfare. Micro-econometric life satisfaction functions are estimated taking into account environmental conditions along with income and other covariates. The estimated coefficients for the environmental good and income can then be used to calculate the implicit willingness-to-pay for the environmental good.life satisfaction approach, subjective well-being, non-market valuation, costbenefit analysis, air pollution

The Life Satisfaction Approach to Environmental Valuation

The present paper examines the joint effect of fixed-term employment and work organization on job satisfaction using individual-level data from the German Socio-Economic Panel GSOEP). Specifically, we analyze whether workers who are heterogeneous in terms of the type of working contract (fixed-term vs. permanent) do also differ with regard to job satisfaction, when they perform under comparable work organizational conditions. Such information would be quite valuable for employers, because they can learn about the responsiveness of heterogeneous workers to innovative work organizational practices. For this purpose, we at first estimate a linear fixed effects model, thereby controlling for unobserved time-constant characteristics. In a second step, we account for potential remaining endogeneity by combining the fixed effects approach with a two-stage estimation strategy. Our empirical results show that in terms of job satisfaction fixed-term workers and their permanent counterparts respond differently to a number of organizational practices including task diversity, employee involvement, social relations at work, general working conditions, and career prospects. The results may be used by employers to improve their concept of diversity management and specifically the job design of heterogeneous workers. �