Unbiased Comparative Evaluation of Ranking Functions

Eliciting relevance judgments for ranking evaluation is labor-intensive and costly, motivating careful selection of which documents to judge. Unlike traditional approaches that make this selection deterministically, probabilistic sampling has shown intriguing promise since it enables the design of estimators that are provably unbiased even when reusing data with missing judgments. In this paper, we first unify and extend these sampling approaches by viewing the evaluation problem as a Monte Carlo estimation task that applies to a large number of common IR metrics. Drawing on the theoretical clarity that this view offers, we tackle three practical evaluation scenarios: comparing two systems, comparing $k$ systems against a baseline, and ranking $k$ systems. For each scenario, we derive an estimator and a variance-optimizing sampling distribution while retaining the strengths of sampling-based evaluation, including unbiasedness, reusability despite missing data, and ease of use in practice. In addition to the theoretical contribution, we empirically evaluate our methods against previously used sampling heuristics and find that they generally cut the number of required relevance judgments at least in half.Comment: Under review; 10 page

Mitigating the Position Bias of Transformer Models in Passage Re-Ranking

Supervised machine learning models and their evaluation strongly depends on the quality of the underlying dataset. When we search for a relevant piece of information it may appear anywhere in a given passage. However, we observe a bias in the position of the correct answer in the text in two popular Question Answering datasets used for passage re-ranking. The excessive favoring of earlier positions inside passages is an unwanted artefact. This leads to three common Transformer-based re-ranking models to ignore relevant parts in unseen passages. More concerningly, as the evaluation set is taken from the same biased distribution, the models overfitting to that bias overestimate their true effectiveness. In this work we analyze position bias on datasets, the contextualized representations, and their effect on retrieval results. We propose a debiasing method for retrieval datasets. Our results show that a model trained on a position-biased dataset exhibits a significant decrease in re-ranking effectiveness when evaluated on a debiased dataset. We demonstrate that by mitigating the position bias, Transformer-based re-ranking models are equally effective on a biased and debiased dataset, as well as more effective in a transfer-learning setting between two differently biased datasets

A Qualitative Study of Incest Offender Implicit Theories With The Help Of A Modified Assessment Tool

This study intends to provide some insight into the various Implicit Theories (ITs) harboured by child sex offenders. ITs (Ward & Keenan, 1999) are the distorted beliefs which enable sex offenders to justify their actions and avoid taking responsibility for their offenses. The present study will examine the ITs of various types of incest offenders as these have not been studied in isolation from those of other types of child molesters. Besides facilitating offending, such ITs may act as responsivity barriers in the treatment of incest offenders. The primary aim of this study is to determine the nature of these distorted beliefs through the use of a modified assessment tool, then, do a qualitative analysis of a selected group from the initial sample to determine what these ITs look like in individual or exemplar cases, and finally discuss how such beliefs may hamper treatment success. By enhancing the understanding of these ITs, the study hopes to emphasize on the importance of strategies to avoid the obstacles in the treatment process caused by such distorted beliefs and help therapists achieve improved treatment results with incest offenders. ITs are currently considered extremely important in understanding the offending behaviour of sex offenders, as well as their attitude towards their offenses and their victims (Brown, Gray, Snowden, 2009). For example, child molesters believe that children incite sexual involvement from them through their actions (for example, sitting in the lap of the offender, hugging or kissing the offender). Such beliefs enable sex offenders to validate their forced sexual intimacy with children and also allow such offenders to justify their continued offenses against their victims (Rice & Harris, 2002). Ward (2002) argued that ITs determine how the offenders interpret their experience with their victims. Offenders are thought to reinterpret, reject, or reconstruct a sexual offence against a child in the face of an inconsistency between their ITs and the evidence (e.g., the child whom an offender may believe is interested in sex may scream or cry when assaulted rather than appear to be a willing participant), but rarely are the ITs modified (Ward & Keenan, 1999). Some studies have suggested that it takes rather compelling evidence on the contrary for the offenders to consider modifying their ITs (Rice & Harris, 2002). Hence, this study was intended to observe these ITs specifically among incest offenders to determine how they look in exemplar cases. The findings of the study strongly indicate the existence of ITs and that offenders do utilize them in order to justify their offense to themselves and the world around them. The current study also hopes to also shed some light on the importance of taking ITs into consideration in strategizing effective treatment strategies for incest offenders

The Productivity Slowdown, Measurement Issues, and the Explosion of Computer Power

macroeconomics, Productivity Slowdown, Measurement Issues, Computer Power