An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric

Many evaluation metrics have been defined to evaluate the effectiveness ad-hoc retrieval and search result diversification systems. However, it is often unclear which evaluation metric should be used to analyze the performance of retrieval systems given a specific task. Axiomatic analysis is an informative mechanism to understand the fundamentals of metrics and their suitability for particular scenarios. In this paper, we define a constraint-based axiomatic framework to study the suitability of existing metrics in search result diversification scenarios. The analysis informed the definition of Rank-Biased Utility (RBU) -- an adaptation of the well-known Rank-Biased Precision metric -- that takes into account redundancy and the user effort associated to the inspection of documents in the ranking. Our experiments over standard diversity evaluation campaigns show that the proposed metric captures quality criteria reflected by different metrics, being suitable in the absence of knowledge about particular features of the scenario under study.Comment: Original version: 10 pages. Preprint of full paper to appear at SIGIR'18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, July 8-12, 2018, Ann Arbor, MI, USA. ACM, New York, NY, US

Diversification Preferences in the Theory of Choice

Diversification represents the idea of choosing variety over uniformity. Within the theory of choice, desirability of diversification is axiomatized as preference for a convex combination of choices that are equivalently ranked. This corresponds to the notion of risk aversion when one assumes the von-Neumann-Morgenstern expected utility model, but the equivalence fails to hold in other models. This paper studies axiomatizations of the concept of diversification and their relationship to the related notions of risk aversion and convex preferences within different choice theoretic models. Implications of these notions on portfolio choice are discussed. We cover model-independent diversification preferences, preferences within models of choice under risk, including expected utility theory and the more general rank-dependent expected utility theory, as well as models of choice under uncertainty axiomatized via Choquet expected utility theory. Remarks on interpretations of diversification preferences within models of behavioral choice are given in the conclusion

Efficient Diversification of Web Search Results

In this paper we analyze the efficiency of various search results diversification methods. While efficacy of diversification approaches has been deeply investigated in the past, response time and scalability issues have been rarely addressed. A unified framework for studying performance and feasibility of result diversification solutions is thus proposed. First we define a new methodology for detecting when, and how, query results need to be diversified. To this purpose, we rely on the concept of "query refinement" to estimate the probability of a query to be ambiguous. Then, relying on this novel ambiguity detection method, we deploy and compare on a standard test set, three different diversification methods: IASelect, xQuAD, and OptSelect. While the first two are recent state-of-the-art proposals, the latter is an original algorithm introduced in this paper. We evaluate both the efficiency and the effectiveness of our approach against its competitors by using the standard TREC Web diversification track testbed. Results shown that OptSelect is able to run two orders of magnitude faster than the two other state-of-the-art approaches and to obtain comparable figures in diversification effectiveness.Comment: VLDB201

Axioms for graph clustering quality functions

We investigate properties that intuitively ought to be satisfied by graph clustering quality functions, that is, functions that assign a score to a clustering of a graph. Graph clustering, also known as network community detection, is often performed by optimizing such a function. Two axioms tailored for graph clustering quality functions are introduced, and the four axioms introduced in previous work on distance based clustering are reformulated and generalized for the graph setting. We show that modularity, a standard quality function for graph clustering, does not satisfy all of these six properties. This motivates the derivation of a new family of quality functions, adaptive scale modularity, which does satisfy the proposed axioms. Adaptive scale modularity has two parameters, which give greater flexibility in the kinds of clusterings that can be found. Standard graph clustering quality functions, such as normalized cut and unnormalized cut, are obtained as special cases of adaptive scale modularity. In general, the results of our investigation indicate that the considered axiomatic framework covers existing `good' quality functions for graph clustering, and can be used to derive an interesting new family of quality functions.Comment: 23 pages. Full text and sources available on: http://www.cs.ru.nl/~T.vanLaarhoven/graph-clustering-axioms-2014

On the impossibility of fair risk allocation

Measuring and allocating risk properly are crucial for performance evaluation and internal capital allocation of portfolios held by banks, insurance companies, investment funds and other entities subject to financial risk. We show that by using a coherent measure of risk it is impossible to allocate risk satisfying the natural requirements of (Solution) Core Compatibility, Equal Treatment Property and Strong Monotonicity. To obtain the result we characterize the Shapley value on the class of totally balanced games and also on the class of exact games

Risk in a large claims insurance market with bipartite graph structure

We model the influence of sharing large exogeneous losses to the reinsurance market by a bipartite graph. Using Pareto-tailed claims and multivariate regular variation we obtain asymptotic results for the Value-at-Risk and the Conditional Tail Expectation. We show that the dependence on the network structure plays a fundamental role in their asymptotic behaviour. As is well-known in a non-network setting, if the Pareto exponent is larger than 1, then for the individual agent (reinsurance company) diversification is beneficial, whereas when it is less than 1, concentration on a few objects is the better strategy. An additional aspect of this paper is the amount of uninsured losses which have to be convered by society. In the situation of networks of agents, in our setting diversification is never detrimental concerning the amount of uninsured losses. If the Pareto-tailed claims have finite mean, diversification turns out to be never detrimental, both for society and for individual agents. In contrast, if the Pareto-tailed claims have infinite mean, a conflicting situation may arise between the incentives of individual agents and the interest of some regulator to keep risk for society small. We explain the influence of the network structure on diversification effects in different network scenarios