A new parsimonious AHP methodology: assigning priorities to many objects by comparing pairwise few reference objects

We propose a development of the Analytic Hierarchy Process (AHP) permitting to use the methodology also for decision problems with a very large number of alternatives and several criteria. While the ap- plication of the original AHP method involves many pairwise comparisons between considered objects, that can be alternatives with respect to considered criteria or criteria between them, our parsimonious proposal is composed of five steps: (i) direct evaluation of the objects at hand; (ii) selection of some reference objects; (iii) application of the original AHP method to the reference objects; (iv) check of the consistency of the pairwise comparisons of AHP and the compatibility between the rating and the prior- itization with a subsequent discussion with the decision maker who can modify the rating or pairwise comparisons of reference objects; (v) revision of the direct evaluation on the basis of the prioritization supplied by AHP on reference objects. Our approach permits to avoid the distortion of comparing more relevant objects (reference points) with less relevant objects. Moreover, our AHP approach avoids rank reversal problems, that is, changes of the order in the prioritizations due to adding or removing one or more objects from the set of considered objects. The new proposal has been tested and experimentally validated

Evaluating the sustainability of impounded river systems and the cost-effectiveness of dam projects: An ecosystem services approach

In recent times, there has been increasing demand in the Canterbury region of New Zealand for the abstraction of water from rivers. The impact of this demand has lead to unacceptable minimum river flows and has adversely affected river ecology. In an effort to resolve these issues dams have been constructed. To evaluate the impact of these dam projects on all river values, an ecosystem services approach is developed. This ecosystem services approach coupled with various evaluation methods are applied for the purposes of assessing the cost-effectiveness of the Opuha Dam and the sustainability of the Opihi river system now modified by the Opuha Dam. To evaluate the cost-effectiveness of this dam project cost utility analysis is applied through the development of an ecosystem services index (ESI). The index is constructed from the aggregation of normalized indicators that represent each ecosystem service and preferential weights of each ecosystem service. The evaluation of sustainability is considered both according to weak and strong criteria. Weak sustainability is evaluated by a non-declining ecosystem services index over time. Strong sustainability is evaluated by the thresholds or safe minimum standards where an ecosystem service, as represented by an indicator, should not pass below. Fifteen ecosystem services provided by the Opihi river were identified and data for forty-two indicators was compiled to assess the provision of these services pre- and post-dam. Fifteen regional and six local stakeholder representatives were interviewed to elicit preferential weights for each ecosystem service. Assessment of both the ESI and safe minimum standards indicates that since dam construction the river has progressed towards both weak and strong sustainability in its provision of ecosystem services. The cost-effectiveness of the dam however was poor. While further work remains to refine the approach, namely to develop more effective indicators of river ecosystem services, the work does present a novel method to evaluate the impacts of dams on river systems.Cost utility analysis, dam projects, ecosystem services, impounded river systems, indicators and sustainability., Environmental Economics and Policy, Q15, Q25, Q27, Q51, Q58, Q57,

On crowdsourcing relevance magnitudes for information retrieval evaluation

4siMagnitude estimation is a psychophysical scaling technique for the measurement of sensation, where observers assign numbers to stimuli in response to their perceived intensity. We investigate the use of magnitude estimation for judging the relevance of documents for information retrieval evaluation, carrying out a large-scale user study across 18 TREC topics and collecting over 50,000 magnitude estimation judgments using crowdsourcing. Our analysis shows that magnitude estimation judgments can be reliably collected using crowdsourcing, are competitive in terms of assessor cost, and are, on average, rank-aligned with ordinal judgments made by expert relevance assessors. We explore the application of magnitude estimation for IR evaluation, calibrating two gain-based effectiveness metrics, nDCG and ERR, directly from user-reported perceptions of relevance. A comparison of TREC system effectiveness rankings based on binary, ordinal, and magnitude estimation relevance shows substantial variation; in particular, the top systems ranked using magnitude estimation and ordinal judgments differ substantially. Analysis of the magnitude estimation scores shows that this effect is due in part to varying perceptions of relevance: different users have different perceptions of the impact of relative differences in document relevance. These results have direct implications for IR evaluation, suggesting that current assumptions about a single view of relevance being sufficient to represent a population of users are unlikely to hold.partially_openopenMaddalena, Eddy; Mizzaro, Stefano; Scholer, Falk; Turpin, AndrewMaddalena, Eddy; Mizzaro, Stefano; Scholer, Falk; Turpin, Andre

The rank reversal problem in multi-criteria decision making : a literature review

Despite the importance of multicriteria decision-making (MCDM) techniques for constructing effective decision models, there are many criticisms due to the occurrence of a problem called rank reversal. Nevertheless, there is a lack of a systematic literature review on this important subject which involves different methods. This study reviews the pertinent literature on rank reversal, based on 130 related articles published from 1980 to 2015 in international journals, which were gathered and analyzed according to the following perspectives: multicriteria technique, year and journal in which the papers were published, co-authorship network, rank reversal types, and research goal. Thus our survey provides recommendations for future research, besides useful information and knowledge regarding rank reversal in the MCDM field