Towards Validating Risk Indicators Based on Measurement Theory (Extended version)

Due to the lack of quantitative information and for cost-efficiency, most risk assessment methods use partially ordered values (e.g. high, medium, low) as risk indicators. In practice it is common to validate risk indicators by asking stakeholders whether they make sense. This way of validation is subjective, thus error prone. If the metrics are wrong (not meaningful), then they may lead system owners to distribute security investments inefficiently. For instance, in an extended enterprise this may mean over investing in service level agreements or obtaining a contract that provides a lower security level than the system requires. Therefore, when validating risk assessment methods it is important to validate the meaningfulness of the risk indicators that they use. In this paper we investigate how to validate the meaningfulness of risk indicators based on measurement theory. Furthermore, to analyze the applicability of the measurement theory to risk indicators, we analyze the indicators used by a risk assessment method specially developed for assessing confidentiality risks in networks of organizations

Towards Validating Risk Indicators Based on Measurement Theory

Due to the lack of quantitative information and for cost-efficiency purpose, most risk assessment methods use partially ordered values (e.g. high, medium, low) as risk indicators.\ud In practice it is common to validate risk scales by asking stakeholders whether they make sense. This way of validation is subjective, thus error prone. If the metrics are wrong (not meaningful), then they may lead system owners to distribute security investments inefficiently. Therefore, when validating risk assessment methods it is important to validate the meaningfulness of the risk scales that they use. In this paper we investigate how to validate the meaningfulness of risk indicators based on measurement theory. Furthermore, to analyze the applicability of measurement theory to risk indicators, we analyze the indicators used by a particular risk assessment method specially developed for assessing confidentiality risks in networks of organizations

Aggregating opinions in non-uniform ordered qualitative scales

Producción CientíficaThis paper introduces a new voting system in the setting of ordered qualitative scales. The process is conducted in a purely ordinal way by considering an ordinal proximity measure that assigns an ordinal degree of proximity to each pair of linguistic terms of the qualitative scale. Once the agents assess the alternatives through the qualitative scale, the alternatives are ranked according to the medians of the ordinal degrees of proximity between the obtained individual assessments and the highest linguistic term of the scale. Since some alternatives may share the same median, an appropriate tie-breaking procedure is introduced. Some properties of the proposed voting system have been provided.Ministerio de Economía, Industria y Competitividad (Project ECO2016-77900-P

Methods for Ordinal Peer Grading

MOOCs have the potential to revolutionize higher education with their wide outreach and accessibility, but they require instructors to come up with scalable alternates to traditional student evaluation. Peer grading -- having students assess each other -- is a promising approach to tackling the problem of evaluation at scale, since the number of "graders" naturally scales with the number of students. However, students are not trained in grading, which means that one cannot expect the same level of grading skills as in traditional settings. Drawing on broad evidence that ordinal feedback is easier to provide and more reliable than cardinal feedback, it is therefore desirable to allow peer graders to make ordinal statements (e.g. "project X is better than project Y") and not require them to make cardinal statements (e.g. "project X is a B-"). Thus, in this paper we study the problem of automatically inferring student grades from ordinal peer feedback, as opposed to existing methods that require cardinal peer feedback. We formulate the ordinal peer grading problem as a type of rank aggregation problem, and explore several probabilistic models under which to estimate student grades and grader reliability. We study the applicability of these methods using peer grading data collected from a real class -- with instructor and TA grades as a baseline -- and demonstrate the efficacy of ordinal feedback techniques in comparison to existing cardinal peer grading methods. Finally, we compare these peer-grading techniques to traditional evaluation techniques.Comment: Submitted to KDD 201

Measuring the satisfaction of multimodal travelers for local transit services in different urban contexts

The importance of measuring customer satisfaction for a public transport service is apparent, even beyond more immediate marketing purposes. The present paper shows how satisfaction measures can be exploited to gain insights on the relationship between personal attitudes, transit use and urban context. We consider nine satisfaction measures of urban transit services, as expressed by a representative sample of Italian multimodal travelers (i.e. users of both private cars and public transport). We use correlations and correspondence analyses to show if and how each attribute is related to the levels of use of public transport, and how the relationship is affected by the urban context. Then we apply a recently developed method to combine ordinal variables into one score, by adapting it to work with large samples and with satisfaction measures which have a neutral point in the scale (i.e. ‘‘neither satisfied nor dissatisfied''). The resulting overall satisfaction levels and frequency of use were not correlated in our sample. We also found the highest satisfaction levels in smaller towns and the lowest ones in metropolitan cities. Since we focus on multimodal travelers, an interpretation paradigm is proposed according to which transit services must be well evaluated by car drivers in smaller towns in order to be considered a real alternative to cars. On the other hand, transit is more competitive on factual elements in larger cities, so that it can still be used by drivers, even if it is not very well evaluate

A Multi-Factorial Risk Prioritization Framework for Food-Borne Pathogens

To lower the incidence of human food-borne disease, experts and stakeholders have urged the development of a science- and risk-based management system in which food-borne hazards are analyzed and prioritized. A literature review shows that most approaches to risk prioritization developed to date are based on measures of health outcomes and do not systematically account for other factors that may be important to decision making. The Multi-Factorial Risk Prioritization Framework developed here considers four factors that may be important to risk managers: public health, consumer risk perceptions and acceptance, market-level impacts, and social sensitivity. The framework is based on the systematic organization and analysis of data on these multiple factors. The basic building block of the information structure is a three-dimensional cube based on pathogen-food-factor relationships. Each cell of the cube has an information card associated with it and data from the cube can be aggregated along different dimensions. The framework is operationalized in three stages, with each stage adding another dimension to decision-making capacity. The first stage is the information cards themselves that provide systematic information that is not pre-processed or aggregated across factors. The second stage maps the information on the various information cards into cobweb diagrams that create a graphical profile of, for example, a food-pathogen combination with respect to each of the four risk prioritization factors. The third stage is formal multi-criteria decision analysis in which decision makers place explicit values on different criteria in order to develop risk priorities. The process outlined above produces a ‘List A’ of priority food-pathogen combinations according to some aggregate of the four risk prioritization factors. This list is further vetted to produce ‘List B’, which brings in feasibility analysis by ranking those combinations where practical actions that have a significant impact are feasible. Food-pathogen combinations where not enough is known to identify any or few feasible interventions are included in ‘List C’. ‘List C’ highlights areas with significant uncertainty where further research may be needed to enhance the precision of the risk prioritization process. The separation of feasibility and uncertainty issues through the use of ‘Lists A, B, and C’ allows risk managers to focus separately on distinct dimensions of the overall prioritization. The Multi-Factorial Risk Prioritization Framework provides a flexible instrument that compares and contrasts risks along four dimensions. Use of the framework is an iterative process. It can be used to establish priorities across pathogens for a particular food, across foods for a particular pathogen and/or across specific food-pathogen combinations. This report provides a comprehensive conceptual paper that forms the basis for a wider process of consultation and for case studies applying the framework.risk analysis, risk prioritization, food-borne pathogens, benefits and costs