Advocacy, analysis and quality. The Bermuda triangle of Statistics

One might muse that what official statistics are to the consolidation of the modern nation state, composite indicators are to the emergence of post-modernity, – meaning by this the philosophical critique of the exact science and rational knowledge programme of Descartes and Galileo. Composite indicators give voice to a plurality of different actors and normative views of post-modernity. Not only has the use of composite indicators increased dramatically over the past ten to fifteen years, but the typology of use has widened. To make a recent example of a hitherto unheard use, in 2012 Bill Emmott, former Director of The Economist, used a battery of composite indicators, with a well dramatized presentational style, to describe the decline of a country, and this in a movie seen by millions of viewers. We consider composite indicators as an object populating a multidimensional space whose main axes are advocacy, analysis and quality. We review these issues and try to offer some elements of an epistemology of composite indicators.JRC.DDG.01-Econometrics and applied statistic

An inventory of Risk-related or Resilience-related Composite Indicators and Ratings

The document presents an overview of some popular risk-related and resilience-related composite indicators and ratings that are currently available in the literature. The description of these indices is taken directly from the author or organization, that is, they are excerpts from websites and publications. The sources from which these excerpts were taken are clearly listed in each index entry.JRC.G.3-Econometrics and applied statistic

The Multidimensional Poverty Assessment Tool (MPAT): Robustness issues and Critical assessment

The Multidimensional Poverty Assessment Tool (MPAT) was developed by the UN International Fund for Agricultural Development with a view to assess local-level poverty in rural regions around the globe. The MPAT is a survey-based thematic indicator of ten dimensions, from Food & Nutrition Security to Domestic Water Supply, Health & Healthcare, to Gender Equality. The aims of this validation report are: (a) to spot eventual conceptual and methodological shortcomings in the MPAT, (b) to identify suitable aggregation methods for the survey items, (c) to assess the internal consistency of the MPAT conceptual framework, and finally, (d) to offer snapshots of the MPAT results. The results show that the MPAT, upon some improvements throughout the entire development, would pass the ¿statistical¿ filters of index quality, and it could thus be reliably used to identify weaknesses and possible remedial actions, prioritize villages or even households with relatively low levels of rural poverty, and ultimately monitor and evaluate policy effectiveness. The analysis undertaken in this work provides no guarantee of the true ability of the MPAT to describe rural poverty world wide. Yet, it provides enough evidence that the MPAT cannot easily be falsified by methodological cunning.JRC.DG.G.9-Econometrics and applied statistic

Participation in Lifelong Learning in Europe: What Can Be Measured and Compared?

This publication analyses participation patterns in lifelong learning in the European countries. It describes the European political context in the field of lifelong learning and discusses the main monitoring issues at the EU level by looking at the EU benchmark set up in this area. Indicators on participation in education and training at various life-time stages are as well presented and analysed in the publication. A composite measure of the overall participation in lifelong learning for European countries is constructed and analysed in the publication. The lifelong learning index shows some progress in the European Union as a whole, mainly due to progress in pre-school and school/higher education participation. But it is too slow to reach the benchmark by 2010 unless major progress is achieved in participation in adult learning, where equity needs to be more fully addressed. In particular, some new Member states will have to increase their participation rates substantially, in order to catch up with the European average.JRC.G.9-Econometrics and statistical support to antifrau

Uncertainty and Sensitivity Analysis of the 2010 Environmental Performance Index

An assessment of the robustness of the 2010 Environmental Performance Index (EPI) ranks requires the evaluation of uncertainties underlying the index and the sensitivity of the country rankings to the methodological choices made during the development of the Index. To test this robustness, the Yale and Columbia University have continued their partnership with the Joint Research Centre (JRC) of the European Commission in Ispra, Italy. This JRC report shows that the 2010 EPI has an architecture that highlights the complexity of translating environmental stewardship into straightforward, clear-cut policy recipes. The trade-offs within the index dimensions are a reminder of the danger of compensability between dimensions while identifying the areas where more work is needed to achieve a coherent framework in particular in terms of the relative importance of the indicators that compose the EPI framework. The 2010 EPI is developed for 163 countries and is based on twenty five indicators grouped in ten policy categories: Environmental burden of disease, Air pollution (effects on humans), Water (effects on humans), Air Pollution (effects on ecosystem), Water (effects on ecosystem), Biodiversity & Habitat, Forestry, Fisheries, Agriculture and Climate Change. The EPI ranking is assessed by evaluating how sensitive the country ranks are to the assumptions made on the index structure and the aggregation of the 25 underlying indicators. The assumptions tested are: ¿ measurement error of the raw data, ¿ EPI structure ¿ grouping at policy categories, ¿ weights assigned to the indicators and/or to the policy categories, ¿ aggregation function at the policy or at the objectives level, and ¿ number of indicators or policy categories.JRC.DG.G.9-Econometrics and applied statistic

Sensitivity Analysis of the 2008 Environmental Performance Index

An assessment of the robustness of the 2008EPI results requires the evaluation of uncertainties underlying the index and the sensitivity of the country scores and rankings to the methodological choices made during the development of the Index. To test this robustness, the EPI team has continued its partnership with the Joint Research Centre (JRC) of the European Commission in Ispra, Italy. This JRC report shows that the 2008 EPI has an architecture that highlights the complexity of translating environmental stewardship into straightforward, clear-cut policy recipes. The trade-offs within the index dimensions are a reminder of the danger of compensability between dimensions while identifying the areas where more work is needed to achieve a coherent framework in particular in terms of the relative importance of the indicators that compose the EPI framework. The 2008 Environmental Performance Index (EPI) is developed for 149 countries and is based on 25 indicators in six policy categories: Environmental Health, Air Pollution, Water, Biodiversity and Habitat, Productive Natural Resources, Climate Change. The EPI aims to bring a data-driven, fact-based and empirical approach to environmental protection and global sustainability. The validity of the EPI scoring and respective ranking is assessed by evaluating how sensitive the country ranks are to the assumptions made on the index structure and the aggregation of the 25 underlying indicators. The assumptions tested are: ¿ measurement error of the raw data, ¿ choice of capping at selected targets for the 25 indicators, ¿ choice to correct for skewed distributions in the indicators values, ¿ weights of the indicators and/or of the subcomponents of the index, and finally ¿ aggregation function at the policy level (six policy categories).JRC.G.9-Econometrics and statistical support to antifrau

Knowledge Economy: Measures and Drivers

In the Knowledge-based Economy conceptual framework that was developed by MERIT a total of 115 individual indicators have been selected to measure the sub-dimensions of the KBE. The high number of individual indicators raises the issue of robustness of the ranking obtained by their aggregation into one composite measure. To tackle this issue a sensitivity analysis is a fundamental step of the KEI composite indicator. In particular, in building the KEI composite an innovative methodological assumption has been made, i.e. it is considered as the final composite index the frequency of all rankings obtained by means of all the simulations carried out. This allows us to deal with the criticism, often made to composite indicators that rankings are presented as they were under conditions of certainty while it is well known that this is not true in general terms. Most practitioners compute a composite indicator by a simple weighted summation mathematical model. Sometimes it is acknowledged that the ranking obtained is subject to some uncertainty, but this issue is treated as a kind of mathematical appendix for technical readers, and all policy suggestions are derived under the assumption of the linear aggregation model. Here the ranking presented is the one derived by considering the whole spectrum of uncertainty. It is important to note that this is a peculiar characteristic of the KEI composite. The scenarios, simulations and indicators developed by the JRC team answer five main research questions: 1. Is it possible to measure the knowledge economy? 2. What are the drivers of the knowledge economy? 3. How does knowledge economy relate to other complex dimensions? 4. Is it possible to reduce the total number of individual indicators of KEI conceptual framework without loosing any relevant information? 5. Are rankings useful at all for deriving policy suggestions?JRC.G.9-Econometrics and statistical support to antifrau

Corruption Perceptions Index 2012 - Statistical Assessment

The Corruption Perceptions Index (CPI) by Transparency International measures perceptions of corruption in the public sector in different countries around the world. Upon invitation of the CPI team, the JRC assessed the new methodology in the CPI 2012 and analyzed the consequences that come with this change. The statistical assessment of the Index was done along three main avenues: an evaluation of conceptual/statistical coherence of the index structure, an interpretation of the rankings based on significance tests, and an evaluation of the impact of key modelling assumptions (imputation and normalisation) on countries’ scores and ranks. The CPI 2012 passes all the statistical filters of the quality control. The main recommendation for the CPI team is to adjust the formula for the standard errors for the small population size (errors that are currently overestimated) and for policy makers to consider the statistical significance (by means of effect size for example) when comparing the CPI scores. The results make clear that even when differences in the CPI country scores are statistically significant they should be carefully interpreted.JRC.G.3-Econometrics and applied statistic

Higher Education Rankings: Robustness Issues and Critical Assessment

It has been often stated that European growth has been disappointing during the past three decades, remaining persistently lower than in the United States and this is to be attributed to the state of innovation and higher education in Europe. Such a conclusion has been in part based on higher education rankings such as the SJTU or the THES. These rankings can potentially be used by prospective students when choosing which university to attend, and bring attention to important issues such as ¿the student experience¿, employability and retention. The rankings and league tables also have a much wider impact ¿ for example, on institutions¿ reputations and potentially on the behaviour of academics, businesses and potential benefactors. Governing bodies take an interest in them as a means of assessing institutional performance, sometimes seizing on them in default of other, more sensitive indicators of institutional performance. There clearly is a demand for rankings in the higher education field, but there are also questions about their quality, impact and eventual validity of the conclusions, which in turn depend heavily upon the choice of indicators, the suitability of the methodologies used, the transparency of the processes and the robustness of the rankings. This JRC report has three main goals: - To throw a considerable amount of light on the approaches and limitations of the SJTU and THES rankings; - To assess the robustness of the two ranking systems with a view to identify for which universities the THES and SJTU ranking systems can be reliably used for drawing conclusions; - To identify factors behind the differences in university performance across Europe compared to the United States. We hope the debate will lead to improvements to league tables methodologies; enable users to better understand the complexities of the league tables, and avoid misunderstanding them; and to help higher education institutions develop approaches that help them satisfy the legitimate information needs of their stakeholders.JRC.G.9-Econometrics and statistical support to antifrau

Monitoring SMEs’ performance in Europe- Indicators fit for purpose

The Small Business Act for Europe (SBA) reflects the Commission's political will to recognize the central role of SMEs in the EU economy. The calculation of 2012 SBA principles has moved from a one-way design process of the previous versions to an iterative process with the JRC (since 2011) with a view to laying the foundation for a sound tool. This report presents the SBA framework, methodology, the refinements made and provides an additional assessment of the conceptual/statistical coherence and uncertainty analysis in the final tool. Extending the discussions offered in the 2012 SBA country factsheets, this report offers key messages on the European landscape of the SMEs achievements, such as the considerable differentiation among the SBA principles in terms of their dominant policy dynamics.JRC.G.3-Econometrics and applied statistic