A Framework for Economics-Driven Assessment of Data Quality Decisions

Economic perspectives have raised growing attention in recent data quality (DQ) literature, as studies have associated DQ decisions with major cost-benefit tradeoffs. Despite the growing interest, DQ research has not yet developed a robust, agreedupon view for assessing and studying the link between DQ and economic outcome. As a contribution, this study proposes a framework, which links costs to the decisions made in managing the information process and improving the DQ, and benefits to the use of information-product outcomes by data consumers. Considering past research contributions, we develop this framework further into a high-level optimization model that permits quantitative assessment of cost-benefit tradeoffs, towards economically-optimal DQ decisions. We demonstrate a possible use of the proposed framework and the derived model, and highlight their potential contribution to an economics-driven view of DQ issues in both research and practice

Quality Management Framework for Climate Datasets

Data from a variety of research programmes are increasingly used by policy makers, researchers, and private sectors to make data-driven decisions related to climate change and variability. Climate services are emerging as the link to narrow the gap between climate science and downstream users. The Global Framework for Climate Services (GFCS) of the World Meteorological Organization (WMO) offers an umbrella for the development of climate services and has identified the quality assessment, along with its use in user guidance, as a key aspect of the service provision. This offers an extra stimulus for discussing what type of quality information to focus on and how to present it to downstream users. Quality has become an important keyword for those working on data in both the private and public sectors and significant resources are now devoted to quality management of processes and products. Quality management guarantees reliability and usability of the product served, it is a key element to build trust between consumers and suppliers. Untrustworthy data could lead to a negative economic impact at best and a safety hazard at worst. In a progressive commitment to establish this relation of trust, as well as providing sufficient guidance for users, the Copernicus Climate Change Service (C3S) has made significant investments in the development of an Evaluation and Quality Control (EQC) function. This function offers a homogeneous user-driven service for the quality of the C3S Climate Data Store (CDS). Here we focus on the EQC component targeting the assessment of the CDS datasets, which include satellite and in-situ observations, reanalysis, climate projections, and seasonal forecasts. The EQC function is characterised by a two-tier review system designed to guarantee the quality of the dataset information. While the need of assessing the quality of climate data is well recognised, the methodologies, the metrics, the evaluation framework, and how to present all this information to the users have never been developed before in an operational service, encompassing all the main climate dataset categories. Building the underlying technical solutions poses unprecedented challenges and makes the C3S EQC approach unique. This paper describes the development and the implementation of the operational EQC function providing an overarching quality management service for the whole CDS data.This study is based on work carried out in the C3S_512 contract funded by Copernicus Programme and operated by ECMWF on behalf of the European Commission (Service Contract number: ECMWF/COPERNICUS720187C3S_512_BSC). We would like to acknowledge the work of colleagues from several European institutions, the data providers and C3S, who contributed to the development of the EQC framework as well as to the QAR production. We would also like to acknowledge the focus group users, who took time to review and provide valuable feedback on the QARs, QATs, minimum requirements and the CDS quality assessment tab. The authors are grateful to the anonymous reviewers for their constructive comments that have helped for the improvement of this paper.Peer Reviewed"Article signat per 23 autors/es: Carlo Lacagnina , Francisco Doblas-Reyes, Gilles Larnicol, Carlo Buontempo, André Obregón, Montserrat Costa-Surós, Daniel San-Martín, Pierre-Antoine Bretonnière, Suraj D. Polade, Vanya Romanova, Davide Putero, Federico Serva, Alba Llabrés-Brustenga, Antonio Pérez, Davide Cavaliere, Olivier Membrive, Christian Steger, Núria Pérez-Zanón, Paolo Cristofanelli, Fabio Madonna, Marco Rosoldi, Aku Riihelä, Markel García Díez"Postprint (published version

Optimal management of bio-based energy supply chains under parametric uncertainty through a data-driven decision-support framework

This paper addresses the optimal management of a multi-objective bio-based energy supply chain network subjected to multiple sources of uncertainty. The complexity to obtain an optimal solution using traditional uncertainty management methods dramatically increases with the number of uncertain factors considered. Such a complexity produces that, if tractable, the problem is solved after a large computational effort. Therefore, in this work a data-driven decision-making framework is proposed to address this issue. Such a framework exploits machine learning techniques to efficiently approximate the optimal management decisions considering a set of uncertain parameters that continuously influence the process behavior as an input. A design of computer experiments technique is used in order to combine these parameters and produce a matrix of representative information. These data are used to optimize the deterministic multi-objective bio-based energy network problem through conventional optimization methods, leading to a detailed (but elementary) map of the optimal management decisions based on the uncertain parameters. Afterwards, the detailed data-driven relations are described/identified using an Ordinary Kriging meta-model. The result exhibits a very high accuracy of the parametric meta-models for predicting the optimal decision variables in comparison with the traditional stochastic approach. Besides, and more importantly, a dramatic reduction of the computational effort required to obtain these optimal values in response to the change of the uncertain parameters is achieved. Thus the use of the proposed data-driven decision tool promotes a time-effective optimal decision making, which represents a step forward to use data-driven strategy in large-scale/complex industrial problems.Peer ReviewedPostprint (published version

The utilisation of health research in policy-making: Concepts, examples, and methods of assessment

Chapter 1: Introduction and Background • The importance of utilising health research in policy-making, and therefore the need to understand the mechanisms involved, is increasingly recognised. Recent reports calling for more resources to improve health in developing countries, and global pressures for accountability, draw greater attention to research-informed policy-making. • For at least twenty years there has been recognition of the multiple meanings or models of research utilisation in policy-making. It has similarly been long recognised that a range of factors is involved in the interactions between health research and policy-makers. • The emerging focus on Health Research Systems (HRS) has identified additional mechanisms through which greater utilisation of research could be achieved. Assessment of the role of health research in policy-making is best undertaken as part of a wider study that also includes the utilisation of health research by industry, medical practitioners, and the public. Chapter 2: The Nature of Policy-Making, Types of Research and Utilisation Models • Policy-making broadly interpreted includes national health policies made by government ministers and officials, policies made by local health service managers, and clinical guidelines from professional bodies. In this report, however, the main focus is on public policy-making rather than that conducted by professional bodies. The utilisation of health research in policy-making should eventually lead to desired outcomes, including health gains. Research can make a contribution in at least three phases of the policy-making process: agenda setting; policy formulation; and implementation. Descriptions of these processes, however, can over-estimate the degree of rationality in policy-making. Therefore, the analysis is informed by a review of the full range of policy-making models. These include rational and incrementalist models. • Various categories of research are likely to be used differently in health policy-making. Applied research might be more readily useable by a policy system than basic research, but health policy-makers tend to relate more willingly to natural sciences than social sciences. When research is based on the priorities of potential users, and/or is research of proven quality, this increases the possibility that it will be translated into policies. There also appears to be a greater chance of research being used in clinical policies about delivering care to patients, than in national policies on the structures of the health service. • Models of research utilisation in policy-making start with a link to rational or instrumental views of policy-making, and include descriptions of how commissioned research can help to find solutions to problems. Other models relate to an incrementalist view in which policy-making involves a series of small steps over a long period; research findings might gradually cause a shift in perceptions about an issue in a process of ‘enlightenment’. Interactive models of research utilisation stress the way in which policy-makers and researchers might develop links over a long period. Research can also be used symbolically to support decisions already taken. Chapter 3: Examples from Previous Studies • A study of health policy-making in two southern African countries illustrates how policy-making processes can be analysed. It addresses agenda setting, policy formulation and implementation. The methods used included documentary analysis and key informant interviews. • Many previous studies of research utilisation can provide lessons for future assessments. Two broad approaches can be identified. Some studies start with pieces, or programmes, of research and examine their impact. Others consider policy on a particular topic and assess the role of research in the policy-making. There are advantages and drawbacks in each approach, and overlaps between them. • To facilitate comparison, studies of research utilisation are best organised around a conceptual framework. Despite that, the influence of contextual factors in different settings makes it difficult to generalise. • The two methods used most frequently, and usually together, come from the qualitative tradition: documentary analysis and in-depth interviews. Questionnaires, bibliometric analysis, insider knowledge and historical approaches have all been applied. A few recent studies have attempted to score or scale the level of utilisation. • The examples suggest there is a greater level of utilisation and final outcomes in terms of health, health equity, and social and economic gain than is often assumed, whilst still showing much underutilisation. There is considerable variation in the degree of utilisation, both within and between studies. Chapter 4: Key Issues in the Analysis of Research Utilisation in Policy-Making • Increasing attention is focusing on the concept of interfaces between researchers and the users of research. This incorporates the idea that there are likely to be different values and interests between the two communities. • In relation to utilisation, the prioritisation debate revolves around two key aspects: whether priorities are being set that will produce research that policy-makers and others will want to use, and whether priorities are being set that will engage the interests and commitment of the research community. • Interactions across the interface between policy-makers and researchers are important in transferring research to policy-makers. This fits especially well with the interactive model of utilisation. Actions by individual researchers can be useful in generating interaction, but it is desirable to consider the role of the HRS in encouraging or facilitating interactions, networks and mechanisms at a system-wide level. The HRS could provide funding and organisational support for various items including: long-term research centres; research brokerage/translator mechanisms; the creation of official committees of policy-makers and researchers; and mechanisms for review and synthesis of research findings. • There is increased recognition of the significance of policy-makers in their role as the receptors of research. In relation to the perspective of policy-makers there is a spectrum of key questions. These range from whether relevant research is available and effectively being brought to their attention, to whether they are able to absorb it and willing to use it. The HRS has a responsibility, especially in the early parts of the spectrum, but the wider health system also has a responsibility to create appropriate institutional mechanisms and ensure there are staff willing and able to incorporate relevant research. • More attention should be given to the role of incentives, both for researchers to produce utilisable research, and for policy-makers, at the system or individual level, to use it. The assessment of utilisation becomes a key issue if rewards are to focus on relevance as well as research excellence. • An appropriate model for assessing research utilisation in policy-making combines analysis of two issues: the role of receptors and the importance of actions at the interfaces. An emphasis on the role of the receptor is necessary because ultimately it is up to the policy-maker to make the decisions. Any assessment of the success of the HRS in relation to utilisation must accept that the wider political context is beyond the control of the HRS, but consider the activities of the HRS, within its given context, to enhance the utilisation of research by increasing the permeability of the interfaces. Chapter 5: Assessment of Research Utilisation in Health Policy-Making • The reasons for assessing the utilisation of research in policy-making include: advocacy, accountability, and increased understanding. For the World Health Organization there could be a role in conducting such assessments with the aim of providing evidence of the effective use of research resources. This could support advocacy for greater resources to be made available for health research. It is important that the purposes of any assessment are taken into account in planning the methods to be used. • Previous studies demonstrated the difficulties of making generalisations about specific factors associated with high levels of utilisation. To address this in any cross-national WHO initiative involving a series of studies in a range of countries, it would be desirable to structure all the studies around a conceptual framework (such as the interfaces and receptor framework considered here) and base the studies in each country on common themes. These could include policies for the adoption of multi-drug therapy for treating leprosy, and for the equitable access to health services. • Analysis of documents and semi-structured interviews would be appropriate methods in each study assessing the role of research in policy-making on a specific policy theme. Questionnaires could also have a role. These approaches would provide triangulation of methods and data-sources and should also provide material to help identify the relative importance, in relation to the level of utilisation recorded, of the HRS mechanisms described in the previous analysis. The types and sources of research used, and reasons for their use, should also be recorded and attempts made to correlate them with the previous priority setting approaches. It is expected that each study will produce its own narrative or story of what caused utilisation in the particular context, but the data gathered could also be applied to descriptive scales of the level research utilisation. The four scales could cover the consistency of policy with research findings, and the degree of influence of research on agenda setting, policy formulation, and implementation. • The findings from the assessments in each participating country should be collated. For each policy theme or topic the analysis would compare two sets of data: the scales for level of research utilisation in each country, and the contextualised lists of the HRS activities and other mechanisms and networks thought to be important. Although the account here has focused on research impact on policy-making, the evaluations would be stronger as part of a wider analysis covering research utilisation and interactions with practitioners, industry and the public. • Given appropriate and targeted topic and country selection, this approach is likely to meet the purpose of using structured methods to provide examples of effective research utilisation. The approach should contribute towards enhanced understanding of the issues and could provide the basis of an assessment tool which, if used widely in countries, could lead to greater utilisation of health research.Research Policy and Co-operation (RPC) Department of the World Health Organization, Geneva; UK Department of Health’s Policy Research Programme; Alliance for Health Policy and Systems Research from the governments of Norway and Sweden; World Bank and International Development Research Council of Canad

A Methodological Approach for Measuring the Impact of HTA

There is a lack of evidence concerning the link between HTA and outcomes in terms of health improvements. This work proposes a framework for assessing the impact of HTA. This impact assessment is a necessary step in then better understanding the value for money of HTA bodies. We emphasis that this is still a work in progress. iDSI has developed a theory of change-based framework in order to evaluate the impact the iDSI has on institutional strengthening – leading to ‘better decisions’ for ‘better health’. This framework recognises that there is a complex translation process between better decisions and better health dependent on many assumptions about local factors and systems, including linkage between decisions and budgets, delivery, implementation, and data accuracy. Work has been undertaken over the last 6 months developing a methodological approach for measuring the impact of health technology assessment (HTA). Two case studies are used to illustrate the approach. At the core of impact assessment is a requirement to link causes and effects, to explain ‘how’ and ‘why’ and to identify – and thus improve or adapt – mechanisms leading to impact. Policy makers also want to know ‘to what extent’ or ‘the magnitude of impact’. The framework developed adopts an economic approach nested in theory of change as a means of both quantifying the magnitude of impact (utilising economic models) as well as explaining why and how impact happens (drawing on theory based approaches) in order to reinforce learning as to how to improve our response and optimise the use of HTA to have the greatest impact in a given context. This should also enable us to capture and explain wider impact – perhaps more intangible aspects which cannot be easily quantified. This may also possibly increase policy-makers’ ‘buy-in’

Exploiting Qualitative Information for Decision Support in Scenario Analysis

The development of scenario analysis (SA) to assist decision makers and stakeholders has been growing over the last few years through mainly exploiting qualitative information provided by experts. In this study, we present SA based on the use of qualitative data for strategy planning. We discuss the potential of SA as a decision-support tool, and provide a structured approach for the interpretation of SA data, and an empirical validation of expert evaluations that can help to measure the consistency of the analysis. An application to a specific case study is provided, with reference to the European organic farming business

A methodology for measuring the sustainability of car transport systems

Measuring the sustainability of car fleets, an important task in developing transport policy, can be accomplished with an appropriate set of indicators. We applied the Process Analysis Method of sustainability assessment to generate an indicator set in a systematic and transparent way, that is consistent with a declared definition of a sustainable transport system. Our method identifies stakeholder groups, the full range of impacts across the environmental, economic and human/social domains of sustainability, and those who generate and receive those impacts. Car users are shown by the analysis to have dual roles, both as individual makers of decisions and as beneficiaries/sufferers of the impacts resulting from communal choice. Thus car users, through their experience of service quality, are a potential force for system change. Our method addresses many of the well-known flaws in measuring transport sustainability. The indicator set created is independent of national characteristics and will be useful to transport policy practitioners and sustainable mobility researchers globally. © 2013 Elsevier Ltd

The utilisation of health research in policy-making: Concepts, examples and methods of assessment

The importance of health research utilisation in policy-making, and of understanding the mechanisms involved, is increasingly recognised. Recent reports calling for more resources to improve health in developing countries, and global pressures for accountability, draw greater attention to research-informed policy-making. Key utilisation issues have been described for at least twenty years, but the growing focus on health research systems creates additional dimensions. The utilisation of health research in policy-making should contribute to policies that may eventually lead to desired outcomes, including health gains. In this article, exploration of these issues is combined with a review of various forms of policy-making. When this is linked to analysis of different types of health research, it assists in building a comprehensive account of the diverse meanings of research utilisation. Previous studies report methods and conceptual frameworks that have been applied, if with varying degrees of success, to record utilisation in policy-making. These studies reveal various examples of research impact within a general picture of underutilisation. Factors potentially enhancing utilisation can be identified by exploration of: priority setting; activities of the health research system at the interface between research and policy-making; and the role of the recipients, or 'receptors', of health research. An interfaces and receptors model provides a framework for analysis. Recommendations about possible methods for assessing health research utilisation follow identification of the purposes of such assessments. Our conclusion is that research utilisation can be better understood, and enhanced, by developing assessment methods informed by conceptual analysis and review of previous studies

Towards guidelines for building a business case and gathering evidence of software reference architectures in industry

Background: Software reference architectures are becoming widely adopted by organizations that need to support the design and maintenance of software applications of a shared domain. For organizations that plan to adopt this architecture-centric approach, it becomes fundamental to know the return on investment and to understand how software reference architectures are designed, maintained, and used. Unfortunately, there is little evidence-based support to help organizations with these challenges. Methods: We have conducted action research in an industry-academia collaboration between the GESSI research group and everis, a multinational IT consulting firm based in Spain. Results: The results from such collaboration are being packaged in order to create guidelines that could be used in similar contexts as the one of everis. The main result of this paper is the construction of empirically-grounded guidelines that support organizations to decide on the adoption of software reference architectures and to gather evidence to improve RA-related practices. Conclusions: The created guidelines could be used by other organizations outside of our industry-academia collaboration. With this goal in mind, we describe the guidelines in detail for their use.Peer ReviewedPostprint (published version

Which economic model for a water-efficient Europe? Report of a CEPS Task Force. CEPS Task Force Report, 27 November 2012

This CEPS Task Force Report focuses on how to improve water efficiency in Europe, notably in public supply, households, agriculture, energy and manufacturing as well as across sectors. It presents a number of recommendations on how to make better use of economic policy instruments to sustainably manage the EU’s water resources. Published in the run-up to the European Commission’s “Blueprint to Safeguard Europe’s Waters”, the report contributes to the policy deliberations in two ways. First, by assessing the viability of economic policy instruments, it addresses a major shortcoming that has so far prevented the 2000 EU Water Framework Directive (WFD) from becoming fully effective in practice: the lack of appropriate, coherent and effective instruments in (some) member states. Second, as the Task Force report is the result of an interactive process involving a variety of stakeholders, it is able to point to the key differences in interpreting and applying WFD principles that have led to a lack of policy coherence across the EU and to offer some pragmatic advice on moving forward