ACCESS: An Inception Report

Imagine a world in which all groups of citizens coming together to realize some public benefit measure and communicate the character and consequences of their work. Imagine further that all those groups have adopted a common reporting system that enables their individual reports to be compared, thus creating powerful descriptions of the relative and collective performance of citizen association for public benefit. Imagine, too, that this common measuring and reporting carries across to all forms of public-private partnership and corporate social responsibility. This is the world envisioned by ACCESS.For the past 18 months a growing number of concerned actors have been meeting, studying, and testing opinion around one of the great structural weaknesses in the world's institutional infrastructure -- inefficient and weak social investment markets. This inception report sets out the results of this enquiry in the form of a proposal to establish a reporting standard for nonprofit organizations seeking to produce social, environmental and, increasingly, financial returns. The ACCESS Reporting standard is one important contribution to redressing a major global system weakness, but it is certainly not the only one. Nor is it one that can operate in isolation from other initiatives. Accordingly, the ACCESS proposed plan of work involves convening a global dialogue on NGO transparency, accountability and performance with the objective of promoting ACCESS and other practical solutions to the challenges of social investment and civil society accountability.This report sets out the background and rationale for these proposals. You will meet the ACCESS sponsors and pilot project partners. Parts of the report are descriptive and analytical but other parts are necessarily theoretical and technical in nature. We make no apology for this. Part of the reason that in 2003 the world does not yet have a reporting standard for social actors is that the theory and technique have not been mastered. For those with a strong orientation toward strategy and action, however, these aspects are presented as well

The utilisation of health research in policy-making: Concepts, examples and methods of assessment

The importance of health research utilisation in policy-making, and of understanding the mechanisms involved, is increasingly recognised. Recent reports calling for more resources to improve health in developing countries, and global pressures for accountability, draw greater attention to research-informed policy-making. Key utilisation issues have been described for at least twenty years, but the growing focus on health research systems creates additional dimensions. The utilisation of health research in policy-making should contribute to policies that may eventually lead to desired outcomes, including health gains. In this article, exploration of these issues is combined with a review of various forms of policy-making. When this is linked to analysis of different types of health research, it assists in building a comprehensive account of the diverse meanings of research utilisation. Previous studies report methods and conceptual frameworks that have been applied, if with varying degrees of success, to record utilisation in policy-making. These studies reveal various examples of research impact within a general picture of underutilisation. Factors potentially enhancing utilisation can be identified by exploration of: priority setting; activities of the health research system at the interface between research and policy-making; and the role of the recipients, or 'receptors', of health research. An interfaces and receptors model provides a framework for analysis. Recommendations about possible methods for assessing health research utilisation follow identification of the purposes of such assessments. Our conclusion is that research utilisation can be better understood, and enhanced, by developing assessment methods informed by conceptual analysis and review of previous studies

The utilisation of health research in policy-making: Concepts, examples, and methods of assessment

Chapter 1: Introduction and Background • The importance of utilising health research in policy-making, and therefore the need to understand the mechanisms involved, is increasingly recognised. Recent reports calling for more resources to improve health in developing countries, and global pressures for accountability, draw greater attention to research-informed policy-making. • For at least twenty years there has been recognition of the multiple meanings or models of research utilisation in policy-making. It has similarly been long recognised that a range of factors is involved in the interactions between health research and policy-makers. • The emerging focus on Health Research Systems (HRS) has identified additional mechanisms through which greater utilisation of research could be achieved. Assessment of the role of health research in policy-making is best undertaken as part of a wider study that also includes the utilisation of health research by industry, medical practitioners, and the public. Chapter 2: The Nature of Policy-Making, Types of Research and Utilisation Models • Policy-making broadly interpreted includes national health policies made by government ministers and officials, policies made by local health service managers, and clinical guidelines from professional bodies. In this report, however, the main focus is on public policy-making rather than that conducted by professional bodies. The utilisation of health research in policy-making should eventually lead to desired outcomes, including health gains. Research can make a contribution in at least three phases of the policy-making process: agenda setting; policy formulation; and implementation. Descriptions of these processes, however, can over-estimate the degree of rationality in policy-making. Therefore, the analysis is informed by a review of the full range of policy-making models. These include rational and incrementalist models. • Various categories of research are likely to be used differently in health policy-making. Applied research might be more readily useable by a policy system than basic research, but health policy-makers tend to relate more willingly to natural sciences than social sciences. When research is based on the priorities of potential users, and/or is research of proven quality, this increases the possibility that it will be translated into policies. There also appears to be a greater chance of research being used in clinical policies about delivering care to patients, than in national policies on the structures of the health service. • Models of research utilisation in policy-making start with a link to rational or instrumental views of policy-making, and include descriptions of how commissioned research can help to find solutions to problems. Other models relate to an incrementalist view in which policy-making involves a series of small steps over a long period; research findings might gradually cause a shift in perceptions about an issue in a process of ‘enlightenment’. Interactive models of research utilisation stress the way in which policy-makers and researchers might develop links over a long period. Research can also be used symbolically to support decisions already taken. Chapter 3: Examples from Previous Studies • A study of health policy-making in two southern African countries illustrates how policy-making processes can be analysed. It addresses agenda setting, policy formulation and implementation. The methods used included documentary analysis and key informant interviews. • Many previous studies of research utilisation can provide lessons for future assessments. Two broad approaches can be identified. Some studies start with pieces, or programmes, of research and examine their impact. Others consider policy on a particular topic and assess the role of research in the policy-making. There are advantages and drawbacks in each approach, and overlaps between them. • To facilitate comparison, studies of research utilisation are best organised around a conceptual framework. Despite that, the influence of contextual factors in different settings makes it difficult to generalise. • The two methods used most frequently, and usually together, come from the qualitative tradition: documentary analysis and in-depth interviews. Questionnaires, bibliometric analysis, insider knowledge and historical approaches have all been applied. A few recent studies have attempted to score or scale the level of utilisation. • The examples suggest there is a greater level of utilisation and final outcomes in terms of health, health equity, and social and economic gain than is often assumed, whilst still showing much underutilisation. There is considerable variation in the degree of utilisation, both within and between studies. Chapter 4: Key Issues in the Analysis of Research Utilisation in Policy-Making • Increasing attention is focusing on the concept of interfaces between researchers and the users of research. This incorporates the idea that there are likely to be different values and interests between the two communities. • In relation to utilisation, the prioritisation debate revolves around two key aspects: whether priorities are being set that will produce research that policy-makers and others will want to use, and whether priorities are being set that will engage the interests and commitment of the research community. • Interactions across the interface between policy-makers and researchers are important in transferring research to policy-makers. This fits especially well with the interactive model of utilisation. Actions by individual researchers can be useful in generating interaction, but it is desirable to consider the role of the HRS in encouraging or facilitating interactions, networks and mechanisms at a system-wide level. The HRS could provide funding and organisational support for various items including: long-term research centres; research brokerage/translator mechanisms; the creation of official committees of policy-makers and researchers; and mechanisms for review and synthesis of research findings. • There is increased recognition of the significance of policy-makers in their role as the receptors of research. In relation to the perspective of policy-makers there is a spectrum of key questions. These range from whether relevant research is available and effectively being brought to their attention, to whether they are able to absorb it and willing to use it. The HRS has a responsibility, especially in the early parts of the spectrum, but the wider health system also has a responsibility to create appropriate institutional mechanisms and ensure there are staff willing and able to incorporate relevant research. • More attention should be given to the role of incentives, both for researchers to produce utilisable research, and for policy-makers, at the system or individual level, to use it. The assessment of utilisation becomes a key issue if rewards are to focus on relevance as well as research excellence. • An appropriate model for assessing research utilisation in policy-making combines analysis of two issues: the role of receptors and the importance of actions at the interfaces. An emphasis on the role of the receptor is necessary because ultimately it is up to the policy-maker to make the decisions. Any assessment of the success of the HRS in relation to utilisation must accept that the wider political context is beyond the control of the HRS, but consider the activities of the HRS, within its given context, to enhance the utilisation of research by increasing the permeability of the interfaces. Chapter 5: Assessment of Research Utilisation in Health Policy-Making • The reasons for assessing the utilisation of research in policy-making include: advocacy, accountability, and increased understanding. For the World Health Organization there could be a role in conducting such assessments with the aim of providing evidence of the effective use of research resources. This could support advocacy for greater resources to be made available for health research. It is important that the purposes of any assessment are taken into account in planning the methods to be used. • Previous studies demonstrated the difficulties of making generalisations about specific factors associated with high levels of utilisation. To address this in any cross-national WHO initiative involving a series of studies in a range of countries, it would be desirable to structure all the studies around a conceptual framework (such as the interfaces and receptor framework considered here) and base the studies in each country on common themes. These could include policies for the adoption of multi-drug therapy for treating leprosy, and for the equitable access to health services. • Analysis of documents and semi-structured interviews would be appropriate methods in each study assessing the role of research in policy-making on a specific policy theme. Questionnaires could also have a role. These approaches would provide triangulation of methods and data-sources and should also provide material to help identify the relative importance, in relation to the level of utilisation recorded, of the HRS mechanisms described in the previous analysis. The types and sources of research used, and reasons for their use, should also be recorded and attempts made to correlate them with the previous priority setting approaches. It is expected that each study will produce its own narrative or story of what caused utilisation in the particular context, but the data gathered could also be applied to descriptive scales of the level research utilisation. The four scales could cover the consistency of policy with research findings, and the degree of influence of research on agenda setting, policy formulation, and implementation. • The findings from the assessments in each participating country should be collated. For each policy theme or topic the analysis would compare two sets of data: the scales for level of research utilisation in each country, and the contextualised lists of the HRS activities and other mechanisms and networks thought to be important. Although the account here has focused on research impact on policy-making, the evaluations would be stronger as part of a wider analysis covering research utilisation and interactions with practitioners, industry and the public. • Given appropriate and targeted topic and country selection, this approach is likely to meet the purpose of using structured methods to provide examples of effective research utilisation. The approach should contribute towards enhanced understanding of the issues and could provide the basis of an assessment tool which, if used widely in countries, could lead to greater utilisation of health research.Research Policy and Co-operation (RPC) Department of the World Health Organization, Geneva; UK Department of Health’s Policy Research Programme; Alliance for Health Policy and Systems Research from the governments of Norway and Sweden; World Bank and International Development Research Council of Canad

Measuring the Difference We're Making: Making Metrics Work for Grantmakers

Measurement and evaluation issues were the focus of a dialogue held February 25, 2010 in Washington, DC, involving 40 representatives of intermediary grantmakers and some of their funders and other partners. Participants used a case study of an organizational capacity monitoring initiative undertaken by The Global Fund for Children as the springbroad for a broader discussion of how to evaluate the work of intermediary grantmakers and their grantee partners around the world

Assessing Impact

We take this opportunity to communicate Heron's approach to assessing impact, with a particular audience in mind: our customers -- grantees, investees and applicants.We thought that it might be useful to outline how and why we support practitioners and their networks in results-based, management-oriented systems for assessing impact. Following this letter, we highlight four organizations that demonstrate impact at the local, regional and national levels

Variation of health-related quality of life assessed by caregivers and patients affected by severe childhood infections.

BACKGROUND: The agreement between self-reported and proxy measures of health status in ill children is not well established. This study aimed to quantify the variation in health-related quality of life (HRQOL) derived from young patients and their carers using different instruments. METHODS: A hospital-based cross-sectional survey was conducted between August 2010 and March 2011. Children with meningitis, bacteremia, pneumonia, acute otitis media, hearing loss, chronic lung disease, epilepsy, mild mental retardation, severe mental retardation, and mental retardation combined with epilepsy, aged between five to 14 years in seven tertiary hospitals were selected for participation in this study. The Health Utilities Index Mark 2 (HUI2), and Mark 3 (HUI3), and the EuroQoL Descriptive System (EQ-5D) and Visual Analogue Scale (EQ-VAS) were applied to both paediatric patients (self-assessment) and caregivers (proxy-assessment). RESULTS: The EQ-5D scores were lowest for acute conditions such as meningitis, bacteremia, and pneumonia, whereas the HUI3 scores were lowest for most chronic conditions such as hearing loss and severe mental retardation. Comparing patient and proxy scores (n = 74), the EQ-5D exhibited high correlation (r = 0.77) while in the HUI2 and HUI3 patient and caregiver scores were moderately correlated (r = 0.58 and 0.67 respectively). The mean difference between self and proxy-assessment using the HUI2, HUI3, EQ-5D and EQ-VAS scores were 0.03, 0.05, -0.03 and -0.02, respectively. In hearing-impaired and chronic lung patients the self-rated HRQOL differed significantly from their caregivers. CONCLUSIONS: The use of caregivers as proxies for measuring HRQOL in young patients affected by pneumococcal infection and its sequelae should be employed with caution. Given the high correlation between instruments, each of the HRQOL instruments appears acceptable apart from the EQ-VAS which exhibited low correlation with the others