Economists have engaged for some time in developing methodologies for assessing the economic impact of agricultural research and in undertaking empirical studies to measure this impact. In recent years they have documented more than 1,800 estimates of rates of return to agricultural research. Economists have paid little attention, however, to how to evaluate the impact of social science research. A symposium conducted by the International Food Policy Research Institute in 1997 was one of the first attempts to address this knowledge gap. In November 2001 the Netherlands Ministry of Foreign Affairs and IFPRI brought together a group of researchers to follow up on the earlier symposium. Their conclusions fall into two broad categories: how to measure or value the economic impact of policy-oriented social science research and how to enhance the effectiveness of such research in policymaking environments. This brief summarizes the key elements of the work of the conference.Agricultural research Congresses ,

IFPRI

English

Research Papers in Economics

INTERNATIONAL FOOD 
POLICY RESEARCH INSTITUTE
sustainable options for ending hunger and poverty
IMPACT EVALUATION
Assessing the Impact of Policy-Oriented Social Science ResearchM
easuring the impact of social science research on policy is
complex and challenging.Workshop participants identified
many areas where attention is warranted as researchers move
ahead in this field.
Scale, attribution, and time horizon. Researchers can assess
the impact of policy-oriented social science on various levels,
examining a project, program, institution, or whole body of
knowledge.To date, most case studies have been at the project
level.As one moves up the scale, it becomes more difficult to
attribute policy responses to individual actors or pieces of
research.
One major rationale for the increased attention to impact
assessment is the need for greater accountability of research
institutions.This need for accountability can encourage
researchers to focus on attributing impacts at the project level.
But in an era of increasing emphasis on research partnerships,
attempts to attribute impacts to individual actors can be coun-
terproductive for future collaboration. Such attempts can also
require daring assumptions that are subject to considerable
doubt.
The focus on the project level also implies that short-term
impacts receive more attention than longer-run impacts, which
are a feature of strategic policy research. Studies of short-term
impact will overlook these important impacts.
Supply- versus demand-side approaches. Ideally, one would
prefer to start an impact assessment from the point at which a
major policy initiative or decision occurs. Researchers would
then work backward from this demand-side outcome toward
the supply side and assess what research played a significant
role in informing or influencing the policy change. Instead,
because of the need for attribution, most impact case studies
have started at the level of an individual project in an institu-
tion and tracked how the research outputs from the supply
side have been used.This approach may cause researchers to
lose useful information on how to improve the impact of 
policy-oriented social science research.
Importance of surprise. Research that offers surprising
results can have enhanced value. Indeed, this is the essence of
Bayesian approaches to measuring impact—a new set of
methodological approaches now being employed.
2 However,
policy-oriented social science research that reinforces current
understanding and policy settings (that is, confirmatory
research) also has obvious value, so surprise is not necessarily
a sine qua non to generating impact.Anticipatory research that
alerts policymakers to possible future scenarios and surprises
also can be extremely valuable in reducing the time lag before
welfare-enhancing policy changes are made.
Choice of indicators. There are various indicators of impact.
Change in economic welfare is an obvious one and is not nec-
essarily equivalent to the welfare of politicians. Changes in dis-
tributional outcomes are another. Generally the consensus is
that portrayal of distributional outcomes in policy-oriented
social science research is more influential in effecting policy
changes  than portrayal of deadweight economic losses from
E
conomists have engaged for some time in developing methodologies for assessing the 
economic impact of agricultural research and in undertaking empirical studies to measure
this impact.
1 In recent years they have documented more than 1,800 estimates of rates of
return to agricultural research. Economists have paid little attention, however, to how to 
evaluate the impact of social science research. A symposium conducted by the International Food
Policy Research Institute (IFPRI®) in 1997 was one of the first attempts to address this knowledge gap.
In November 2001 the Netherlands Ministry of Foreign Affairs and IFPRI brought together a group of
researchers to follow up on the earlier symposium. Their conclusions fall into two broad categories:
how to measure or value the economic impact of policy-oriented social science research and how to
enhance the effectiveness of such research in policymaking environments.
ASSESSING THE IMPACT OF POLICY-ORIENTED SOCIAL SCIENCE RESEARCH
MEASURING IMPACT
1 “Impact” here refers to the effect on food security, nutrition, poverty, the environment, and capacity strengthening in developing countries.
2 Bayesian decision theory provides a framework for placing value on research information that is used by policymakers to update their beliefs
concerning various states of the world and the impacts of their policy choices.current or alternative policies. Often research depicting nation-
al, regional, or more local consequences of policy alternatives is
more influential in changing policies than global estimates.
Bibliometric indexes offer one measure of science and knowl-
edge impacts. Other indicators are an improvement in data
quality as a result of social science research and evidence of
increasing demand for research by policymakers, supplemented
by additional resources.
It is difficult to derive appropriate indicators when
policy research has reinforced the wisdom of the
status quo or resulted in inappropriate policies
or “poisoned wells.” Bayesian approaches can-
not handle such outcomes.These approaches
are useful, however, in evaluating positive pay-
offs and well-defined policy changes by single
decisionmakers.Alternatives to the Bayesian
approaches are clearly needed.
Assessing the economic value of the time saved in
effecting policy changes as a result of policy research is a
valid measure of its impact when interviews with policymakers
offer clear evidence of this. Historical retrospective narrative is
also a valuable technique for more qualitative insights, which
are especially useful when the assessment starts with a demand-
side approach.
Case studies. Case studies raise a number of methodological
research issues. For example, should researchers make use of 
random sampling or purposive sampling (“cherry-picking”) in
selecting among projects and programs to evaluate? Each
approach has its pros and cons, and no clear consensus seems 
to have emerged.
Interviewing and elicitation techniques also remain a concern,
especially when the selection of interviewees depends largely
on the researcher who undertook the original policy research.
To be meaningful, interviewees should have some familiarity
with the research being evaluated. By definition, therefore, they
will constitute a biased sample. It is useful to differentiate
among audience types in constructing survey instruments and
samples to ensure that an adequate cross-section of respon-
dents provides feedback.
Impact assessments by independent professional peers offer the
advantage of objectivity and credibility, although the costs often
limit the number of projects and programs that can be subject-
ed to this type of evaluation. In such instances “cherry-picking”
of successful stories is more likely to be used.
Time lags. Gaps or time lags in the policy process
occur between the time when research gener-
ates information and the time when policies are
formulated, from then to when policy changes
are announced, and again from then to when
the policies are implemented.The factors that
influence these lags are good candidates for 
further research.
As mentioned, anticipatory research can be especially
valuable in reducing time lags in policy adoption. Research
that misses key policy decisionmaking events is much less useful
than that which is available as an input into them, especially if
the research results are not confirmatory in nature.When
anticipatory research is not done, the cost in terms of wrong
policy decisions can be high.
Ex ante and ex post assessments. Measuring the impact of
research requires attention to assessments both before policy
change (ex ante) and after (ex post). Researchers can use ex
ante assessments to help gauge the success of policy research
in achieving its objectives as a part of monitoring and evalua-
tion. Even though all projects in a portfolio may not undergo
formal independent ex post assessment, it is still valuable for
researchers to document impact as a way of deriving their own
lessons and hence enhancing their own effectiveness.There is
no substitute, however, for impact evaluation by independent
professional peers to ensure the credibility and accountability
of institutions.
W
orkshop participants identified a number of ways for
social scientists to increase the chances of having policy
impact and the size of that impact.
Communications. Enhancing impact requires addressing vari-
ous audiences, and a communications strategy is needed at the
outset to help identify and target these groups. Researchers
must not only present their findings in peer-reviewed publica-
tions, but also convey their messages to a largely economically
illiterate public via effective oral communication.To fill this new
role, economists will require training and new incentive systems.
Advocacy is a powerful influence, but researchers must walk a
fine line between advocating specific policies and retaining
credibility as objective analysts. Here policy champions in the
policy arena may be especially helpful.There is also a need to
identify and cultivate epistemic or knowledge communities and
coalitions around desirable policy changes.
Understanding policy processes. Understanding the process
is an important ingredient in ensuring that research outputs
and outcomes result in appropriate policy responses. Political
scientists and sociologists may be able to assist in this. It may
be possible to help create a conducive policy environment,
favorable to the use of policy advice arising from policy
research, through a well-thought-out communications strategy.
For international policy research institutions, posting
researchers in developing countries for an extended period
allows them to build up an understanding of the policy
processes and position the research to have maximum effect.
It also allows them to be in a position to respond to emergent
ENHANCING IMPACTpolicy issues and to be present when major policy decisions
are being made and hence exert appropriate influence.
Policy research capacity. To achieve sustainable impacts from
policy-oriented social science research in developing countries,
strengthening the capacity of national institutions is vital and
should be a feature of all collaboration between international
and national research agencies. Indeed policy research, capacity
strengthening, and effective communications are in effect joint
products. Choosing to work in countries where policy research
capacity is strong and data are readily available may offer
greater immediate impacts than in countries with weak capacity
and data. On equity grounds, however, there may be a stronger
case for focusing on the latter group of countries. An
implicit trade-off may have to be made in such instances.
Research outputs. The most influential policy-oriented social
science research appears to be research that presents the dis-
tributional consequences of alternative policy options—who
gains and who loses—rather than research that only reveals
the size of the efficiency gains. Of special significance are the
impacts on poverty and food security. More research is need-
ed on how best to assess and convey such information. Here
the sustainable livelihoods approach has promise as a supple-
ment to the more common measures.
Another valuable output is primary data, especially at the
household level, together with simple statistical analyses that
set the stage for later, more sophisticated research, including
modeling. Simple statistics can often surprise policymakers and
whet their appetite for more detailed research, leading to new
policy options.The result can be a more conducive environ-
ment and greater potential for impact.This primary data col-
lection may be especially important in transitional economies
with poor databases and little exposure to market and house-
hold economics.
Objectivity and quality. Policymakers seem to respond bet-
ter to research that emanates from institutions that have a
reputation for quality, credibility, and objectivity. In an environ-
ment where interest groups bring their own research to bear
on major policy questions, having independent research infor-
mation from an institution of standing can help create a con-
sensus.This “honest broker” image can help policy research
institutions greatly enhance the impact of their work.
Research priorities. Correctly anticipating the major policy
issues of the future is one of the primary ingredients in estab-
lishing priorities for policy-oriented social science research.
Although change in a long-standing policy (such as crop insur-
ance) may offer large societal benefits, this fact alone may not
justify more research. Payoffs may be higher from research on
changing components of the program (for instance, not adding
specialty crops to the crop insurance program) than from
research on the bigger question for which success is more
valuable but less likely.
Implications for the Future
A
number of lessons have emerged for donors,
governments, and researchers about how to
enhance the effectiveness of policy-oriented social 
science research.
Donors and governments should:
• encourage the development of independent, well-
managed, high-quality policy research institutions;
• improve the linkages between research and policy 
formulation; and
• invest in studying the policy processes, training, and 
promoting economic literacy.
Research institutions should:
• know what impacts donors value;
• ensure that the impacts of value to donors coincide
with those of the people and their governments;
• make ex ante and ex post impact evaluation a part of
their core business;
• create incentive and reward systems consistent with
the policy objectives of the agencies that commission
or make use of research outputs;
• undertake more multidisciplinary research on evaluat-
ing and enhancing impact, including policy processes;
• build policy epistemic communities involving all
stakeholders; and
• never compromise on quality and objectivity in the
quest for impact.
Because much remains to be learned about evaluating 
the impact of policy-oriented social science research,
the workshop participants concluded that IFPRI should
take the lead in developing a consortium to help
improve interdisciplinary methods of assessing impact.
The consortium would consist of institutions, donors,
and individuals and would work in partnership with
developing countries. Institutions could learn from each
other about best practices and in the process exploit
synergies, thereby increasing effectiveness and reducing
the costs of what is an expensive undertaking.
A more detailed report on the workshop described
here is available as Synthesis Report of Workshop on
Assessing the Impact of Policy-Oriented Social Science
Research, by Jim Ryan, Impact Assessment Discussion
Paper 15 (Washington, D.C.: IFPRI, 2002).
INTERNATIONAL FOOD POLICY 
RESEARCH INSTITUTE
2033 K Street, NW
Washington, DC 20006-1002 USA
Copyright © 2002 International Food Policy Research Institute.All rights reserved. Sections of this document may be reproduced
without the express permission of but with acknowledgment to the International Food Policy Research Institute.
Telephone: +1-202-862-5600  •  Fax: +1-202-467-4439
Email: ifpri@cgiar.org  •  Website: www.ifpri.org

Impact evaluation

http://www.ifpri.org/sites/default/files/publications/ib5.pdf

Impact evaluation

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics