Gaillard, Jacques

Horizon / Pleins textes

In Research Progress Measurement and Management Decision Making, pp. 28-36 
proceedings of a Workshop Organized by 
the Board on Science and Technology for International Policy (BOSTlD) 
for USAID 
December 1991 
THE USE OF EVALUATION AND INDICATORS FOR RESEARCH MANAGE"'I' 
Some Examples and the European Communauty 
ORSTOM 
France 
This presentation deals with the evaluation practices in France and in the European Community, 
focusing on specialized tropical research institutions and funding programs aimed at structuring and 
strengthening the European scientific communities involved in tropical science. 
While some 10 years ago most scientists were opposed, or at least reluctant, to evaluation other 
than "peer reviews" of scientists, evaluations of scientists, research programs and institutions are 
becoming common practices today. Thus, evaluations are receiving a wider acceptance among the 
scientific community not only to assess scientific achievements, but also as a tool to design and implement 
better science and technology (S&T) policies. At the same time the need for 'quantifiable criteria' has 
been emphasized leading to the concept of output and performance indicators. This short presentation 
deals with evaluation practices in France and the European Community (EC) focusing on specialized 
tropical research institutions and funding programs aiming at structuring and strengthening the European 
scientific communities specializing in tropical scientific research. It also proposes general 
recommendations derived from a number of evaluations with which I have been associated. But fist, 
I think it is necessary to briefly present eh specificity of the French system and the EC's programs as $ 
n 
.. < - -  
Text of presentation revised by Dr. Gaillard. 
compared to the U.S. system. 
France 
French Come ration in $&T for DeveloDmenf 
As you may know, in France there are no research donor institutions such as A.I.D., the 
Canadian International Development Research Centre (IDRC,) etc., but research institutions specializing 
in tropical research. The French system so far has been (and is still) more researcher than program 
oriented. The main course of the system is set by three Ministries, which have traditionally relied on a 
number of specialized institutions, of which the Institut Français de Recherche Scientifique pour le 
Developpement en Cooperation (ORSTOM) is one. There are two others: the Centre de cooperation 
internationale en recherche agronomique pour le developpement (CIRAD), which is entirely devoted to 
agricultural research; and the Institut Pasteur Outre-Mer (IPOM), which is the overseas Pasteur Institute 
working on tropical diseases. . 
Despite a clear reorientation towards Latin American countries (mainly Mexico, bolivia, Ecuador, 
and Brazil) and to a much lesser extent Asian countries (mainly Thailand, Indonesia, and soon most 
probably soon Vietnam) during the 19803, priority is still given to French-speaking African countries. 
As for research areas, agricultural research is receiving the bulk of the support, followed by medical 
research. Reforms implemented between 1982 and 1985 have slightly modified the structure and the 
direction of the system, allowing among other things, a greater mobilization of the entire French scientific 
community, better coordination as well as a broadening of the research areas to be supported. These 
changes have been accompanied by an increased participation in the international research system and a 
29 
c 
t 
renewal of methods of cooperations, thus, partnership research is gradually replacing scientific and 
technological assistance. 
France is also working at strengthening European alliances. The idea of a European Association 
is envisaged to establish between member countries joint channels of scientific cooperation with 
developing countries. The establishment of a European Foundation for supporting research activities in 
Africa has also been proposed by the French government to the European community. Collaboration 
between European institutions have also been enhanced by the Science Technology and Development 
(STD) program of the EC. 
Evaluation Activities and Research Management in France 
Given the specificity of the above described system evaluation activities have tended to focus 
more on the evaluation of the researchers themselves, through mainly in-house peer reviews, than the 
evaluation of the research programs and institutions. Evaluation of departments and research units have 
however become more and more a common practice within CIRAD and ORSTOM during the 1980’s. 
At the national level, a committee, le Comite national d’kvaluation de la recherche (CNER), was 
established in 1989 to evaluate the implementation and the results of the national S&T policy. CNER 
is responsible for the evaluation of research programs, institutions and procedures. It reports directly to 
the President of the Republic through an annual public report. Based on the outcome of the evaluations, 
CNER submits suggestions and recommendations to the pertinent Ministries. Seven evaluations have 
been carried out so far, and ORSTOM is among the next institutions to be evaluated. 
30 
The S~ec ificitv of Researc h Activities in Cooue ration for Development 
Even AI iesearch in partnership is more and more acknowledged as a key variable, one ca clearly 
distinguish three different objectives which are at the same time conflicting and complementary: research; 
cooperation (partnership); and development. 
Experience has shown that the evaluation of a researcher’s2 (or research program’s) achievements 
depending on which one of the three objectives is concerned cannot be evaluated the same way. The 
evaluation should differentiate between the achievements of the research results, the extent to which the 
work has been carried out in partnership with scientists from developing countries, and the contribution 
to development. This requires a different set of indicators and a different kind of expertise. In the case 
of research one needs to assess, for example, the validity, the recognition, and the degree of innovation 
of the research results. This can only be done by individuals who are themselves competent researchers 
in the fields concerned. The degree of cooperation and partnership can be, for example, evaluated by 
measuring the degree of involvement of the various partners in the project design, the balanced sharing 
of activities among partners, the exchange and training of scientists among the partners, and in general 
the variety of mechanisms set up to exchange experiences. The contribution to development is even more 
complex. It should involve not only indicators of adoption of research results, but also the overall impact 
of these results on production and society as a whole. This also requires another type of expertise which 
most often is not available in a research institution. 
. 
I 
%e criteria for evaluating researchers at ORSTOM are being reconsidered. In addition to 
evaluating the scientific achievements, the peer review committees are given new evaluation criteria, 
including implementation of research results, diffusion of knowledge in general, research in 
partnership with scientists from developing countries, research supervision and management, and 
research training. 
31 
The outcome of the evaluation will finally depend on the relative weight one decides to give to 
each of these objectives and to each of the corresponding criteria, as well as on the negotiation taking 
place among the different actors involved in the evaluation process. For strategic planning exercises of 
institutions such as ORSTOM and CIRAD, one often needs to combine these three types of evaluations. 
The European Community 
In Europe the funding at the EC level represents approximately 3 percent of the total European 
public R&D funding. Although this level of funding could be considered relatively low, it has important 
marginal effects and it influences both the definition of priorities and the scientific collaborations in 
Europe. Since the early 1980's, the EC has conducted more than 40 evaluations. In addition to scientific 
and technical achievements, the objective of these evaluations has been to assess the "added value" due 
to the implementation of these activities at the European level. 
This requires an appropriate methodology to measure intra-European cooperations and networking 
effects. This methodology is different from those normally used in the evaluation of scientific outputs. 
Specific indicators have been developed and become essential tools of science management of the EC's 
programs. This includes indicators measuring the improvement of the relationships between "pairs of 
laboratories" or between a whole set of laboratories, the transnationality of papers and their scientific 
impact as compared with other papers published in the same journals and the constitution of a "European 
club of contractors" as shown in the Evaluation Report of the Non Nuclear Energy Programme (Callon, 
M., et al., 1989). Another important outcome of these evaluations is that objectives and goals of the EC 
programs are linked to evaluation criteria. Thus, the importance of planning evaluations already exists 
during project design. Hence, evaluation plans are becoming an integral part of effective planning. 
32 
I discuss below in greater details the only research funding program within the EC aiming, among 
other objectives, at networking laboratories in Europe and in developing countries. 
The Promam o n Science. Technolom. and Develoument of the Eurouean Co "unity 
The Program on Science, Technology, and Development (STD) of the European Community is specific 
as it intends to promote collaborative research between the laboratories of member countries of the 
European Communities and those of developing countries in the field of tropical medicine and agriculture. 
The program is entering its third phase (1991-1994), the second phase being now evaluated. A typical 
project involves at least two research teams in Europe and at least one developing country team with an 
average budget of US$300,000 per project. 
The objectives and expected results are, again, manyfold. They include development of 
knowledge and contribution to development, but also other results as well, especially: 
the identification of competent laboratories in Europe and in developing countries having an 
interest to collaborate (even if they were not at first dedicated to tropical studies); 
- the strengthening of collaboration among them; 
* the contribution to a gradual structuring of scientific communities in the developing countries; 
* 
environment and health. 
the structuring of a European scientific capacity in the field of tropical agriculture, 
33 
‘i 
&. 
The multidimensionality of objectives is reflected in the individual assessment form being used 
by the experts evaluating the proposals submitted to the program. Thus, criteria have been devised to 
evaluate the degree of innovation, the feasibility, the appropriateness of cooperation, the strengthening 
of research capacities in the developing country, the training and exchange of scientists, the links with 
work supported by EC programs or member-states, the relevance to development as well as some 
environmental implications. A relative rate is given to each of these criteria which leads to a final 
ranking in three categories: excellent; good; or rejected. 
The evaluation of the first phase has, among other things, demonstrated a clear bias towards 
scientific quality warranted by the competent laboratories selected in Europe as opposed to capacity 
strengthening. Very few approved projects had been designed by scientists in developing countries, and 
I less than 1/3 of the projects have been administered by research teams from the developing countries. 
~ 
The research training component also proved to be very weak. Representatives from the developing 
I countries also requested that more for their experts be associated with the selection process. These 
~ 
criticisms and recommendations have been taken into consideration in the implementation of the second 
~ phase of the program. 
I 
One of the objectives of the evaluation of the second phase of the program is to propose a system 
for strategic management which would better take into account the variety of objectives of the program. 
It is, however, too early to report any result. 
, -. 
As a way of conclusion, I would like to raise a number of general issues and recommendations 
derived from the above examples as well as a number of evaluations with which I have been associated. 
34 
General Issues and Recommendations 
It is a necessary, but not a sufficient, condition that evaluation begin at project design by 
providing a clear definition of objectives and expected results, as well as adapted indicators. One should 
also recognize that different objectives require different types of evaluations and different types of 
expertise. One should also balance the power of different actors involved in the selection and evaluation 
process in order to better reach these objectives. 
Given the multidimensionality of the objectives, which is most often the case of donor 
organizations, there is not a single evaluation methodology which can provide convincing results. 
Several, sometimes overlapping, methodologies are necessary. They may include peer reviews, enquiries 
addressed to grant recipients, institutional studies, contextual studies, just to give a few examples. This 
is the convergence or the divergence of the results obtained with the different methodologies that can 
bring about more convincing conclusions. 
Most often economic or financial indicators are inadequate for assessing scientific or technical 
projects particularly in developing countries. Strict adherence to a financial calendar can endanger the 
logic and success of the scientific calendar. Contextual factors and in particular socioeconomic issues 
are often most important to explain a project’s failure or success. 
To evaluate the work of grant recipients it is most often advisable to adopt a pro-active approach. 
This will diminish the risk of having them taking a negative view and perceiving the evaluation solely 
as a control of the use of funds and ultimately as a sanction. 
35 
The results of the evaluation should be public and transparent. They should, as much as possible, 
be made available to the people who contributed and/or participated in the evaluation (e.g., through mail 
questionnaires or interviews). Scientists, in particular, often feel that the money spent in evaluations is 
lost and stolen from research. Thus, the importance of proving the usefulness of evaluation and 
indicators as a tool for improved science management. 
Evaluations are expensive. The cost of conducting evaluations and even more so the cost of 
implementing the results of the evaluations are most of the time badly underestimated. Who has not 
participated in evaluations whose results have not been implemented just because no provision had been 
made for it or because it was not made clear from the beginning who should be using and implementing 
the evaluation results? I have. 
36 


Research progress measurement and management decision making

Abstract

Similar works

Full text

Available Versions

Horizon / Pleins textes