A Propensity Score Approach In The Impact Evaluation On Scientific Production In Brazilian Biodiversity Research: The Biota Program

Evaluation has become a regular practice in the management of science, technology and innovation (ST&I) programs. Several methods have been developed to identify the results and impacts of programs of this kind. Most evaluations that adopt such an approach conclude that the interventions concerned, in this case ST&I programs, had a positive impact compared with the baseline, but do not control for any effects that might have improved the indicators even in the absence of intervention, such as improvements in the socio-economic context. The quasi-experimental approach therefore arises as an appropriate way to identify the real contributions of a given intervention. This paper describes and discusses the utilization of propensity score (PS) in quasi-experiments as a methodology to evaluate the impact on scientific production of research programs, presenting a case study of the BIOTA Program run by FAPESP, the State of São Paulo Research Foundation (Brazil). Fundamentals of quasi-experiments and causal inference are presented, stressing the need to control for biases due to lack of randomization, also a brief introduction to the PS estimation and weighting technique used to correct for observed bias. The application of the PS methodology is compared to the traditional multivariate analysis usually employed.101185107Abadie, A., Semiparametric difference-in-differences estimators (2005) The Review of Economic Studies, 72 (1), pp. 1-19Azoulay, P., Zivin, J.S.G., Manso, G., Incentives and creativity: Evidence from the academic life sciences (2011) RAND Journal of Economics, 42 (3), pp. 527-554Böhmer, S., von Ins, M., Different—Not just by label: Research-oriented academic careers in Germany (2009) Research Evaluation, 18 (3), pp. 177-184Carney, J., Smith, W., Parsad, A., Johnston, K., Millsap, M., (2008) Evaluation of the Faculty Early Career Development (CAREER) Program, , Abt Associates Inc, Bethesda, MD:Chen, S., Mu, R., Ravallion, M., Are there lasting impacts of aid to poor areas? Evidence for rural China. Policy Research Working Paper 4084 (2008) World Bank, , Washington: DCCrump, R.K., Hotz, J., Imbens, W., Mitinik, O.A., Dealing with limited overlap in estimation of average treatment effects (2009) Biometrika, 96 (1), pp. 187-199D’Agostino, R.B., Jr., Rubin, D.B., Estimating and using propensity scores with partially missing data (2000) Journal of the American Statistical Association, 95, pp. 749-759Ferraro, P.J., Counterfactual thinking and impact evaluation in environmental policy (2009) New Directions for Evaluation, 122 (Special Issue: Environmental Program and Policy Evaluation: Addressing Methodological Challenges)Ferraro, P.J., Pattanayak, S.K., Money for nothing? A call for empirical evaluation of biodiversity conservation investments (2006) PLoS Biology, 4 (4), p. e105Friedman, J.H., Greedy function approximation: A gradient boosting machine (2001) Annals of Statistics, 29 (5), pp. 1189-1232Frondel, M., Schmidt, C.M., Evaluating environmental programs: The perspective of modern evaluation research (2005) Ecological Economics, 55 (4), pp. 515-526Hastie, T., Tibshirani, R., Friedman, J., (2001) The elements of statistical learning, , Springer, New York:Hirano, K., Imbens, G.W., Ridder, G., Efficient estimation of average treatment effects using the estimated propensity score (2003) Econometrica, 71, pp. 1161-1189Hornbostel, S., Böhmer, S., Klingsporn, B., Neufeld, J., von Ins, M., Funding of young scientist and scientific excellence (2009) Scientometrics, 79 (1), pp. 171-190Hoshino, T., A Bayesian propensity score adjustment for latent variable modeling and MCMC algorithm (2008) Computational Statistics & Data Analysis, 52 (3), pp. 1413-1429Ihaka, R., Gentleman, R., R: A language for data analysis and graphics (1996) Journal of Computational and Graphical Statistics, 5 (3), pp. 299-314Inglesi-Lotz, R., Pouris, A., Scientometric impact assessment of a research policy instrument: The case of rating researchers on scientific outputs in South Africa (2011) Scientometrics, 88 (3), pp. 747-760Jacob, B.A., Lefgren, L., The impact of NIH postdoctoral training grants on scientific productivity (2011) Research Policy, 40, pp. 864-874Jalan, J., Ravallion, M., Estimating the benefit incidence of an antipoverty program by propensity-score matching (2003) Journal of Business and Economic Statistics, 21 (1), pp. 19-30Joly, C.A., Rodrigues, R.R., Metzger, J.P., Haddad, C.F.B., Verdade, L.M., Oliveira, M.C., Bolazani, V.S., Biodiversity conservation research, training, and policy in São Paulo (2010) Science, 328, pp. 1358-1359Khandker, S.R., Koolwal, G.B., Samad, H.A., (2009) Handbook on impact evaluation quantitative methods and practices, , World Bank, Washington, DC:Liu, X., Zhang, L., Hong, S., Global biodiversity research during 1900–2009: A bibliometric analysis (2011) Biodiversity Conservation, 20 (807), pp. 807-826Lynch, L., Gray, W., Geoghegan, J., Are farmland preservation program easement restrictions capitalized into farmland prices? What can a propensity score matching analysis tell us? (2007) Applied Economic Perspectives and Policy, 29 (3), pp. 502-509McCarey, D., Ridgeway, G., Morral, A., Propensity score estimation with boosted regression for evaluating adolescent substance abuse treatment (2004) Psychological Methods, 9 (4), pp. 403-425Mena-Chalco, J.P., Cesar, R.M., Jr., scriptLattes: An open-source knowledge extraction system from the Lattes platform (2009) Journal of the Brazilian Computer Society, 15 (4), pp. 31-39Mutz, R., Daniel, H.-D., The generalized propensity score methodology for estimating unbiased journal impact factors (2012) Scientometrics, 92 (2), pp. 377-390Nelder, J., Wedderburn, R., Generalized linear models (1972) Journal of the Royal Statistical Society. Series A (General), 135 (3), pp. 370-384Ravallion, M., Evaluating anti-poverty programs (2008) Handbook of development economics, pp. 3787-3846. , Schultz TP, Strauss J, (eds), North-Holland, Amsterdam:Ridgeway, G., The state of boosting (1999) Computing Science and Statistics, 31, pp. 172-181http://cran.r-project.org/doc/packages/gbm.pdf, Ridgeway, G. (2005). GBM 1.5 package manualRidgeway, G., Assessing the effect of race bias in post-traffic stop outcomes using propensity scores (2006) Journal of Quantitative Criminology, 22 (1), pp. 1-29Ridgeway, G., McCaffrey, D., Morral, A., (2006) twang, , http://cran.r-project.org/web/packages/twang/index.html, Toolkit for weighting and analysis of nonequivalent groups, Software for using matching methods in R:Robins, J.M., Rotnitzky, A., Comment on “inference for semiparametric models: Some questions and an answer” by P. Bickel and J. Kwon (2001) Statistica Sinica, 11, pp. 920-936Rodrigues, R.R., Bonomi, V.L.R., (2008) Diretrizes para conservação e restauração da biodiversidade no estado de São Paulo, , (eds), SMA-SP & FAPESP, São Paulo:Rosenbaum, P., Rubin, D., The central role of the propensity score in observational studies for causal effects (1983) Biometrika, 70 (1), pp. 41-55Rubin, D., Estimating causal effects of treatments in randomized and nonrandomized studies (1974) Journal of Educational Psychology, 66 (5), pp. 688-701Rubin, D., Using propensity scores to help design observational studies: Application to the tobacco litigation (2001) Health Services and Outcomes Research Methodology, 2, pp. 169-188Salles-Filho, S., Avila, F.D., Sepulveda, J., Colugnati, F.A.B., Multidimensional assessment of technology and innovation programs: The impact evaluation of INCAGRO-Peru (2010) Research Evaluation, 19 (5), pp. 361-372Salles-Filho, S., Bonacelli, M.B.M., Zackiewicz, M., Castro, P.F.D., Bin, A., (2007). Development and application of a methodology for evaluating S,T&I Programs: The decomposition method. In Workshop Internacional sobre Inovações Metodológicas na Avaliação de Impacto da pesquisa Agropecuária, 2008, Brasília, XII Seminário Latino-Iberoamericano de Gestión Tecnológica - ALTEC 2007 (2007) Buenos Aires, 1, pp. 2-6Salles-Filho, S., Castro, P. F. D., Bonacelli, M. B. M., Zeitoum, C., & Sá, F. P. (2010b). Indicators for evaluation of results and impacts of research program in conservation and sustainable use of biodiversity, the case of BIOTA/FAPESP. In International conference: Getting post 2010 biodiversity targets right, Bragança PaulistaStuart, E.A., Matching methods for causal inference: A review and a look forward (2010) Statistical Science, 25 (1), pp. 1-21White, H., A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity (1980) Econometrica, 48, pp. 817-83

Evaluation Of R&d Results And Impacts - Methodological Improvements And Challenges Based On A Case Study [avaliação De Resultados E Impactos Da Pesquisa E Desenvolvimento Avanços E Desafios Metodológicos A Partir De Estudo De Caso]

This paper discusses efforts to improve and adapt a method of evaluation of results and impacts of research programs and its application to the PROSABprogram (Basic Sanitation Research Program) sponsored by FINEP, Brazil's federal agency for financing studies and projects. The PROSAB program is executed mainly by universities, although its purpose is to meet the technological needs of Brazilian society in the area of sanitation. The study concentrates on the discussion of the importance of evaluating research programs and on the creation of a methodology that can capture the inherent aspects resulting from those programs. The study was developed along two lines: evaluation of results, which concentrated on measuring the program's inputs and outputs; and evaluation of impacts, which was based on measurements of the intensity and importance of potential or perceived transformations in Brazilian society as a result of the program. The evaluation of results focused on the people executing the program (PROSAB researchers), and the unit of reference for the analysis was the research project. The evaluation of impacts encompassed the entire set of actors related to Brazil's sanitation sector, comprising the research community, sanitation companies, public actors at local and federal levels, and the unit of analysis was the program itself.152381392BACH,L.;COHENDET,P.;LAMBERT,G.;LEDOUX,M.Measuring and Managing Spinoffs: The Case of Spinoffs Generated by ESA programs. J. S. Greenberg and H. R. Hertzfeld (Editors). Space Economics, Progress in Astronautics and Aeronautics Series, V-144, Hardback, Reston-VA, p. 171-206, 1992BACH, L., CONDE-MOLIST, N., LEDOUX, M.J., MATT, M., SCHAEFFER, V., Evaluation of the economic effects of BriteEuram. programmes on the European industry (1995) Scientometrics, 34 (3), pp. 325-349. , Budapest, vBONACELLI, M. B. M.ZACKIEWICKZ, M.BIN, A. Impactos sociais de programas tecnológicos na agricultura do Estado de São Paulo. In: ALTEC. (Org.). In: Seminário Latino- Iberoamericano de Gestión Tecnológica, 10, 2003, Cuidad de México. Anais... Cuidad de México: ALTEC, 2003CALLON, M., The dynamics of techno-economic networks (1992) Technical Change and Company Strategies, , Coombs, R, Saviotti, P, Walsh, V, Eds, London: Academic Press LimitedCOZZENS, S., Assessing federally-supported academic research in the United States (2000) Research Evaluation, 8 (1), pp. 5-10. , Beech Tree Publishing, Oxford-UK, vFURTADO, A.T., TERRA, B., PASSOS, C.A.S., PLONSKY, G.A., Indicadores de C&T para avaliar os Programas de Cooperação entre Universidade e Indústria, , Uma Análise doPADCT III. In: Simpósio de Gestão da Inovação Tecnológica, 17, 2002, São Paulo. Anais... São Paulo: NPGT-USP, 2002FURTADO, A.T., Políticas Públicas para a Inovação Tecnológica na Agricultura do Estado de São Paulo: Métodos para avaliação de impactos de pesquisa. Relatório Final de Atividades (2003) Programa Políticas Públicas/Fapesp, , Departamento de Política Científica e Tecnológica, UnicampFURTADO, A.T., Avaliação dos resultados e impactos do Prosab. Relatório Final de Atividades (2005) Fundo Verde Amarelo/ Finep, , http://www.finep.gov.br/prosab/relatorio_final.pdf, Departamento de Política Científica e Tecnológica, Unicamp, Disponível em:, Acesso em: junho 2007GEORGIOU, L., ROESSNER, D., Evaluating technology programs: Tools and methods (2000) Research Policy, 29 (4-5), pp. 657-678. , Amsterdam, v(1982) Peer commentary on peer review: A case study in scientific quality control, , HARNAD, S, ed, NY: Cambridge University PressHONG, H.D., BONDEN, M., (2003) Programme Evaluation: Theory and Practice, , England, Aldershot: Ashgate Publishing Limited(2002) Nacional de Saneamento Básico 2000, , http://www.ibge.gov.br/home/estatistica/populacao/condicaodevida/pnsb/pnsb.pdf, Rio de Janeiro: IBGE, Disponível em:, Acesso em: 15 fevereiro 2005Atlas do Saneamento, I.B.G.E., Rio de Janeiro, I.B.G.E., (2004), CD -ROMLINK, A., SCOTT, J.T., Evaluating Public Research Institutions: The US Advanced Technology Program's intramural research initiative (2005) Studies in Global Competition Series, , London: RoutledgeLUNDVALL, B. A. Introduction. In: Lundvall, B.A. National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning. London: Pinter, 1992. p. 1-19(1993) National Innovation Systems: A comparative analysis, , NELSON, R, ed, Oxford: Oxford University Press(1999) Improving Evaluation Practices, , OCDE, BestPractice Guidelines for Evaluation and Background Paper. ParisDiretório dos Grupos de Pesquisa, , http://lattes.cnpq.br/pl, Disponível em:, Acesso em: 15 Fevereiro 2005ROESSNER, D., Quantitative and qualitative methods and measures in the evaluation of research (2000) Research Evaluation, 9 (2), pp. 125-132. , Beech Tree Publishing, Oxford-UK, vVAN RAAN, A.F.J., Advancedbibliometric methods as quantitative core of peer review based evaluation and foresight exercises (1996) Scientometrics, 36 (3), pp. 397-420. , Budapest, vVELHO, L., Indicadores de C&T e seu uso em Política Científica. (1992) Sociedade e Estado, 7 (1-2), pp. 63-77. , Brasília, vWORTHEN, B. R.SANDERS, J. R.FITZPATRICK, J. L. Avaliaç ão de Programas: concepções e práticas. Trad. Dinah de Abreu Azevedo. São Paulo: Editora Gente e Edusp, 2004ZACKIEWICZ, M. Trajetórias e Desafios da Avaliação em Ciência, Tecnologia e Inovação. Campinas. Tese - (Doutorado em. Política Científica e Tecnológica), Instituto de Geociências, Unicam