Using a bibliometric approach to support research policy decisions: The case of the Flemish BOF-key.

In this paper, we describe the development of a methodology and an instrument to support a major research funding allocation decision by the Flemish government. Over the last decade, and in parallel with the decentralization and the devolution of the Belgian federal policy authority towards the various regions and communities in the country, science and technology policy have become a major component of regional policy making. In the Flemish region, there has been an increasing focus on basing the funding allocation decisions that originate from this policy decentralization on 'objective, quantifiable and repeatable' decision parameters. One of the data sources and indicator bases that have received ample attention in this evolution is the use of bibliometric data and indicators. This has now led to the creation of a dedicated research and policy support staff, called 'Steunpunt O&O Statistieken,' and the first time application of bibliometric data and methods to support a major inter-university funding allocation decision. In this paper, we analyze this evolution. We show how bibliometric data have for the first time been used to allocate 93 million Euro of public research money between 6 Flemish universities for the fiscal year 2003, based on Web-of-Science SCI data provided to 'Steunpunt O&O Statistieken' via a license agreement with Thomson-ISI. We also discuss the limitations of the current approach that was based on inter-university publication and citation counts. We provide insights into future adaptations that might make it more representative of the total research activity at the universities involved (e.g. by including data for the humanities) and of its visibility (e.g. by including impact measures). Finally, based on our current experience and interactions with the universities involved, we speculate on the future of the specific bibliometric approach that has now been adopted. More specifically, we hypothesize that the allocation method now developed and under further improvement will become more criticized if it turns out that it (1) also starts influencing intra-university research allocation decisions and, as a consequence (2) introduces adverse publication and citation behaviors at the universities involved.Policy; Decisions; Decision;

Models of financing research: public funding mechanisms for universities in Flanders

This paper gives an overview of the models of financing research at universities in Flanders, Belgium. The Flemish government installed parallel mechanisms to distribute financial means for scientific research at the universities: research is supported via the allocation of block grants to the universities based on specific interuniversity allocation keys on the one hand and via project-based funding allocated on competitive basis by public funding agencies on the other hand. The composition of the allocation key of both the Special Research Fund and the Industrial Research Fund and the impact of the research performance-based parameters of these allocation keys on the research policy of universities and on the peer-reviewed assessment of the quality of research proposals submitted to the Fund for Scientific Research – Flanders, one of the Flemish public funding agencies, are discussed

A review of the literature on citation impact indicators

Citation impact indicators nowadays play an important role in research evaluation, and consequently these indicators have received a lot of attention in the bibliometric and scientometric literature. This paper provides an in-depth review of the literature on citation impact indicators. First, an overview is given of the literature on bibliographic databases that can be used to calculate citation impact indicators (Web of Science, Scopus, and Google Scholar). Next, selected topics in the literature on citation impact indicators are reviewed in detail. The first topic is the selection of publications and citations to be included in the calculation of citation impact indicators. The second topic is the normalization of citation impact indicators, in particular normalization for field differences. Counting methods for dealing with co-authored publications are the third topic, and citation impact indicators for journals are the last topic. The paper concludes by offering some recommendations for future research

Patent data as a tool to monitor S & T portfolio's.

This article deals with the use of patent data to monitor science and technology (S&T) portfolios. S&T portfolios have become central tools to examine and to monitor the vitality of both institutions and regions in the innovation game that underpins their economic growth and development. Those portfolios have to be monitored not only at the intra-organizational level, but also at the inter-organizational level and at the levels of specific systems of innovation. Therefore, the development of appropriate, easy-to-use and transparent, benchmark indicators to assess the strengths and weaknesses of organizational S&T portfolios is tantamount. In this paper, we report the construction of such a benchmark indicator and we assess its usefulness by applying it to the European Patent Database.Data; Science; Regions; Systems;

Reviewing, indicating, and counting books for modern research evaluation systems

In this chapter, we focus on the specialists who have helped to improve the conditions for book assessments in research evaluation exercises, with empirically based data and insights supporting their greater integration. Our review highlights the research carried out by four types of expert communities, referred to as the monitors, the subject classifiers, the indexers and the indicator constructionists. Many challenges lie ahead for scholars affiliated with these communities, particularly the latter three. By acknowledging their unique, yet interrelated roles, we show where the greatest potential is for both quantitative and qualitative indicator advancements in book-inclusive evaluation systems.Comment: Forthcoming in Glanzel, W., Moed, H.F., Schmoch U., Thelwall, M. (2018). Springer Handbook of Science and Technology Indicators. Springer Some corrections made in subsection 'Publisher prestige or quality

Research Performance Based Funding Systems: a Comparative Assessment

Performance based funding is used by most EU Member States to increase the performance of their public research system. This report analyses the different nature of systems in EU Member States, selected associated and third countries. It aims to inform Member States which are in a process of mutual learning to improve the design of their allocation systems. • Research Performance based funding systems provide incentives to increase scientific performance and concentrates resources in well performing organisations • The nature of systems in place differs widely • The specific features of RPBF assessment designs can generate unintended consequences • The assessment suggests RPBF as a potential avenue for several Member States • The choice for specific designs, taking into accounts costs and potential benefits, should take into account the national contextJRC.J.6-Innovation Systems Analysi