Research assessment in the humanities: problems and challenges

Research assessment is going to play a new role in the governance of universities and research institutions. Evaluation of results is evolving from a simple tool for resource allocation towards policy design. In this respect "measuring" implies a different approach to quantitative aspects as well as to an estimation of qualitative criteria that are difficult to define. Bibliometrics became so popular, in spite of its limits, just offering a simple solution to complex problems. The theory behind it is not so robust but available results confirm this method as a reasonable trade off between costs and benefits. Indeed there are some fields of science where quantitative indicators are very difficult to apply due to the lack of databases and data, in few words the credibility of existing information. Humanities and social sciences (HSS) need a coherent methodology to assess research outputs but current projects are not very convincing. The possibility of creating a shared ranking of journals by the value of their contents at either institutional, national or European level is not enough as it is raising the same bias as in the hard sciences and it does not solve the problem of the various types of outputs and the different, much longer time of creation and dissemination. The web (and web 2.0) represents a revolution in the communication of research results mainly in the HSS, and also their evaluation has to take into account this change. Furthermore, the increase of open access initiatives (green and gold road) offers a large quantity of transparent, verifiable data structured according to international standards that allow comparability beyond national limits and above all is independent from commercial agents. The pilot scheme carried out at the university of Milan for the Faculty of Humanities demonstrated that it is possible to build quantitative, on average more robust indicators, that could provide a proxy of research production and productiivity even in the HSS

Open Access Scientometrics and the UK Research Assessment Exercise

Scientometric predictors of research performance need to be validated by showing that they have a high correlation with the external criterion they are trying to predict. The UK Research Assessment Exercise (RAE) -- together with the growing movement toward making the full-texts of research articles freely available on the web -- offer a unique opportunity to test and validate a wealth of old and new scientometric predictors, through multiple regression analysis: Publications, journal impact factors, citations, co-citations, citation chronometrics (age, growth, latency to peak, decay rate), hub/authority scores, h-index, prior funding, student counts, co-authorship scores, endogamy/exogamy, textual proximity, download/co-downloads and their chronometrics, etc. can all be tested and validated jointly, discipline by discipline, against their RAE panel rankings in the forthcoming parallel panel-based and metric RAE in 2008. The weights of each predictor can be calibrated to maximize the joint correlation with the rankings. Open Access Scientometrics will provide powerful new means of navigating, evaluating, predicting and analyzing the growing Open Access database, as well as powerful incentives for making it grow faster

Becoming PBRF-able: Research assessment and education in New Zealand

It seems ironic that, designed as they are to quantify, evaluate and reward the research quantum of academic institutions, departments and individuals, research assessment exercises have themselves become objects of their research and critique. As many in this volume and elsewhere attest, the impact of research assessment runs deeper than mere measurement of “what is already there”: such processes are productive, or formative (Henkel, 2005, McNay, 2003; Sikes, 2006). Of course bringing about change is intended in the sense of increasing research quantity, enhancing its quality, etc. However, there are suggestions that by changing the conditions of knowledge production, research assessment exercises may also alter the shape and direction of disciplines by diverting and channelling researchers’ intellectual attention and political engagement, influencing what they study, how they do it, and how they report and write (Beck and Yong, 2005; Bernstein, 2000)

Research assessment: A contribution to solving the publication credit allocation problem

In allocation problems, a given set of goods are assigned to agents in such a way that the social welfare is maximised, that is, the largest possible global worth is achieved. When goods are indivisible, it is possible to use money compensation to perform a fair allocation taking into account the actual contribution of all agents to the social welfare. Coalitional games provide a formal mathematical framework to model such problems, in particular the Shapley value is a solution concept widely used for assigning worths to agents in a fair way. Unfortunately, computing this value is a #P-hard problem, so that applying this good theoretical notion is often quite difficult in real-world problems

The 1996 research assessment exercise : the library and information management panel

Reports on the 1996 Research Assessment Exercise (RAE), the fourth such exercise aimed at providing funding councils of UK universities (including former polytechnics) with the necessary data to rate the quality of UK academic research for predetermined units of assessment in order to fund research selectively. Previous RAEs were conducted in 1986, 1989, and 1992 (for a report of the 1992 RAE see JOLIS 26 (3) Sep 94, 141-7 (LISA ref. 9409765)). Reports generally on the work of the Library and Information Management Panel in agreeing criteria specific to their assessment task, particularly the five principal modes of publication: research monographs; articles in scholarly periodicals; refereed conference papers; published research reports; and book chapters. Discusses the methodology used by the Panel, research submissions received and the overall results

Comment: Bibliometrics in the Context of the UK Research Assessment Exercise

Research funding and reputation in the UK have, for over two decades, been increasingly dependent on a regular peer-review of all UK departments. This is to move to a system more based on bibliometrics. Assessment exercises of this kind influence the behavior of institutions, departments and individuals, and therefore bibliometrics will have effects beyond simple measurement. [arXiv:0910.3529]Comment: Published in at http://dx.doi.org/10.1214/09-STS285A the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org