Web indicators for research evaluation. Part 3: books and non standard outputs

This literature review describes web indicators for the impact of books, software, datasets, videos and other non-standard academic outputs. Although journal articles dominate academic research in the health and natural sciences, other types of outputs can make equally valuable contributions to scholarship and are more common in other fields. It is not always possible to get useful citation-based impact indicators for these due to their absence from, or incomplete coverage in, traditional citation indexes. In this context, the web is particularly valuable as a potential source of impact indicators for non-standard academic outputs. The main focus in this review is on books because of the much greater amount of relevant research for them and because they are regarded as particularly valuable in the arts and humanities and in some areas of the social sciences

Reviewing, indicating, and counting books for modern research evaluation systems

In this chapter, we focus on the specialists who have helped to improve the conditions for book assessments in research evaluation exercises, with empirically based data and insights supporting their greater integration. Our review highlights the research carried out by four types of expert communities, referred to as the monitors, the subject classifiers, the indexers and the indicator constructionists. Many challenges lie ahead for scholars affiliated with these communities, particularly the latter three. By acknowledging their unique, yet interrelated roles, we show where the greatest potential is for both quantitative and qualitative indicator advancements in book-inclusive evaluation systems.Comment: Forthcoming in Glanzel, W., Moed, H.F., Schmoch U., Thelwall, M. (2018). Springer Handbook of Science and Technology Indicators. Springer Some corrections made in subsection 'Publisher prestige or quality

Performance-based publisher ratings and the visibility/impact of books:Small fish in a big pond, or big fish in a small pond?

This study compares publisher ratings to the visibility and impact of individual books, based on a 2017 data set from three Nordic performance-based research funding systems (PRFS) (Denmark, Norway, and Finland). Although there are Journal Impact Factors ( JIFs) for journals, there is no similar indicator for book publishers. National publisher lists are used instead to account for the general “quality” of books, leading to institutional rewards. But, just as the JIF is not recommended as a proxy for the “citedness” of a paper, a publisher rating is also not recommended as a proxy for the impact of an individual book. We introduce a small fish in a big pond versus big fish in a small pond metaphor, where a “fish” is a book and “the pond” represents its publishing house. We investigate how books fit on this metaphorical fish and pond continuum, using WorldCat holdings (visibility) and Google Scholar citations (impact), and test other variables to determine their predictive value with respect to these two indicators. Our statistics show that publisher levels do not have predictive value when other variables are held constant. This has implications for PRFS and book evaluations in general, as well as ongoing developments related to a newly proposed international publisher registry.Peer reviewe

Library Catalog Analysis and Library Holdings Counts: origins, methodological issues and application to the field of Informetrics

Unrevised version to be published in "Evaluative informetrics – the art of metrics based research assessment. Festschrift in honour of Henk F. Moed" , edited by Cinzia Daraio and Wolfgang Glänzel.In 2009, Torres-Salinas & Moed proposed the use of library catalogs to analyze the impact and dissemination of academic books in different ways. Library Catalog Analysis (LCA) can be defined as the application of bibliometric techniques to a set of online library catalogs in order to describe quantitatively a scientific-scholarly field on the basis of published book titles. The aim of the present chapter is to conduct an in-depth analysis of major scientific contributions since the birth of LCA in order to determine the state of the art of this research topic. Hence, our specific objectives are: 1) to discuss the original purposes of library holdings 2) to present correlations between library holdings and altmetrics indicators and interpret their feasible meanings 3) to analyze the principal sources of information 4) to use WorldCat Identities to identify the principal authors and works in the field of Informetrics

Validità e limiti della library catalog analysis per la valutazione della ricerca nelle scienze umane e sociali

Dopo una breve analisi delle potenzialità e dei limiti dell’analisi citazionale come strumento per la valutazione della ricerca scientifica in tutti i campi, e in modo specifico nel settore delle scienze umane e sociali, nel saggio viene esaminata la proposta avanzata da alcuni autori – Adrianus J. M. Linmans, Howard D. White e Daniel Torres-Salinas insieme a Henk F. Moed – di utilizzare il rilevamento del numero delle presenze delle monografie nelle raccolte bibliotecarie come indice di qualità scientifica dei lavori degli autori sottoposti a valutazione. Rispetto alle proposte dei tre autori – la libcitation, la library holding analysis e la library catalog analysis – si mettono in evidenza alcuni elementi critici che potrebbero rendere poco attendibili i computi relativi alla diffusione delle monografie nelle raccolte bibliotecarie, riscontrabili attraverso l’uso di cataloghi online. Nello specifico si analizzano quelli legati alla composizione delle raccolte e alla presenza di copie donate e non acquistate dalle biblioteche, elementi entrambi sottovalutati dagli autori che vengono presi in esame. Infine, si pone in risalto come sia inopportuno accettare come indicatore per la valutazione della ricerca nelle scienze umane il rilievo delle presenze delle monografie nelle biblioteche, e come sia invece condivisibile l’uso proposto da Torres-Salinas e da Moed dei dati catalografici disponibili online come fonte per indagini bibliometriche sulla distribuzione delle monografie e sulla produzione editoriale.The paper addresses the potentialities and the limits of the citation analysis used as tool for the evaluation of the scientific research in all the fields and in specific way in the field of the Human and Social sciences. Above all, the paper analyzes the proposal suggested by some authors – Adrianus J. M. Linmans, Howard D. White and Daniel Torres-Salinas together with Henk F. Moed – of using the survey of the number of the monographs in the holding of libraries as index of scientific quality of the works of the authors subjected to evaluation. Regarding the methodologies proposed by three authors – the Libcitation, the Library holding analysis and the Library catalogue analysis – there are put in evidence some critical elements that might make not reliable the reckonings relative to the diffusion of the monographs in the holdings of libraries, verifiable using on-line catalogues. In particular, are analyzed those tied to the composition of the library holdings and to the presence of copies donated and not acquired by the library, elements both underestimated by the authors examined. At last, it is put into prominence how it is inopportune to accept as indicator in the evaluation of research in HSS the count of the presences of monographs in libraries, and how it is instead shareable the use suggested by Torres-Salinas and Moed of online catalografic data as sources for bibliometric analysis on the distribution of monographs and on the publishers’ production