A Review of Theory and Practice in Scientometrics

Scientometrics is the study of the quantitative aspects of the process of science as a communication system. It is centrally, but not only, concerned with the analysis of citations in the academic literature. In recent years it has come to play a major role in the measurement and evaluation of research performance. In this review we consider: the historical development of scientometrics, sources of citation data, citation metrics and the “laws" of scientometrics, normalisation, journal impact factors and other journal metrics, visualising and mapping science, evaluation and policy, and future developments

Theories of Informetrics and Scholarly Communication

Scientometrics have become an essential element in the practice and evaluation of science and research, including both the evaluation of individuals and national assessment exercises. Yet, researchers and practitioners in this field have lacked clear theories to guide their work. As early as 1981, then doctoral student Blaise Cronin published The need for a theory of citing - a call to arms for the fledgling scientometric community to produce foundational theories upon which the work of the field could be based. More than three decades later, the time has come to reach out the field again and ask how they have responded to this call. This book compiles the foundational theories that guide informetrics and scholarly communication research. It is a much needed compilation by leading scholars in the field that gathers together the theories that guide our understanding of authorship, citing, and impact

Theories of Informetrics and Scholarly Communication

Scientometrics have become an essential element in the practice and evaluation of science and research, including both the evaluation of individuals and national assessment exercises. Yet, researchers and practitioners in this field have lacked clear theories to guide their work. As early as 1981, then doctoral student Blaise Cronin published "The need for a theory of citing" —a call to arms for the fledgling scientometric community to produce foundational theories upon which the work of the field could be based. More than three decades later, the time has come to reach out the field again and ask how they have responded to this call. This book compiles the foundational theories that guide informetrics and scholarly communication research. It is a much needed compilation by leading scholars in the field that gathers together the theories that guide our understanding of authorship, citing, and impact

Determining cognitive distance between publication portfolios of evaluators and evaluees in research evaluation: an exploration of informetric methods

This doctoral thesis develops informetric methods for determining cognitive distance between publication portfolios of evaluators and evaluees in research evaluation. In a discipline specific research evaluation, when an expert panel evaluates research groups, it is an open question how one can determine the extent to which the panel members are in a position to evaluate the research groups. This thesis contributes to the literature by proposing six different informetric approaches to measure the match between evaluators and evaluees using their publications as a representation of their expertise. An expert panel is specifically appointed for the research evaluation. Experts are typically selected in one of two ways: (1) straightforward selection: the person(s) in charge of the research evaluation has access to a list of acknowledged experts in specific fields, and limits its selection process to ensuring the experts’ independence regarding the program under evaluation; and (2) gradual selections: preferred profiles of experts are developed with respect to the specialization under scrutiny in the evaluation. Both ways leave some freedom for an “old boys’ network” to appoint someone without properly evaluating their qualifications. There are also other ways for expert selection, for example, inviting open application or the research groups that will be evaluated can propose their choice of experts. In research evaluation, an expert panel usually comprises independent specialists, each of which is recognized in at least one of the fields addressed by the unit under evaluation. The expertise of the panel members should be congruent with the research groups to ensure the quality and trustworthiness of the evaluation. All things being equal, panel members who are credible experts in the field are also most likely to provide valuable, relevant recommendations and suggestions that should lead to improved research quality. However, there was an absence of methods to determine the cognitive distance between evaluators and evaluees in research evaluation when we started working in July 2013. In this thesis, we develop and test informetric methods to identify the cognitive distances between the (members of) an expert panel on the one hand, and the (whole of the) units of assessment (typically research groups) on the other. More generally, we introduce a number of methods that allow measuring cognitive distances based on publication portfolios. In academia, publications are considered key indicators of expertise that help to identify qualified or similar experts to assign papers for review, and to form an expert panel. Our main objective is to propose informetric methods to identify panel members who have closely related expertise in the research domain of the research groups based on their publications profile. The main factor that we have taken into account is the cognitive distance between an expert panel and research groups. We consider the publication portfolio of the involved researchers to reflect the position of the unit in cognitive space and, hence, to determine cognitive distance. Expressed in general terms we measure cognitive distance between units based on how often they have published in the same or similar journals. Our investigations lead to the development of new methods of expert panel composition for the research evaluation exercises. We explore different ways of quantifying the cognitive distance between panel members and research group's publication profiles. We consider all the publications of the research groups (during the eight years preceding their evaluation) and panel members indexed in Web of Science (WoS). We pursue the investigation at two levels of aggregation: WoS subject categories (SCs) and journals. The aggregated citation relations among SCs or journals provide a matrix. From the matrix, one can construct a similarity matrix. From the similarity matrix, one can construct a global SCs or journal map in which similar SCs or journals are located more closely together. The maps can be visualized using a visualization program. During the visualization process, a multi-dimensional space is reduced to a projection in two dimensions. In this process, similar SCs or journals are positioned closer to each other. We propose three methods, namely the use of barycenters, of similarity-adapted publication vector (SAPV) and of weighted cosine similarity (WCS). We take into account the similarity between WoS SCs and between journals, either by incorporating a similarity matrix (in the case of SAPV and WCS) or a 2-dimensional base map derived from it (in the case of barycenters). We determine the coordinates of barycenters using a 2-dimensional base map based on the publication profiles of research groups and panel members, and calculate the Euclidean distances between the barycenters. We also identify SAPV using the similarity matrix and calculated the Euclidean distances between the SAPVs. Finally, we calculate WCS using the similarity matrix. The SAPV and WCS methods use a square N-dimensional similarity matrix. Here N is equivalent to 224 WoS SCs and 10,675 journals. We used the distance/similarity between panel members and research groups as an indicator of cognitive distance. Small differences in Euclidean distances (both between barycenters and SAPVs) or in cosine similarity values bear little meaning. For this reason, we employ a bootstrapping approach in order to determine a 95% confidence interval (CI) for each distance or similarity value. If two CIs do not overlap, difference between the values is statistically significant at the 0.05 level. Although it is possible for two values to have a statistically significant difference while having overlapping CIs, the difference is less likely to have practical meaning. Two levels of aggregation and three methods lead to six informetric approaches to quantify the cognitive distance. Our proposed approaches hold advantages over a simple comparison of publication portfolios. Our approaches quantify the cognitive distance between a research group and panel members. We also compare our proposed approaches. We examine which of the approaches best reflects the prior assignment of main assessor to each research group, how much influence the level of aggregation (journals and WoS SCs) plays, and how much the dimensionality matters. The results show that, regardless of the method used, the level of aggregation has only a minor influence, whereas the influence of the number of dimensions is substantial. The results also show that the number of dimensions plays a major role in the case of identifying shortest cognitive distance. While the SAPV and WCS methods agree at most of cases at both the levels of aggregation the barycenter approaches yield different results. We find that the barycenter approaches score highest at both levels of aggregation to identify the previously assigned main assessor. When it comes to uniquely identifying the main assessor, all methods score better at the journal level than at the WoS SC level. Our approaches, but of course not the numerical result, are independent of the similarity matrix or map used. All six approaches give the opportunity to assess the composition of the panel in terms of cognitive distance if one or more panel members are replaced and compare the relative contribution of each potential panel member to the panel fit as a whole, by observing the changes to the distance between the panel’s and the groups’. In addition, our approaches allow the panel composition authority to see in advance about the panel’s fit to the research groups that are going to be evaluated. Therefore, the concerned authority will have the opportunity to replace outliers among the panel members to make the panel fit well with the research groups to be evaluated. For example, the authority can find a best-fitting expert panel by replacing a more distant panel member with a potential panel member located closer to the groups

Theories of Informetrics and Scholarly Communication

Scientometrics have become an essential element in the practice and evaluation of science and research, including both the evaluation of individuals and national assessment exercises. Yet, researchers and practitioners in this field have lacked clear theories to guide their work. As early as 1981, then doctoral student Blaise Cronin published "The need for a theory of citing" —a call to arms for the fledgling scientometric community to produce foundational theories upon which the work of the field could be based. More than three decades later, the time has come to reach out the field again and ask how they have responded to this call. This book compiles the foundational theories that guide informetrics and scholarly communication research. It is a much needed compilation by leading scholars in the field that gathers together the theories that guide our understanding of authorship, citing, and impact

Collaboration between UK Universities: A machine-learning based webometric analysis

A thesis submittedCollaboration is essential for some types of research, which is why some agencies include collaboration among the requirements for funding research projects. Studying collaborative relationships is important because analyses of collaboration networks can give insights into knowledge based innovation systems, the roles that different organisations play in a research field and the relationships between scientific disciplines. Co-authored publication data is widely used to investigate collaboration between organisations, but this data is not free and thus may not be accessible for some researchers. Hyperlinks have some similarities with citations, so hyperlink data may be used as an indicator to estimate the extent of collaboration between academic institutions and may be able to show types of relationships that are not present in co-authorship data. However, it has been shown that using raw hyperlink counts for webometric research can sometimes produce unreliable results, so researchers have attempted to find alternate counting methods and have tried to identify the reasons why hyperlinks may have been created in academic websites. This thesis uses machine learning techniques, an approach that has not previously been widely used in webometric research, to automatically classify hyperlinks and text in university websites in an attempt to filter out irrelevant hyperlinks when investigating collaboration between academic institutions. Supervised machine learning methods were used to automatically classify the web page types that can be found in Higher Education Institutions’ websites. The results were assessed to see whether ii automatically filtered hyperlink data gave better results than raw hyperlink data in terms of identifying patterns of collaboration between UK universities. Unsupervised learning methods were used to automatically identify groups of university departments that are collaborating or that may benefit from collaborating together, based on their co-appearance in research clusters. Results show that the machine learning methods used in this thesis can automatically identify both the source and target web page categories of hyperlinks in university websites with up to 78% accuracy; which means that it can increase the possibility for more effective hyperlink classification or for identifying the reasons why hyperlinks may have been created in university websites, if those reasons can be inferred from the relationship between the source and target page types. When machine learning techniques were used to filter hyperlinks that may not have been created because of collaboration from the hyperlink data, there was an increased correlation between hyperlink data and other collaboration indicators. This emphasises the possibility for using machine learning methods to make hyperlink data a more reliable data source for webometric research. The reasons for university name mentions in the different web page types found in an academic institution’s website are broadly the same as the reasons for link creation, this means that classification based on inter-page relationships may also be used to improve name mentions data for webometrics research. iii Clustering research groups based on the text in their homepages may be useful for identifying those research groups or departments with similar research interests which may be valuable for policy makers in monitoring research fields; based on the sizes of identified clusters and for identifying future collaborators; based on co-appearances in clusters, if identical research interests is a factor that can influence the choice of a future collaborator. In conclusion, this thesis shows that machine learning techniques can be used to significantly improve the quality of hyperlink data for webometrics research, and can also be used to analyse other web based data to give additional insights that may be beneficial for webometrics studies

Mining Scholarly Publications for Research Evaluation

Scientific research can lead to breakthroughs that revolutionise society by solving long-standing problems. However, investment of public funds into research requires the ability to clearly demonstrate beneficial returns, accountability, and good management. At the same time, with the amount of scholarly literature rapidly expanding, recognising key research that presents the most important contributions to science is becoming increasingly difficult and time-consuming. This creates a need for effective and appropriate research evaluation methods. However, the question of how to evaluate the quality of research outcomes is very difficult to answer and despite decades of research, there is still no standard solution to this problem. Given this growing need for research evaluation, it is increasingly important to understand how research should be evaluated, and whether the existing methods meet this need. However, the current solutions, which are predominantly based on counting the number of interactions in the scholarly communication network, are insufficient for a number of reasons. In particular, they struggle in capturing many aspects of the academic culture and often significantly lag behind current developments. This work focuses on the evaluation of research publications and aims at creating new methods which utilise publication content. It studies the concept of research publication quality, methods assessing the performance of new research publication evaluation methods, analyses and extends the existing methods, and, most importantly, presents a new class of metrics which are based on publication manuscripts. By bridging the fields of research evaluation and text- and data-mining, this work provides tools for analysing the outcomes of research, and for relieving information overload in scholarly publishing