Evaluation of Spanish institutional repositories based on criteria related to technology, procedures, content, marketing and personnel

Purpose: To provide, through a set of indicators, an overview of the way in which Spanish institutional repositories are run and the services they offer their respective institutions and other users. The selected descriptors are based on aspects related to technology, procedures, content, marketing and the personnel responsible for managing repositories. Design, methodology and approach: In order to establish the indicators, a thorough review of the literature was carried out to identify existing indicators that are used to assess repositories. These were divided into five categories (technology, procedures, content, marketing and personnel) with a total of 48 components. An online survey was conducted with the repositories managers of 66 Spanish research institutions in order to verify the degree of fulfilment of the selected indicators. Findings: The survey received forty-six responses, which represented a response rate of 69%. Of these, 44 came from universities and two from research centres. Sixty-five per cent of the repositories have the capacity to import data from and export data to other university systems, mainly CRIS (32%). Most repositories have mechanisms for the large-scale import and export of metadata and digital objects (83%). The use of altmetrics in repositories is widespread (44%). Authors and librarians deposit most frequently (37% and 32%, respectively), in spite of the fact that 44% do not have full-time staff working in the repository. In more than 80% of the repositories, between 90% and 100% of the deposits are full-text documents. With respect to the tools used to promote the repository within the institution, these are primarily face-to-face training sessions (82%), followed by support materials such as manuals and help pages (65%). The academic authorities encourage open access among researchers in 56% of cases, a significant element in repository marketing. Originality and value: This work proposes a model based on five dimensions and 48 indicators to assess institutional repositories. This approach has been applied to Spanish institutional repositories to provide up-to-date information about their management procedures and promotional methods and the services they offer authors and the university community. This overview of Spanish repositories has provided an insight into the way in which repositories have evolved in recent years and allowed potential improvements to be identified based on the most advanced repositories. This model can also be exported to assess institutional repositories in other countries

Bibliometrics versus altmetrics: Researchers’ attitudes in Slovakia

Objective. The paper presents principal theoretical starting points and an overview of attitudes to altmetrics-based research evaluation among scientists of various disciplines in Slovakia, more specifically in comparison with traditional, bibliometric approaches. Design/Methodology/Approach. We used a questionnaire and asked researchers from various disciplines about their knowledge of principle characteristics defining the altmetrics and about their attitudes towards the application of bibliometrics and altmetrics approaches in the processes of research evaluation. The questionnaire covered a wide range of issues, predominantly associated with interpretation and perception of the importance of alternative indicators, including the factors influencing the interpretation. In this analysis, we concentrate on those parts of research reflecting the willingness of respondents to accept alternative indicators as part of research/science assessment. Results/Discussion. The attitude to the application of alternative indicators in research assessment is not significantly more negative among Slovak scientists if we compare it to other countries. The citations are still a respected indicator perceived as a tool for creating bonds within science. If they are perceived as obsolete or insufficient, it relates to the experience of researchers whose results are predominantly aimed at non-academic target group or if they are active in the research of social phenomena and problems. Due to pragmatic reasons connected with research financing, however, registering the citations is a generally accepted priority. Familiarity and practical experience with social media were identified as one of the factors influencing the interpretation of alternative indicators in the environment of both public and academic media. Conclusions. Traditional bibliometric methods, especially in the area of research evaluation, are still preferred over the altmetrics. The scientists realize that the kind of attention captured by the indicators based on social media is not comparable with the citation-based impact. Or, more precisely, that altmetrics reflects a different kind of impact whose correspondence with a traditional world of bibliographic citations remains to be explored. Originality/Value. The article presents the results of original research carried out in the community of scientists in Slovakia, applying quantitative research methods. It contributes to the results of previous studies on the attitudes and acceptance of altmetrics in the research community, with a special focus on research assessment

Educational Technology as Seen Through the Eyes of the Readers

In this paper, I present the evaluation of a novel knowledge domain visualization of educational technology. The interactive visualization is based on readership patterns in the online reference management system Mendeley. It comprises of 13 topic areas, spanning psychological, pedagogical, and methodological foundations, learning methods and technologies, and social and technological developments. The visualization was evaluated with (1) a qualitative comparison to knowledge domain visualizations based on citations, and (2) expert interviews. The results show that the co-readership visualization is a recent representation of pedagogical and psychological research in educational technology. Furthermore, the co-readership analysis covers more areas than comparable visualizations based on co-citation patterns. Areas related to computer science, however, are missing from the co-readership visualization and more research is needed to explore the interpretations of size and placement of research areas on the map.Comment: Forthcoming article in the International Journal of Technology Enhanced Learnin

The metric tide: report of the independent review of the role of metrics in research assessment and management

This report presents the findings and recommendations of the Independent Review of the Role of Metrics in Research Assessment and Management. The review was chaired by Professor James Wilsdon, supported by an independent and multidisciplinary group of experts in scientometrics, research funding, research policy, publishing, university management and administration. This review has gone beyond earlier studies to take a deeper look at potential uses and limitations of research metrics and indicators. It has explored the use of metrics across different disciplines, and assessed their potential contribution to the development of research excellence and impact. It has analysed their role in processes of research assessment, including the next cycle of the Research Excellence Framework (REF). It has considered the changing ways in which universities are using quantitative indicators in their management systems, and the growing power of league tables and rankings. And it has considered the negative or unintended effects of metrics on various aspects of research culture. The report starts by tracing the history of metrics in research management and assessment, in the UK and internationally. It looks at the applicability of metrics within different research cultures, compares the peer review system with metric-based alternatives, and considers what balance might be struck between the two. It charts the development of research management systems within institutions, and examines the effects of the growing use of quantitative indicators on different aspects of research culture, including performance management, equality, diversity, interdisciplinarity, and the ‘gaming’ of assessment systems. The review looks at how different funders are using quantitative indicators, and considers their potential role in research and innovation policy. Finally, it examines the role that metrics played in REF2014, and outlines scenarios for their contribution to future exercises