A Survey of Quality Engineering-ManagementJournals by Bibliometric Indicators

This paper analyses some of the most popular scientific journals in the Quality field from the point of view of three bibliometric indicators: the Hirsch (h) index for journals, the total number of citations and the h-spectrum. In particular, h-spectrum is a novel tool based on h, making it possible to (i) identify a reference profile of the typical authors of a journal; (ii) compare different journals; and (iii) provide a rough indication of their ‘bibliometric positioning' in the scientific community. Results of this analysis can be helpful for guiding potential authors and members of the scientific community in the Quality Engineering/Management area. A large amount of empirical data are presented and discusse

The metric tide: report of the independent review of the role of metrics in research assessment and management

This report presents the findings and recommendations of the Independent Review of the Role of Metrics in Research Assessment and Management. The review was chaired by Professor James Wilsdon, supported by an independent and multidisciplinary group of experts in scientometrics, research funding, research policy, publishing, university management and administration. This review has gone beyond earlier studies to take a deeper look at potential uses and limitations of research metrics and indicators. It has explored the use of metrics across different disciplines, and assessed their potential contribution to the development of research excellence and impact. It has analysed their role in processes of research assessment, including the next cycle of the Research Excellence Framework (REF). It has considered the changing ways in which universities are using quantitative indicators in their management systems, and the growing power of league tables and rankings. And it has considered the negative or unintended effects of metrics on various aspects of research culture. The report starts by tracing the history of metrics in research management and assessment, in the UK and internationally. It looks at the applicability of metrics within different research cultures, compares the peer review system with metric-based alternatives, and considers what balance might be struck between the two. It charts the development of research management systems within institutions, and examines the effects of the growing use of quantitative indicators on different aspects of research culture, including performance management, equality, diversity, interdisciplinarity, and the ‘gaming’ of assessment systems. The review looks at how different funders are using quantitative indicators, and considers their potential role in research and innovation policy. Finally, it examines the role that metrics played in REF2014, and outlines scenarios for their contribution to future exercises

How journal rankings can suppress interdisciplinary research. A comparison between Innovation Studies and Business & Management

This study provides quantitative evidence on how the use of journal rankings can disadvantage interdisciplinary research in research evaluations. Using publication and citation data, it compares the degree of interdisciplinarity and the research performance of a number of Innovation Studies units with that of leading Business & Management schools in the UK. On the basis of various mappings and metrics, this study shows that: (i) Innovation Studies units are consistently more interdisciplinary in their research than Business & Management schools; (ii) the top journals in the Association of Business Schools' rankings span a less diverse set of disciplines than lower-ranked journals; (iii) this results in a more favourable assessment of the performance of Business & Management schools, which are more disciplinary-focused. This citation-based analysis challenges the journal ranking-based assessment. In short, the investigation illustrates how ostensibly 'excellence-based' journal rankings exhibit a systematic bias in favour of mono-disciplinary research. The paper concludes with a discussion of implications of these phenomena, in particular how the bias is likely to affect negatively the evaluation and associated financial resourcing of interdisciplinary research organisations, and may result in researchers becoming more compliant with disciplinary authority over time.Comment: 41 pages, 10 figure

Bibliometric studies on single journals: a review

This paper covers a total of 82 bibliometric studies on single journals (62 studies cover unique titles) published between 1998 and 2008 grouped into the following fields; Arts, Humanities and Social Sciences (12 items); Medical and Health Sciences (19 items); Sciences and Technology (30 items) and Library and Information Sciences (21 items). Under each field the studies are described in accordance to their geographical location in the following order, United Kingdom, United States and Americana, Europe, Asia (India, Africa and Malaysia). For each study, elements described are (a) the journal’s publication characteristics and indexation information; (b) the objectives; (c) the sampling and bibliometric measures used; and (d) the results observed. A list of journal titles studied is appended. The results show that (a)bibliometric studies cover journals in various fields; (b) there are several revisits of some journals which are considered important; (c) Asian and African contributions is high (41.4 of total studies; 43.5 covering unique titles), United States (30.4 of total; 31.0 on unique titles), Europe (18.2 of total and 14.5 on unique titles) and the United Kingdom (10 of total and 11 on unique titles); (d) a high number of bibliometrists are Indians and as such coverage of Indian journals is high (28 of total studies; 30.6 of unique titles); and (e) the quality of the journals and their importance either nationally or internationally are inferred from their indexation status

Opinion mining and sentiment analysis in marketing communications: a science mapping analysis in Web of Science (1998–2018)

Opinion mining and sentiment analysis has become ubiquitous in our society, with applications in online searching, computer vision, image understanding, artificial intelligence and marketing communications (MarCom). Within this context, opinion mining and sentiment analysis in marketing communications (OMSAMC) has a strong role in the development of the field by allowing us to understand whether people are satisfied or dissatisfied with our service or product in order to subsequently analyze the strengths and weaknesses of those consumer experiences. To the best of our knowledge, there is no science mapping analysis covering the research about opinion mining and sentiment analysis in the MarCom ecosystem. In this study, we perform a science mapping analysis on the OMSAMC research, in order to provide an overview of the scientific work during the last two decades in this interdisciplinary area and to show trends that could be the basis for future developments in the field. This study was carried out using VOSviewer, CitNetExplorer and InCites based on results from Web of Science (WoS). The results of this analysis show the evolution of the field, by highlighting the most notable authors, institutions, keywords, publications, countries, categories and journals.The research was funded by Programa Operativo FEDER Andalucía 2014‐2020, grant number “La reputación de las organizaciones en una sociedad digital. Elaboración de una Plataforma Inteligente para la Localización, Identificación y Clasificación de Influenciadores en los Medios Sociales Digitales (UMA18‐ FEDERJA‐148)” and The APC was funded by the same research gran

Assessing the payback from health R & D: From ad hoc studies to regular monitoring

Chapter 1 : Introduction • The increasing demands for the benefits of payback from publicly funded R&D to be assessed are based partly on the need to justify or account for expenditure on R&D, and partly on the desire for information to assist resource allocation and the better management of R&D funds. The former consideration is particularly strong in relation to the R&D expenditure that comes out of the wider NHS budget. • In this report a range of categories of payback will be identified along with a variety of methods for assessing them. • The aim of the report is to make recommendations as to how the outcomes from health research might best be monitored on a regular basis. The specific context of the report is the NHS R&D Programme but many of the issues will be relevant for a wide range of funders of health R&D. • The introduction sets out not only a plan of the report but also suggests that readers familiar with the general arguments and existing literature may choose to jump to Chapter 6. Chapter 2 : Review of Existing Approaches to Assessing the Payback from Research • Existing work describes various approaches to valuing research. Some are ex ante and attempt to predict the outcomes of research being considered, others are ex post or retrospective. • The five categories of benefit or payback from health R&D that have been identified involve contributions: to knowledge; to research capacity and future research; to improved information for decision making; to the efficiency, efficacy and equity of health care services; and to the nation’s economic performance. These are shown in Table 1 of the report • The process by which R&D generates final outcomes can be modelled as a sequence. This includes primary outputs such as publications; secondary outputs in the form of policy or administrative decisions; and final outcomes which comprise the health and economic benefits. Feedback loops are also introduced and mitigate the limitations of a linear approach. • Qualitative and quantitative approaches can be used but there are immense problems with time lags and attributing outcomes, and sometimes even outputs, to specific items of research funding. • Four common methods of measuring payback can be used. Expert review, by peers or, sometimes, users is the traditional way of assessing the quality of research. Bibliometric techniques can involve not only counting publications but also using datasets such as the Science Citation Index and Wellcome’s Research Outputs Database (ROD). The various methods of economic analysis of payback are difficult to undertake given the costs and problems of acquiring relevant information and estimating benefits. Social science methods include case studies, which can provide useful information but are resource intensive, and questionnaires to researchers and potential research users. Chapter 3 : Characteristics of a Routine Monitoring System • In moving from ad hoc or research studies of payback towards a more regular monitoring it is noted that whereas there has always been a tradition of evaluation of research, in the public services in general there is now a greater emphasis on audit and performance measurement and indicators. A review of these various systems suggests we should be looking to develop a system of outcomes monitoring that incorporates performance indicators (PIs) and measurement rather than an audit system that is trying to monitor activities against predetermined targets. • Standard characteristics of performance measurement systems do not necessarily apply to research where, for example, there are non-standard outputs. Difficulties have arisen in the USA in attempting to apply the Government Performance and Results Act to research funding agencies. It is shown that because the findings of basic research, in particular, enter a knowledge pool in which people and ideas interact, it is difficult to use a PIs’ approach to track eventual outcomes. However, for some types of health research it has proved more feasible to trace the flow between research outputs and outcomes. • An outcomes monitoring system could be useful if it met the following criteria: relevant to, with as comprehensive coverage as possible of, the funders objectives; relevant to the funder’s decision making processes; encourages accurate compliance; minimises unintended consequences; and has acceptable costs. Chapter 4 : Differences Between Research Types • The range of differences between types of research can be relevant for the design of a routine monitoring system. The OECD distinguishes between basic research, applied research and experimental development. Most DH/NHS research is applied. There might be more of a tradition of publication of findings in applied research in health than in other fields. Nevertheless, the publication and incentives patterns operating in basic research mean that it would be inappropriate to use bibliometric indicators in a simple way across all fields even in health research. • Despite having some differences from health research in publication patterns and in the detailed categories of payback, the broad approach proposed in Chapter 6 could be applied to social care research. • Research that is commissioned, especially by the government, has some of the minimum conditions built into it that are associated with outcomes being generated, in particular because the funder has identified that a contribution in this area will be valuable. Chapter 5 : What Units of Research? • The term programme has various meanings including being used to describe a collection of projects on a common theme and to describe a block of funding for a research unit. • Three main streams or modes of funding can be identified: projects, which are administratively grouped into programmes including a responsive programme; institutions/centres/units; individual researchers. These 3 streams are displayed in Figure 1. It is probable that the regular data-gathering for a monitoring system would operate at the basic level of each stream or mode. • Previous work demonstrates that the full range of benefits can sometimes be applied at the level of projects, either in the responsive mode or in programmes, through the use of questionnaires to researchers. Expert and user review and user surveys have also been applied. • Institutions and centres increasingly have experience not only of traditional periodic expert review but also of producing annual reports, although there are debates about what dimensions to include in such reviews and reports. • Individuals in receipt of research development awards have completed questionnaires during and after the awards. These concentrate on the development of research capacity but can go wider. Chapter 6 : A Possible Comprehensive Outcomes Monitoring System • The proposed system is intended for DH/NHS to monitor the outcomes from its R&D in order to justify the R&D expenditure and assist with managing the portfolio. More detailed information is required for the latter purpose. • We propose a multidimensional approach be adopted to cover all the dimensions of payback and that information be gathered from three sets of sources and Table 3 shows which methods would cover which output/outcome categories. • Firstly, possibly annually, a questionnaire (possibly electronic) covering most payback categories should gather data from the basic level of each funding stream ie. from lead researchers of projects, from research institutions/centres, and from individual award holders. • Secondly, supplementary information should be gathered from external databases (including the citation indices and Wellcome’s ROD). • Thirdly, a range of approaches ie. user surveys, reviews by experts and peers, case studies including economic evaluations, and analysis of sources used in policy documents such as NICE guidelines, would be undertaken on a sample basis. They would provide not only supplementary information but, as with the external databases, would also verify the data collected directly from researchers. • These proposals can be evaluated against the criteria set out in Chapter 3: • The system is relevant to DH’s objectives of generating payback in a range of categories. • Various problems have to be overcome before the system could be fully decision relevant. Firstly it might be necessary to ask researchers to apportion the contribution made to specific outputs from various funding streams. Second, to be decision relevant the information would have to be analysed and presented in a manner consistent with funders’ decision making processes. This would involve a) showing how for each outcome and output, for example publications, data from one project or stream could be compared with those from another and b) demonstrating how different outputs and outcomes could be aggregated. • The questions of accuracy of data, minimisation of unintended consequences and the acceptability of the net costs are also addressed. Chapter 7 : Research and Monitoring • Whilst this report is primarily concerned with moving from ad hoc studies towards a routine monitoring system there are issues that need further research. • Before embarking on full implementation the feasibility needs to be tested of items such as on-line recording of data and asking researchers to attribute proportions of research outputs to separate funding agencies. • Once the system is implemented the value of some items can be better assessed, for example the additional value provided by self reporting of publications beyond that gained from relying on external databases. • The data provided by the system would provide opportunities for further payback research on, for example, links between publications and other categories of payback. • Some items such as network analysis could potentially be added to the monitoring system after further examination of them. • Finally the benefit from the monitoring system itself should be assessed.Department of Health; Wellcome Trus

A Review of Theory and Practice in Scientometrics

Scientometrics is the study of the quantitative aspects of the process of science as a communication system. It is centrally, but not only, concerned with the analysis of citations in the academic literature. In recent years it has come to play a major role in the measurement and evaluation of research performance. In this review we consider: the historical development of scientometrics, sources of citation data, citation metrics and the “laws" of scientometrics, normalisation, journal impact factors and other journal metrics, visualising and mapping science, evaluation and policy, and future developments