Reviewing, indicating, and counting books for modern research evaluation systems

In this chapter, we focus on the specialists who have helped to improve the conditions for book assessments in research evaluation exercises, with empirically based data and insights supporting their greater integration. Our review highlights the research carried out by four types of expert communities, referred to as the monitors, the subject classifiers, the indexers and the indicator constructionists. Many challenges lie ahead for scholars affiliated with these communities, particularly the latter three. By acknowledging their unique, yet interrelated roles, we show where the greatest potential is for both quantitative and qualitative indicator advancements in book-inclusive evaluation systems.Comment: Forthcoming in Glanzel, W., Moed, H.F., Schmoch U., Thelwall, M. (2018). Springer Handbook of Science and Technology Indicators. Springer Some corrections made in subsection 'Publisher prestige or quality

Gaining insight into the Clinical Practice Guideline development processes: qualitative study in a workshop to implement the GRADE proposal in Spain

BACKGROUND: The GRADE method represents a new approach to grading the quality of evidence and strength of recommendations in the preparation of Clinical Practice Guidelines (CPG). In the context of a pilot study to assess the implementability of the system in Spain, we considered it relevant to gain an insight into the significance of the perceptions and attitudes expressed by the actual experts participating in the system try-out. METHODS: Qualitative research with an ethnographic approach, through non-participant observation and focus groups within the context of a consensus workshop in which 19 CPG experts participated to evaluate the GRADE proposal using 12 evidence tables taken from hypertension, asthma and arthritis CPGs. The interventions were recorded, under a guarantee of confidentiality. The transcriptions and field notes were analyzed, based on a sociological discourse analysis model, and the provisional findings were re-sent to participants in order to improve their validity. RESULTS: 1) Certain problems over procedure and terminology hindered the acceptance of this new method as a common reference system for the preparation of CPGs. 2). A greater closeness to clinical practice was accompanied by concerns over value judgments and subjectivity, with a demand for greater explicitness in the consensus process. 3). The type of "evidence" on which the guidelines are based, how and by whom the evidence is prepared, and what the role of the different actors should be, all constitute unresolved concerns in the CPG preparation and implementation processes. 4). The grading process is not neutral: professional background, prior experience and the degree of leadership all condition the participants' input and interactions. CONCLUSION: The findings obtained allow the quantitative evaluation to be better interpreted and, in turn, go beyond the particularities of the GRADE method. Adaptation to the complexities of clinical practice, the need for carefully designed multi-disciplinary work and the reflexivity present in the CPG preparation process, all represent lines of debate that are necessary to improve the CPG quality in the Spanish health care sector

Trends of public health research output from India during 2001-2008

BACKGROUND: An understanding of how public health research output from India is changing in relation to the disease burden and public health priorities is required in order to inform relevant research development. We therefore studied the trends in the public health research output from India during 2001-2008 that was readily available in the public domain. METHODS: The scope and type of the published research from India in 2007 that was included in the PubMed database was assessed and compared with a previous similar assessment for 2002. Papers were classified based on the review of abstracts and original public health research papers were assessed in detail. Impact factors for the journals were used to compute quality-adjusted research output. The websites of governmental organizations, academic and research institutions and international organizations were searched in order to identify and review reports on original public health research produced in India from 2001 to 2008. The reports were classified based on the topics covered and quality and their trends over time were assessed. RESULTS: The number of original health research papers from India in PubMed doubled from 4494 in 2002 to 9066 in 2007. This included a 3.1-fold increase in public health research papers, but these comprised only 5% of the total papers in 2007. Within public health, the increase was lowest for the health system and policy category. Several major causes of disease burden in India continued to be underrepresented in the quality-adjusted public health research output in 2007. The number of papers evaluating population health interventions increased from 2002 to 2007, but there were none on the leading non-communicable causes of disease burden or on road traffic injuries. The number of identified original public health research reports increased by 64.7% from 204 in 2001-2004 to 336 in 2005-2008. The proportion of reports on reproductive and child health was very high but decreased slightly from 38.7% of the total in 2001-2004 to 31.5% in 2005-2008 (P = 0.09); those on the leading chronic non-communicable conditions and injuries increased from 6.4% to 13.4% (P = 0.01) but this was still much lower than their contribution to the disease burden. Health system/policy issues were the topic in 27.4% reports but health information issues were covered in a miniscule 0.6% reports. The proportion of reports that were evaluations increased slightly from 26% in 2001-2004 to 31.5% in 2005-2008, with this proportion being higher among the reports commissioned by international organizations (P < 0.001). The proportion of reports commissioned by Indian governmental organizations alone, or in collaboration with international organizations, doubled from 2001-2004 to 2005-2008 (P < 0.001). Only 25% of the total 540 reports had a quality score of adequate or better. The quality of reports produced by collaborations between Indian and international organizations was higher than those produced by Indian or international organizations alone (P < 0.001). CONCLUSION: This is the first analysis from India that includes research reports in addition to published papers. It provides the most up-to-date understanding of public health research output from India. The increase in available public health research output and the increase in commissioning of this research by Indian governmental organizations are encouraging. However, the distribution of research topics and the quality of research reports continue to be unsatisfactory. It is necessary for health policy to address these continuing deficits in public health research in order to reduce the very large disease burden in India.13 page(s

Mathematical properties of weighted impact factors based on measures of prestige of the citing journals

The final publication is available at Springer via http://dx.doi.org/10.1007/s11192-015-1741-0An abstract construction for general weighted impact factors is introduced. We show that the classical weighted impact factors are particular cases of our model, but it can also be used for defining new impact measuring tools for other sources of information as repositories of datasets providing the mathematical support for a new family of altmet- rics. Our aim is to show the main mathematical properties of this class of impact measuring tools, that hold as consequences of their mathematical structure and does not depend on the definition of any given index nowadays in use. In order to show the power of our approach in a well-known setting, we apply our construction to analyze the stability of the ordering induced in a list of journals by the 2-year impact factor (IF2). We study the change of this ordering when the criterium to define it is given by the numerical value of a new weighted impact factor, in which IF2 is used for defining the weights. We prove that, if we assume that the weight associated to a citing journal increases with its IF2, then the ordering given in the list by the new weighted impact factor coincides with the order defined by the IF2. We give a quantitative bound for the errors committed. We also show two examples of weighted impact factors defined by weights associated to the prestige of the citing journal for the fields of MATHEMATICS and MEDICINE, GENERAL AND INTERNAL, checking if they satisfy the increasing behavior mentioned above.Ferrer Sapena, A.; Sánchez Pérez, EA.; González, LM.; Peset Mancebo, MF.; Aleixandre Benavent, R. (2015). Mathematical properties of weighted impact factors based on measures of prestige of the citing journals. Scientometrics. 105(3):2089-2108. https://doi.org/10.1007/s11192-015-1741-0S208921081053Ahlgren, P., & Waltman, L. (2014). The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments. Journal of Informetrics, 8, 985–996.Aleixandre Benavent, R., Valderrama Zurián, J. C., & González Alcaide, G. (2007). Scientific journals impact factor: Limitations and alternative indicators. El Profesional de la Información, 16(1), 4–11.Altmann, K. G., & Gorman, G. E. (1998). The usefulness of impact factor in serial selection: A rank and mean analysis using ecology journals. Library Acquisitions-Practise and Theory, 22, 147–159.Arnold, D. N., & Fowler, K. K. (2011). Nefarious numbers. Notices of the American Mathematical Society, 58(3), 434–437.Beliakov, G., & James, S. (2012). Using linear programming for weights identification of generalized bonferroni means in R. In: Proceedings of MDAI 2012 modeling decisions for artificial intelligence. Lecture Notes in Computer Science, Vol. 7647, pp. 35–44.Beliakov, G., & James, S. (2011). Citation-based journal ranks: The use of fuzzy measures. Fuzzy Sets and Systems, 167, 101–119.Buela-Casal, G. (2003). Evaluating quality of articles and scientific journals. Proposal of weighted impact factor and a quality index. Psicothema, 15(1), 23–25.Dorta-Gonzalez, P., & Dorta-Gonzalez, M. I. (2013). Comparing journals from different fields of science and social science through a JCR subject categories normalized impact factor. Scientometrics, 95(2), 645–672.Dorta-Gonzalez, P., Dorta-Gonzalez, M. I., Santos-Penate, D. R., & Suarez-Vega, R. (2014). Journal topic citation potential and between-field comparisons: The topic normalized impact factor. Journal of Informetrics, 8(2), 406–418.Egghe, L., & Rousseau, R. (2002). A general frame-work for relative impact indicators. Canadian Journal of Information and Library Science, 27(1), 29–48.Gagolewski, M., & Mesiar, R. (2014). Monotone measures and universal integrals in a uniform framework for the scientific impact assessment problem. Information Sciences, 263, 166–174.Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA, 295(1), 90–93.Habibzadeh, F., & Yadollahie, M. (2008). Journal weighted impact factor: A proposal. Journal of Informetrics, 2(2), 164–172.Klement, E., Mesiar, R., & Pap, E. (2010). A universal integral as common frame for Choquet and Sugeno integral. IEEE Transaction on Fuzzy System, 18, 178–187.Leydesdorff, L., & Opthof, T. (2010). Scopus’s source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations. Journal of the American Society for Information Science and Technology, 61, 2365–2369.Li, Y. R., Radicchi, F., Castellano, C., & Ruiz-Castillo, J. (2013). Quantitative evaluation of alternative field normalization procedures. Journal of Informetrics, 7(3), 746–755.Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4, 265–277.NISO. (2014). Alternative metrics initiative phase 1. White paper. http://www.niso.org/apps/group-public/download.php/13809/Altmetrics-project-phase1-white-paperOwlia, P., Vasei, M., Goliaei, B., & Nassiri, I. (2011). Normalized impact factor (NIF): An adjusted method for calculating the citation rate of biomedical journals. Journal of Biomedical Informatics, 44(2), 216–220.Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics. Information Processing and Management, 12, 297–312.Pinto, A. C., & Andrade, J. B. (1999). Impact factor of scientific journals: What is the meaning of this parameter? Quimica Nova, 22, 448–453.Raghunathan, M. S., & Srinivas, V. (2001). Significance of impact factor with regard to mathematics journals. Current Science, 80(5), 605.Ruiz Castillo, J., & Waltman, L. (2015). Field-normalized citation impact indicators using algorithmically constructed classification systems of science. Journal of Informetrics, 9, 102–117.Saha, S., Saint, S., & Christakis, D. A. (2003). Impact factor: A valid measure of journal quality? Journal of the Medical Library Association, 91, 42–46.Torra, V., & Narukawa, Y. (2008). The h-index and the number of citations: Two fuzzy integrals. IEEE Transaction on Fuzzy System, 16, 795–797.Torres-Salinas, D., & Jimenez-Contreras, E. (2010). Introduction and comparative study of the new scientific journals citation indicators in journal citation reports and scopus. El Profesional de la Información, 19, 201–207.Waltman, L., & van Eck, N. J. (2008). Some comments on the journal weighted impact factor proposed by Habibzadeh and Yadollahie. Journal of Informetrics, 2(4), 369–372.Waltman, L., van Eck, N. J., van Leeuwen, T. N., & Visser, M. S. (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7, 272–285.Zitt, M. (2011). Behind citing-side normalization of citations: some properties of the journal impact factor. Scientometrics, 89, 329–344.Zitt, M., & Small, H. (2008). Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology, 59, 1856–1860.Zyczkowski, K. (2010). Citation graph, weighted impact factors and performance indices. Scientometrics, 85(1), 301–315

Developmental Programming Mediated by Complementary Roles of Imprinted Grb10 in Mother and Pup

Developmental programming links growth in early life with health status in adulthood. Although environmental factors such as maternal diet can influence the growth and adult health status of offspring, the genetic influences on this process are poorly understood. Using the mouse as a model, we identify the imprinted gene Grb10 as a mediator of nutrient supply and demand in the postnatal period. The combined actions of Grb10 expressed in the mother, controlling supply, and Grb10 expressed in the offspring, controlling demand, jointly regulate offspring growth. Furthermore, Grb10 determines the proportions of lean and fat tissue during development, thereby influencing energy homeostasis in the adult. Most strikingly, we show that the development of normal lean/fat proportions depends on the combined effects of Grb10 expressed in the mother, which has the greater effect on offspring adiposity, and Grb10 expressed in the offspring, which influences lean mass. These distinct functions of Grb10 in mother and pup act complementarily, which is consistent with a coadaptation model of imprinting evolution, a model predicted but for which there is limited experimental evidence. In addition, our findings identify Grb10 as a key genetic component of developmental programming, and highlight the need for a better understanding of mother-offspring interactions at the genetic level in predicting adult disease risk

Primary versus secondary source of data in observational studies and heterogeneity in meta-analyses of drug effects: a survey of major medical journals

The data from individual observational studies included in meta-analyses of drug effects are collected either from ad hoc methods (i.e. "primary data") or databases that were established for non-research purposes (i.e. "secondary data"). The use of secondary sources may be prone to measurement bias and confounding due to over-the-counter and out-of-pocket drug consumption, or non-adherence to treatment. In fact, it has been noted that failing to consider the origin of the data as a potential cause of heterogeneity may change the conclusions of a meta-analysis. We aimed to assess to what extent the origin of data is explored as a source of heterogeneity in meta-analyses of observational studies.publishe

Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes

Does peer review fulfill its declared objective of identifying the best science and the best scientists? In order to answer this question we analyzed the Long-Term Fellowship and the Young Investigator programmes of the European Molecular Biology Organization. Both programmes aim to identify and support the best post doctoral fellows and young group leaders in the life sciences. We checked the association between the selection decisions and the scientific performance of the applicants. Our study involved publication and citation data for 668 applicants to the Long-Term Fellowship programme from the year 1998 (130 approved, 538 rejected) and 297 applicants to the Young Investigator programme (39 approved and 258 rejected applicants) from the years 2001 and 2002. If quantity and impact of research publications are used as a criterion for scientific achievement, the results of (zero-truncated) negative binomial models show that the peer review process indeed selects scientists who perform on a higher level than the rejected ones subsequent to application. We determined the extent of errors due to over-estimation (type I errors) and under-estimation (type 2 errors) of future scientific performance. Our statistical analyses point out that between 26% and 48% of the decisions made to award or reject an application show one of both error types. Even though for a part of the applicants, the selection committee did not correctly estimate the applicant's future performance, the results show a statistically significant association between selection decisions and the applicants' scientific achievements, if quantity and impact of research publications are used as a criterion for scientific achievement

Data envelopment analysis in financial services: a citations network analysis of banks, insurance companies and money market funds

Development and application of the data envelopment analysis (DEA) method, have been the subject of numerous reviews. In this paper, we consider the papers that apply DEA methods specifically to financial services, or which use financial services data to experiment with a newly introduced DEA model. We examine 620 papers published in journals indexed in the Web of Science database, from 1985 to April 2016. We analyse the sample applying citations network analysis. This paper investigates the DEA method and its applications in financial services. We analyse the diffusion of DEA in three sub-samples: (1) banking groups, (2) money market funds, and (3) insurance groups by identifying the main paths, that is, the main flows of the ideas underlying each area of research. This allows us to highlight the main approaches, models and efficiency types used in each research areas. No unique methodological preference emerges within these areas. Innovations in the DEA methodologies (network models, slacks based models, directional distance models and Nash bargaining game) clearly dominate recent research. For each subsample, we describe the geographical distribution of these studies, and provide some basic statistics related to the most active journals and scholars