Quality Weighted Citations Versus Total Citations in the Sciences and Social Sciences

__Abstract__ The paper analyses academic journal quality and research impact using quality weighted citations versus total citations, based on the widely-used Thomson Reuters ISI Web of Science citations database (ISI). A new Index of Citations Quality (ICQ) is presented, based on quality weighted citations. The new index is used to analyse the leading 500 journals in both the Sciences and Social Sciences using quantifiable Research Assessment Measures (RAMs) that are based on alternative transformations of citations. It is shown that ICQ is a useful additional measure to 2YIF and other well known RAMs for the purpose of evaluating the impact and quality, as well as ranking, of journals as it contains information that has very low correlations with the information contained in the well known RAMs for both the Sciences and Social Sciences

Ranking Economics and Econometrics ISI Journals by Quality Weighted Citations

__Abstract__ The paper analyses academic journal quality and impact using quality weighted citations that are based on the widely-used Thomson Reuters ISI Web of Science citations database (ISI). A recently developed Index of Citations Quality (ICQ), based on quality weighted citations, is used to analyse the top 276 Economics and top 10 Econometrics journals in the ISI Economics category using alternative quantifiable Research Assessment Measures (RAMs). It is shown that ICQ is a useful additional measure to the 2-Year Impact Factor (2YIF) and other well known RAMs available in ISI for the purpose of evaluating journal impact and quality, as well as ranking, of Economics and Econometrics journals as it contains information that has very low correlations with the information contained in alternative well-known RAMs. Among other findings, the top Econometrics journals have some of the highest ICQ scores in the ISI category of Economics

Quality Weighted Citations versus Total Citations in the Sciences and Social Sciences, with an Application to Finance and Accounting

__Abstract__ The premise underlying the use of citations data is that higher quality journals generally have a higher number of citations. The impact of citations can be distorted in a number of ways. Journals can, and do, inflate the number of citations through self citation practices, which may be coercive. Another method for distorting journal impact is through a set of journals agreeing to cite each other, that is, by exchanging citations. This may be less coercive than self citations, but is nonetheless unprofessional and distortionary. Both journal self citations and exchanged citations have the effect of increasing a journal’s impact factor, which may be deceptive. The paper analyses academic journal quality and research impact using quality weighted citations versus total citations, based on the widely-used Thomson Reuters ISI Web of Science citations database (ISI). A new Index of Citations Quality (ICQ) is presented, based on quality weighted citations. The new index is used to analyse the leading 500 journals in both the Sciences and Social Sciences, as well as 58 leading journals in Finance and Accounting, using quantifiable Research Assessment Measures (RAMs) that are based on alternative transformations of citations. It is shown that ICQ is a useful additional measure to 2YIF and other well known RAMs for the purpose of evaluating the impact and quality, as well as ranking, of journals as it contains information that has very low correlations with the information contained in the well known RAMs for both the Sciences and Social Sciences, as well as in Finance and Accounting

Ranking Economics and Econometrics ISI Journals by Quality Weighted Citations

__Abstract__ The paper analyses academic journal quality and impact using quality weighted citations that are based on the widely-used Thomson Reuters ISI Web of Science citations database (ISI). A recently developed Index of Citations Quality (ICQ), based on quality weighted citations, is used to analyse the top 276 Economics and top 10 Econometrics journals in the ISI Economics category using alternative quantifiable Research Assessment Measures (RAMs). It is shown that ICQ is a useful additional measure to the 2-Year Impact Factor (2YIF) and other well known RAMs available in ISI for the purpose of evaluating journal impact and quality, as well as ranking, of Economics and Econometrics journals as it contains information that has very low correlations with the information contained in alternative well-known RAMs. Among other findings, the top Econometrics journals have some of the highest ICQ scores in the ISI category of Economics

Mathematical properties of weighted impact factors based on measures of prestige of the citing journals

The final publication is available at Springer via http://dx.doi.org/10.1007/s11192-015-1741-0An abstract construction for general weighted impact factors is introduced. We show that the classical weighted impact factors are particular cases of our model, but it can also be used for defining new impact measuring tools for other sources of information as repositories of datasets providing the mathematical support for a new family of altmet- rics. Our aim is to show the main mathematical properties of this class of impact measuring tools, that hold as consequences of their mathematical structure and does not depend on the definition of any given index nowadays in use. In order to show the power of our approach in a well-known setting, we apply our construction to analyze the stability of the ordering induced in a list of journals by the 2-year impact factor (IF2). We study the change of this ordering when the criterium to define it is given by the numerical value of a new weighted impact factor, in which IF2 is used for defining the weights. We prove that, if we assume that the weight associated to a citing journal increases with its IF2, then the ordering given in the list by the new weighted impact factor coincides with the order defined by the IF2. We give a quantitative bound for the errors committed. We also show two examples of weighted impact factors defined by weights associated to the prestige of the citing journal for the fields of MATHEMATICS and MEDICINE, GENERAL AND INTERNAL, checking if they satisfy the increasing behavior mentioned above.Ferrer Sapena, A.; Sánchez Pérez, EA.; González, LM.; Peset Mancebo, MF.; Aleixandre Benavent, R. (2015). Mathematical properties of weighted impact factors based on measures of prestige of the citing journals. Scientometrics. 105(3):2089-2108. https://doi.org/10.1007/s11192-015-1741-0S208921081053Ahlgren, P., & Waltman, L. (2014). The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments. Journal of Informetrics, 8, 985–996.Aleixandre Benavent, R., Valderrama Zurián, J. C., & González Alcaide, G. (2007). Scientific journals impact factor: Limitations and alternative indicators. El Profesional de la Información, 16(1), 4–11.Altmann, K. G., & Gorman, G. E. (1998). The usefulness of impact factor in serial selection: A rank and mean analysis using ecology journals. Library Acquisitions-Practise and Theory, 22, 147–159.Arnold, D. N., & Fowler, K. K. (2011). Nefarious numbers. Notices of the American Mathematical Society, 58(3), 434–437.Beliakov, G., & James, S. (2012). Using linear programming for weights identification of generalized bonferroni means in R. In: Proceedings of MDAI 2012 modeling decisions for artificial intelligence. Lecture Notes in Computer Science, Vol. 7647, pp. 35–44.Beliakov, G., & James, S. (2011). Citation-based journal ranks: The use of fuzzy measures. Fuzzy Sets and Systems, 167, 101–119.Buela-Casal, G. (2003). Evaluating quality of articles and scientific journals. Proposal of weighted impact factor and a quality index. Psicothema, 15(1), 23–25.Dorta-Gonzalez, P., & Dorta-Gonzalez, M. I. (2013). Comparing journals from different fields of science and social science through a JCR subject categories normalized impact factor. Scientometrics, 95(2), 645–672.Dorta-Gonzalez, P., Dorta-Gonzalez, M. I., Santos-Penate, D. R., & Suarez-Vega, R. (2014). Journal topic citation potential and between-field comparisons: The topic normalized impact factor. Journal of Informetrics, 8(2), 406–418.Egghe, L., & Rousseau, R. (2002). A general frame-work for relative impact indicators. Canadian Journal of Information and Library Science, 27(1), 29–48.Gagolewski, M., & Mesiar, R. (2014). Monotone measures and universal integrals in a uniform framework for the scientific impact assessment problem. Information Sciences, 263, 166–174.Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA, 295(1), 90–93.Habibzadeh, F., & Yadollahie, M. (2008). Journal weighted impact factor: A proposal. Journal of Informetrics, 2(2), 164–172.Klement, E., Mesiar, R., & Pap, E. (2010). A universal integral as common frame for Choquet and Sugeno integral. IEEE Transaction on Fuzzy System, 18, 178–187.Leydesdorff, L., & Opthof, T. (2010). Scopus’s source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations. Journal of the American Society for Information Science and Technology, 61, 2365–2369.Li, Y. R., Radicchi, F., Castellano, C., & Ruiz-Castillo, J. (2013). Quantitative evaluation of alternative field normalization procedures. Journal of Informetrics, 7(3), 746–755.Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4, 265–277.NISO. (2014). Alternative metrics initiative phase 1. White paper. http://www.niso.org/apps/group-public/download.php/13809/Altmetrics-project-phase1-white-paperOwlia, P., Vasei, M., Goliaei, B., & Nassiri, I. (2011). Normalized impact factor (NIF): An adjusted method for calculating the citation rate of biomedical journals. Journal of Biomedical Informatics, 44(2), 216–220.Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics. Information Processing and Management, 12, 297–312.Pinto, A. C., & Andrade, J. B. (1999). Impact factor of scientific journals: What is the meaning of this parameter? Quimica Nova, 22, 448–453.Raghunathan, M. S., & Srinivas, V. (2001). Significance of impact factor with regard to mathematics journals. Current Science, 80(5), 605.Ruiz Castillo, J., & Waltman, L. (2015). Field-normalized citation impact indicators using algorithmically constructed classification systems of science. Journal of Informetrics, 9, 102–117.Saha, S., Saint, S., & Christakis, D. A. (2003). Impact factor: A valid measure of journal quality? Journal of the Medical Library Association, 91, 42–46.Torra, V., & Narukawa, Y. (2008). The h-index and the number of citations: Two fuzzy integrals. IEEE Transaction on Fuzzy System, 16, 795–797.Torres-Salinas, D., & Jimenez-Contreras, E. (2010). Introduction and comparative study of the new scientific journals citation indicators in journal citation reports and scopus. El Profesional de la Información, 19, 201–207.Waltman, L., & van Eck, N. J. (2008). Some comments on the journal weighted impact factor proposed by Habibzadeh and Yadollahie. Journal of Informetrics, 2(4), 369–372.Waltman, L., van Eck, N. J., van Leeuwen, T. N., & Visser, M. S. (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7, 272–285.Zitt, M. (2011). Behind citing-side normalization of citations: some properties of the journal impact factor. Scientometrics, 89, 329–344.Zitt, M., & Small, H. (2008). Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology, 59, 1856–1860.Zyczkowski, K. (2010). Citation graph, weighted impact factors and performance indices. Scientometrics, 85(1), 301–315

Vector-valued impact measures and generation of specific indexes for research assessment

A mathematical structure for defining multi-valued bibliometric indices is provided with the aim of measuring the impact of general sources of information others than articles and journals-for example, repositories of datasets. The aim of the model is to use several scalar indices at the same time for giving a measure of the impact of a given source of information, that is, we construct vector valued indices. We use the properties of these vector valued indices in order to give a global answer to the problem of finding the optimal scalar index for measuring a particular aspect of the impact of an information source, depending on the criterion we want to fix for the evaluation of this impact. The main restrictions of our model are (1) it uses finite sets of scalar impact indices (altmetrics), and (2) these indices are assumed to be additive. The optimization procedure for finding the best tool for a fixed criterion is also presented. In particular, we show how to create an impact measure completely adapted to the policy of a specific research institution.Calabuig, JM.; Ferrer Sapena, A.; Sánchez Pérez, EA. (2016). Vector-valued impact measures and generation of specific indexes for research assessment. Scientometrics. 108(3):1425-1443. doi:10.1007/s11192-016-2039-6S142514431083Aleixandre Benavent, R., Valderrama Zurián, J. C., & González Alcaide, G. (2007). Scientific journals impact factor: Limitations and alternative indicators. El Profesional de la Información, 16(1), 4–11.Alguliyev, R., Aliguliyev, R. & Ismayilova, N. (2015). Weighted impact factor (WIF) for assessing the quality of scientific journals. arXiv:1506.02783Beauzamy, B. (1982). Introduction to Banach spaces and their geometry. Amsterdam: North-Holland.Beliakov, G., & James, S. (2011). Citation-based journal ranks: the use of fuzzy measures. Fuzzy Sets and Systems, 167, 101–119.Buela-Casal, G. (2003). Evaluating quality of articles and scientific journals. Proposal of weighted impact factor and a quality index. Psicothema, 15(1), 23–25.Diestel, J., & Uhl, J. J. (1977). Vector measures. Providence: Am. Math. Soc.Dorta-González, P., & Dorta-González, M. I. (2013). Comparing journals from different fields of science and social science through a JCR subject categories normalized impact factor. Scientometrics, 95(2), 645–672.Dorta-González, P., Dorta-González, M. I., Santos-Penate, D. R., & Suarez-Vega, R. (2014). Journal topic citation potential and between-field comparisons: The topic normalized impact factor. Journal of Informetrics, 8(2), 406–418.Egghe, L., & Rousseau, R. (2002). A general frame-work for relative impact indicators. Canadian Journal of Information and Library Science, 27(1), 29–48.Ferrer-Sapena, A., Sánchez-Pérez, E. A., González, L. M., Peset, F. & Aleixandre-Benavent, R. (2016). The impact factor as a measuring tool of the prestige of the journals in research assessment in mathematics. Research Evaluation, 1–9. doi: 10.1093/reseval/rvv041 .Ferrer-Sapena, A., Sánchez-Pérez, E. A., González, L. M., Peset, F., & Aleixandre-Benavent, R. (2015). Mathematical properties of weighted impact factors based on measures of prestige of the citing journals. Scientometrics, 105(3), 2089–2108.Gagolewski, M., & Mesiar, R. (2014). Monotone measures and universal integrals in a uniform framework for the scientific impact assessment problem. Information Sciences, 263, 166–174.Habibzadeh, F., & Yadollahie, M. (2008). Journal weighted impact factor: A proposal. Journal of Informetrics, 2(2), 164–172.Klement, E., Mesiar, R., & Pap, E. (2010). A universal integral as common frame for Choquet and Sugeno integral. IEEE Transactions on Fuzzy Systems, 18, 178–187.Leydesdorff, L., & Opthof, T. (2010). Scopus’s source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations. Journal of the American Society for Information Science and Technology, 61, 2365–2369.Li, Y. R., Radicchi, F., Castellano, C., & Ruiz-Castillo, J. (2013). Quantitative evaluation of alternative field normalization procedures. Journal of Informetrics, 7(3), 746–755.Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4, 265–277.Owlia, P., Vasei, M., Goliaei, B., & Nassiri, I. (2011). Normalized impact factor (NIF): An adjusted method for calculating the citation rate of biomedical journals. Journal of Biomedical Informatics, 44(2), 216–220.Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics. Information Processing and Management, 12, 297–312.Piwowar, H. (2013). Altmetrics: Value all research products. Nature, 493(7431), 159–159.Pudovkin,A.I., & Garfield, E. (2004). Rank-normalized impact factor: A way to compare journal performance across subject categories. In Proceedings of the 67th annual meeting of the American Society for Information science and Technology, 41, 507-515.Rousseau, R. (2002). Journal evaluation: Technical and practical issues. Library Trends, 50(3), 418–439.Ruiz Castillo, J., & Waltman, L. (2015). Field-normalized citation impact indicators using algorithmically constructed classification systems of science. Journal of Informetrics, 9, 102–117.Torra, V., & Narukawa, Y. (2008). The h-index and the number of citations: Two fuzzy integrals. IEEE Transactions on Fuzzy Systems, 16, 795–797.Waltman, L., & van Eck, N. J. (2008). Some comments on the journal weighted impact factor proposed by Habibzadeh and Yadollahie. Journal of Informetrics, 2(4), 369–372.Waltman, L., & van Eck, N. J. (2010). The relation between Eigenfactor, audience factor, and influence weight. Journal of the American Society for Information Science and Technology, 61, 1476–1486.Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ’alternative metrics’ in scientific publications. Scientometrics, 101(2), 1491–1513.Zitt, M. (2010). Citing-side normalization of journal impact: A robust variant of the Audience Factor. Journal of Informetrics, 4(3), 392–406.Zitt, M. (2011). Behind citing-side normalization of citations: Some properties of the journal impact factor. Scientometrics, 89, 329–344.Zitt, M., & Small, H. (2008). Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology, 59, 1856–1860.Zyczkowski, K. (2010). Citation graph, weighted impact factors and performance indices. Scientometrics, 85(1), 301–315

Co-author weighting in bibliometric methodology and subfields of a scientific discipline

Collaborative work and co-authorship are fundamental to the advancement of modern science. However, it is not clear how collaboration should be measured in achievement-based metrics. Co-author weighted credit introduces distortions into the bibliometric description of a discipline. It puts great weight on collaboration - not based on the results of collaboration - but purely because of the existence of collaborations. In terms of publication and citation impact, it artificially favors some subdisciplines. In order to understand how credit is given in a co-author weighted system (like the NRC's method), we introduced credit spaces. We include a study of the discipline of physics to illustrate the method. Indicators are introduced to measure the proportion of a credit space awarded to a subfield or a set of authors.Comment: 11 pages, 1 figure, 4 table

A review of the characteristics of 108 author-level bibliometric indicators

An increasing demand for bibliometric assessment of individuals has led to a growth of new bibliometric indicators as well as new variants or combinations of established ones. The aim of this review is to contribute with objective facts about the usefulness of bibliometric indicators of the effects of publication activity at the individual level. This paper reviews 108 indicators that can potentially be used to measure performance on the individual author level, and examines the complexity of their calculations in relation to what they are supposed to reflect and ease of end-user application.Comment: to be published in Scientometrics, 201

Evaluating research - Peer review team assessment and journal-based bibliographic measures: New Zealand PBRF research output scores in 2006

This paper concerns the relationship between the assessment of the research of individual academics by peer or expert review teams with a variety of bibliometric schemes based on journal quality weights. Specifically, for a common group of economists from New Zealand departments of economics the relationship between Performance-Based Research Fund (PBRF) Research Output measures for those submitting new research portfolios in 2006 are compared with evaluations of journal based research over the 2000-2005 assessment period. This comparison identifies the journal weighting schemes that appear most similar to PBRF peer evaluations. The paper provides an indication of the ‘power or aggressiveness’ of PBRF evaluations in terms of the weighting given to quality. The implied views of PBRF peer review teams are also useful in assessing common assumptions made in evaluating journal based research