Citation distributions are so skewed that using the mean or any other central
tendency measure is ill-advised. Unlike G. Prathap's scalar measures (Energy,
Exergy, and Entropy or EEE), the Integrated Impact Indicator (I3) is based on
non-parametric statistics using the (100) percentiles of the distribution.
Observed values can be tested against expected ones; impact can be qualified at
the article level and then aggregated.Comment: Scientometrics, in pres

A. Schubert

AFJ Raan van

AI Pudovkin

E Garfield

G Prathap

H Theil

HF Moed

I Rafols

L Bornmann

L Leydesdorff

L Waltman

Loet Leydesdorff

O Amsterdamska

PO Seglen

T Opthof

Tobias Opthof

W Glänzel

Y. Gingras

English

arXiv

Crossref

A rejoinder on energy versus impact indicatorsLoet Leydesdorff • Tobias OpthofReceived: 26 August 2011 / Published online: 20 September 2011 The Author(s) 2011. This article is published with open access at Springerlink.comAbstract Citation distributions are so skewed that using the mean or any other centraltendency measure is ill-advised. Unlike G. Prathap’s scalar measures (Energy, Exergy, andEntropy or EEE), the Integrated Impact Indicator (I3) is based on non-parametric statisticsusing the (100) percentiles of the distribution. Observed values can be tested againstexpected ones; impact can be qualified at the article level and then aggregated.Keywords Citation  Scalar  Integration  EEE  I3  QualityPrathap (2011a) in his Letter applies newly developed scalar measures for bibliometrics(Energy, Exergy, and Entropy; EEE) to the data provided in Table 1 of Van Raan (2006,p. 495). EEE operates on averages and ignores the shape of the underlying distributions ofcitations (‘‘the citation curves’’). (Let us note about EEE that energy and exergy sharedimensionality, but entropy is expressed in Watts/Kelvin. Thus, the expression Energy -Exergy = Entropy as suggested by Prathap (2011b) is, in our opinion, invalid without thespecification of a meta-physical analog on of the ‘‘temperature.’’)Like Prathap (2011b) and following Bornmann and Mutz (2011), Leydesdorff et al.(2011) have elaborated the percentile-rank as a scalar sum by using the same dataset thatled to the original contention about how citation data should be normalized (Opthof andL. Leydesdorff (&)Amsterdam School of Communication Research (ASCoR), University of Amsterdam,Kloveniersburgwal 48, 1012 CX Amsterdam, The Netherlandse-mail: loet@leydesdorff.netT. OpthofExperimental Cardiology Group, Heart Failure Research Center, Academic Medical Center AMC,Meibergdreef 9, 1105 AZ Amsterdam, The Netherlandse-mail: t.opthof@inter.nl.netT. OpthofDepartment of Medical Physiology, University Medical Center, Utrecht, The Netherlands123Scientometrics (2012) 90:745–748DOI 10.1007/s11192-011-0502-yLeydesdorff 2010; Van Raan et al. 2010). More recently, Leydesdorff and Bornmann(2011) have developed this scalar measure into the Integrated Impact Indicator (I3).1The difference between I3 and EEE is that I3 takes the shapes of the distribution intoaccount and allows for non-parametric significance tests, whereas Prathap’s systems viewignores this shape and uses averages on the assumption of the Central Limit Theorem(Gla¨nzel 2010). However, citation distributions are extremely skewed (Seglen 1992, 1997;cf. Leydesdorff 2008) and central tendency statistics give misleading results. Usingparametric statistics, one can neither reliably test the significance of observations nor thesignificance of differences in rankings.Prathap (2011a) was able to compute using the mean values of JCS (journal citationscores) and FCS (field citation scores) because his concept of entropy is no longer prob-abilistic entropy (cf. Leydesdorff 1995; Theil 1972), but thermodynamic entropy (Prathap2011b, p. 523f). However, the impact of two hits is not their average, but their sum. In thecase of collisions, this is the vector sum of the momenta. We agree that in the case ofcitations one should use a scalar sum.The scalar sum of citations (that is, total citations) would as yet be insufficientlyqualified. The quality along the skewed citation curve must first be normalized in terms ofpercentiles. Bornmann and Mutz (2011) normalized in terms of six percentile-rank classes,but the more general case is normalization in terms of quantiles as a continuous variablewhich can thereafter be organized using different evaluation schemes (Leydesdorff et al.2011; Leydesdorff and Bornmann 2011a; Rousseau 2011). Different aggregations arepossible because the impacts, once normalized in terms of percentiles, are determined atthe paper level. This Integrated Impact Indicator (I3) can be formalized as an integration asfollows:I3 ¼Xixi  f ðxiÞ ð1ÞCitations are discrete events and therefore the integral is in this case a step function:using Eq. 1, the frequency of papers in each percentile (xi) is multiplied by the percentile ofeach paper (f(xi)). The resulting scalar (R) of the total impact can then be scaled (i) in termsof various evaluation schemes (e.g., quartiles, or the six evaluation categories used in theU.S. Science and Engineering Indicators NSB (2010) and by Bornmann and Mutz (2011));(ii) tested for their significance against a theoretically specified expectation; (iii) expressedas a single number, namely a percentage of total impact contained in the reference set; and(iv) used to compare among and between various units of analysis such as journals,countries, institutes, and cities; by aggregating cases in a statistically controllable way(Theil 1972).In summary, the discussion over Rates of Averages versus Averages of Rates (Gingrasand Larivie`re 2011) has taught us that a rate of averages is merely a quotient number thatdoes not allow for testing, and is mathematically inconsistent (Waltman et al. 2011). Themean observed citation ratio (MOCR) should not be divided by the mean expected citationratio (RCR = MOCR/MECR; Schubert and Braun 1986; cf. Gla¨nzel et al. 2009, p. 182),but observed values can be tested against expected values by using appropriate statistics.Secondly, citation indicators based on averaging skewed distributions—such asPrathap’s EEE and the new ‘‘crown indicator’’ MNCS—are unreliable. For example,Leydesdorff et al. (2011) have shown that in the case of seven Principal Investigators at theAcademic Medical Center of the University of Amsterdam, the number one ranked PI1 The software for measuring this indicator is available at http://www.leydesdorff.net/software/i3.746 L. Leydesdorff, T. Opthof123would fall to fifth position, whereas the sixth-ranked PI would become the highest-rankedauthor if percentiles or percentile ranks are used.Thirdly, one should not test sets of documents as independent samples against eachother, but as subsets of a reference set (Bornmann et al. 2008): each subset contributes apercentage impact to the set. The reference set allows for normalization and the specifi-cation of an expectation. (This specification can further be informed on theoreticalgrounds.) Using quantiles and percentile ranks, the observed values can be tested againstthe expected ones using non-parametric statistics.Furthermore, and not specific as criticism of EEE, field delineations do not have to bebased on ex ante classification schemes such as the ISI Subject Categories. Hitherto,journal classifications have been unprecise and unreliable (Boyack and Klavans 2011;Leydesdorff 2006; Pudovkin and Garfield 2002; Rafols and Leydesdorff 2009). Fractionalattribution of citations in the citing documents, however, can be used for normalization ofdifferences in citation potentials (Garfield 1979) reflecting differences in citation behaviorat the level of individual papers (Leydesdorff and Bornmann 2011b; Leydesdorff andOpthof 2010; Moed 2010).Given these recent improvements in citation normalization—such as the use of paper-based measures both cited and citing—the theoretical question remains whether citationscan be used as indicators of scientific quality, and if so, when? (Amsterdamska andLeydesdorff 1989; Bornmann et al. 2008; Garfield 1979; Leydesdorff 1998; Leydesdorffand Amsterdamska 1990). Opthof and Leydesdorff (2011) opened this discussion byasking whether citation analysis enables us to legitimate the strategic selection of‘‘excellent’’ as against merely ‘‘good’’ research?Open Access This article is distributed under the terms of the Creative Commons Attribution Noncom-mercial License which permits any noncommercial use, distribution, and reproduction in any medium,provided the original author(s) and source are credited.ReferencesAmsterdamska, O., & Leydesdorff, L. (1989). Citations: Indicators of significance? Scientometrics, 15(5–6),449–471.Bornmann, L., & Mutz, R. (2011). Further steps towards an ideal method of measuring citation performance:The avoidance of citation (ratio) averages in field-normalization. Journal of Informetrics, 5(1),228–230.Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H. D. (2008). Citation counts for research evaluation:Standards of good practice for analyzing bibliometric data and presenting and interpreting results.Ethics in Science and Environmental Politics, 8(1), 93–102.Boyack, K. W., & Klavans, R. (2011). Multiple dimensions of journal specificity: Why journals can’t beassigned to disciplines. In E. Noyons, P. Ngulube & J. Leta (Eds.), The 13th Conference of theInternational Society for Scientometrics and Informetrics (Vol. I, pp. 123–133). Durban, South Africa:ISSI, Leiden University and the University of Zululand.Garfield, E. (1979). Is citation analysis a legitimate evaluation tool? Scientometrics, 1(4), 359–375.Gingras, Y., & Larivie`re, V. (2011). There are neither ‘‘king’’ nor ‘‘crown’’ in scientometrics: Comments ona supposed ‘‘alternative’’ method of normalization. Journal of Informetrics, 5(1), 226–227.Gla¨nzel, W. (2010). On reliability and robustness of scientometrics indicators based on stochastic models.An evidence-based opinion paper. Journal of Informetrics, 4(3), 313–319.Gla¨nzel, W., Thijs, B., Schubert, A., & Debackere, K. (2009). Subfield-specific normalized relative indi-cators and a new generation of relational charts: Methodological foundations illustrated on theassessment of institutional research performance. Scientometrics, 78(1), 165–188.Leydesdorff, L. (1995). The challenge of scientometrics: The development, measurement, and self-organi-zation of scientific communications. Leiden: DSWO Press, Leiden University. Retrieved fromA rejoinder on energy versus impact indicators 747123http://www.universal-publishers.com/book.php?method=ISBN&book=1581126816. Accessed 12 Sep2011.Leydesdorff, L. (1998). Theories of citation? Scientometrics, 43(1), 5–25.Leydesdorff, L. (2006). Can scientific journals be classified in terms of aggregated journal–journal citationrelations using the journal citation reports? Journal of the American Society for Information Scienceand Technology, 57(5), 601–613.Leydesdorff, L. (2008). Caveats for the use of citation indicators in research and journal evaluation. Journalof the American Society for Information Science and Technology, 59(2), 278–287.Leydesdorff, L., & Amsterdamska, O. (1990). Dimensions of citation analysis. Science, Technology andHuman Values, 15(3), 305–335.Leydesdorff, L., & Bornmann, L. (2011a). Integrated Impact Indicators (I3) compared with Impact Factors(IFs): An alternative design with policy implications. Journal of the American Society for InformationScience and Technology (in press).Leydesdorff, L., & Bornmann, L. (2011b). How fractional counting affects the Impact Factor: Normalizationin terms of differences in citation potentials among fields of science. Journal of the American Societyfor Information Science and Technology, 62(2), 217–229.Leydesdorff, L., & Opthof, T. (2010). Scopus’ Source Normalized Impact per Paper (SNIP) versus theJournal Impact Factor based on fractional counting of citations. Journal of the American Society forInformation Science and Technology, 61(11), 2365–2396.Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the tables in citation analysis onemore time: Principles for comparing sets of documents. Journal of the American Society for Infor-mation Science and Technology, 62(7), 1370–1381.Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics,4(3), 265–277.National Science Board. (2010). Science and engineering indicators. Washington, DC: National ScienceFoundation. Retrieved from http://www.nsf.gov/statistics/seind10/. Accessed 12 Sep 2011.Opthof, T., & Leydesdorff, L. (2010). Caveats for the journal and field normalizations in the CWTS(‘‘Leiden’’) evaluations of research performance. Journal of Informetrics, 4(3), 423–430.Opthof, T., & Leydesdorff, L. (2011). A comment to the paper by Waltman et al., Scientometrics, 87,467–481, 2011. Scientometrics, 88(3), 1011–1016.Prathap, G. (2011a). A comment to the papers by Opthof and Leydesdorff, Scientometrics, 88, 1011–1016,2011 and Waltman et al., Scientometrics, 88, 1017–1022, 2011. Scientometrics. doi: 10.1007/s11192-011-0500-0.Prathap, G. (2011b). The Energy–Exergy–Entropy (or EEE) sequences in bibliometric assessment.Scientometrics, 87(3), 515–524.Pudovkin, A. I., & Garfield, E. (2002). Algorithmic procedure for finding semantically related journals.Journal of the American Society for Information Science and Technology, 53(13), 1113–1119.Rafols, I., & Leydesdorff, L. (2009). Content-based and algorithmic classifications of journals: Perspectiveson the dynamics of scientific communication and indexer effects. Journal of the American Society forInformation Science and Technology, 60(9), 1823–1835.Rousseau, R. (2011). Percentile rank scores are congruous indicators of relative performance, or aren’t they?Retrieved from http://arxiv.org/abs/1108.1860. Accessed 12 Sep 2011.Schubert, A., & Braun, T. (1986). Relative indicators and relational charts for comparative assessment ofpublication output and citation impact. Scientometrics, 9(5), 281–291.Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science,43(9), 628–638.Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BritishMedical Journal, 314, 498–502.Theil, H. (1972). Statistical decomposition analysis. Amsterdam: North-Holland.van Raan, A. F. J. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and withpeer judgment for 147 chemistry research groups. Scientometrics, 67(3), 491–502.van Raan, A. F. J., van Leeuwen, T. N., Visser, M. S., van Eck, N. J., & Waltman, L. (2010). Rivals for thecrown: Reply to Opthof and Leydesdorff. Journal of Informetrics, 4(3), 431–435.Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. J. (2011). Towards a newcrown indicator: Some theoretical considerations. Journal of Informetrics, 5(1), 37–47.748 L. Leydesdorff, T. Opthof123

Scientometrics

A rejoinder on energy versus impact indicators

Springer - Publisher Connector

Leydesdorff, L.

Opthof, T.

Search4Dev

Citation distributions are so skewed that using the mean or any other central tendency measure is ill-advised. Unlike G. Prathap’s scalar measures (Energy, Exergy, and Entropy or EEE), the Integrated Impact Indicator (I3) is based on non-parametric statistics using the (100) percentiles of the distribution. Observed values can be tested against expected ones; impact can be qualified at the article level and then aggregated

International Migration, Integration and Social Cohesion online publications

AKJournals (Akadémiai Kiadó)

A rejoinder on energy versus impact indicators
Loet Leydesdorff • Tobias Opthof
Received: 26 August 2011 / Published online: 20 September 2011
 The Author(s) 2011. This article is published with open access at Springerlink.com
Abstract Citation distributions are so skewed that using the mean or any other central
tendency measure is ill-advised. Unlike G. Prathap’s scalar measures (Energy, Exergy, and
Entropy or EEE), the Integrated Impact Indicator (I3) is based on non-parametric statistics
using the (100) percentiles of the distribution. Observed values can be tested against
expected ones; impact can be qualified at the article level and then aggregated.
Keywords Citation  Scalar  Integration  EEE  I3  Quality
Prathap (2011a) in his Letter applies newly developed scalar measures for bibliometrics
(Energy, Exergy, and Entropy; EEE) to the data provided in Table 1 of Van Raan (2006,
p. 495). EEE operates on averages and ignores the shape of the underlying distributions of
citations (‘‘the citation curves’’). (Let us note about EEE that energy and exergy share
dimensionality, but entropy is expressed in Watts/Kelvin. Thus, the expression Energy -
Exergy = Entropy as suggested by Prathap (2011b) is, in our opinion, invalid without the
specification of a meta-physical analog on of the ‘‘temperature.’’)
Like Prathap (2011b) and following Bornmann and Mutz (2011), Leydesdorff et al.
(2011) have elaborated the percentile-rank as a scalar sum by using the same dataset that
led to the original contention about how citation data should be normalized (Opthof and
L. Leydesdorff (&)
Amsterdam School of Communication Research (ASCoR), University of Amsterdam,
Kloveniersburgwal 48, 1012 CX Amsterdam, The Netherlands
e-mail: loet@leydesdorff.net
T. Opthof
Experimental Cardiology Group, Heart Failure Research Center, Academic Medical Center AMC,
Meibergdreef 9, 1105 AZ Amsterdam, The Netherlands
e-mail: t.opthof@inter.nl.net
T. Opthof
Department of Medical Physiology, University Medical Center, Utrecht, The Netherlands
123
Scientometrics (2012) 90:745–748
DOI 10.1007/s11192-011-0502-y
Leydesdorff 2010; Van Raan et al. 2010). More recently, Leydesdorff and Bornmann
(2011) have developed this scalar measure into the Integrated Impact Indicator (I3).1
The difference between I3 and EEE is that I3 takes the shapes of the distribution into
account and allows for non-parametric significance tests, whereas Prathap’s systems view
ignores this shape and uses averages on the assumption of the Central Limit Theorem
(Gla¨nzel 2010). However, citation distributions are extremely skewed (Seglen 1992, 1997;
cf. Leydesdorff 2008) and central tendency statistics give misleading results. Using
parametric statistics, one can neither reliably test the significance of observations nor the
significance of differences in rankings.
Prathap (2011a) was able to compute using the mean values of JCS (journal citation
scores) and FCS (field citation scores) because his concept of entropy is no longer prob-
abilistic entropy (cf. Leydesdorff 1995; Theil 1972), but thermodynamic entropy (Prathap
2011b, p. 523f). However, the impact of two hits is not their average, but their sum. In the
case of collisions, this is the vector sum of the momenta. We agree that in the case of
citations one should use a scalar sum.
The scalar sum of citations (that is, total citations) would as yet be insufficiently
qualified. The quality along the skewed citation curve must first be normalized in terms of
percentiles. Bornmann and Mutz (2011) normalized in terms of six percentile-rank classes,
but the more general case is normalization in terms of quantiles as a continuous variable
which can thereafter be organized using different evaluation schemes (Leydesdorff et al.
2011; Leydesdorff and Bornmann 2011a; Rousseau 2011). Different aggregations are
possible because the impacts, once normalized in terms of percentiles, are determined at
the paper level. This Integrated Impact Indicator (I3) can be formalized as an integration as
follows:
I3 ¼
X
i
xi  f ðxiÞ ð1Þ
Citations are discrete events and therefore the integral is in this case a step function:
using Eq. 1, the frequency of papers in each percentile (xi) is multiplied by the percentile of
each paper (f(xi)). The resulting scalar (R) of the total impact can then be scaled (i) in terms
of various evaluation schemes (e.g., quartiles, or the six evaluation categories used in the
U.S. Science and Engineering Indicators NSB (2010) and by Bornmann and Mutz (2011));
(ii) tested for their significance against a theoretically specified expectation; (iii) expressed
as a single number, namely a percentage of total impact contained in the reference set; and
(iv) used to compare among and between various units of analysis such as journals,
countries, institutes, and cities; by aggregating cases in a statistically controllable way
(Theil 1972).
In summary, the discussion over Rates of Averages versus Averages of Rates (Gingras
and Larivie`re 2011) has taught us that a rate of averages is merely a quotient number that
does not allow for testing, and is mathematically inconsistent (Waltman et al. 2011). The
mean observed citation ratio (MOCR) should not be divided by the mean expected citation
ratio (RCR = MOCR/MECR; Schubert and Braun 1986; cf. Gla¨nzel et al. 2009, p. 182),
but observed values can be tested against expected values by using appropriate statistics.
Secondly, citation indicators based on averaging skewed distributions—such as
Prathap’s EEE and the new ‘‘crown indicator’’ MNCS—are unreliable. For example,
Leydesdorff et al. (2011) have shown that in the case of seven Principal Investigators at the
Academic Medical Center of the University of Amsterdam, the number one ranked PI
1 The software for measuring this indicator is available at http://www.leydesdorff.net/software/i3.
746 L. Leydesdorff, T. Opthof
123
would fall to fifth position, whereas the sixth-ranked PI would become the highest-ranked
author if percentiles or percentile ranks are used.
Thirdly, one should not test sets of documents as independent samples against each
other, but as subsets of a reference set (Bornmann et al. 2008): each subset contributes a
percentage impact to the set. The reference set allows for normalization and the specifi-
cation of an expectation. (This specification can further be informed on theoretical
grounds.) Using quantiles and percentile ranks, the observed values can be tested against
the expected ones using non-parametric statistics.
Furthermore, and not specific as criticism of EEE, field delineations do not have to be
based on ex ante classification schemes such as the ISI Subject Categories. Hitherto,
journal classifications have been unprecise and unreliable (Boyack and Klavans 2011;
Leydesdorff 2006; Pudovkin and Garfield 2002; Rafols and Leydesdorff 2009). Fractional
attribution of citations in the citing documents, however, can be used for normalization of
differences in citation potentials (Garfield 1979) reflecting differences in citation behavior
at the level of individual papers (Leydesdorff and Bornmann 2011b; Leydesdorff and
Opthof 2010; Moed 2010).
Given these recent improvements in citation normalization—such as the use of paper-
based measures both cited and citing—the theoretical question remains whether citations
can be used as indicators of scientific quality, and if so, when? (Amsterdamska and
Leydesdorff 1989; Bornmann et al. 2008; Garfield 1979; Leydesdorff 1998; Leydesdorff
and Amsterdamska 1990). Opthof and Leydesdorff (2011) opened this discussion by
asking whether citation analysis enables us to legitimate the strategic selection of
‘‘excellent’’ as against merely ‘‘good’’ research?
Open Access This article is distributed under the terms of the Creative Commons Attribution Noncom-
mercial License which permits any noncommercial use, distribution, and reproduction in any medium,
provided the original author(s) and source are credited.
References
Amsterdamska, O., & Leydesdorff, L. (1989). Citations: Indicators of significance? Scientometrics, 15(5–6),
449–471.
Bornmann, L., & Mutz, R. (2011). Further steps towards an ideal method of measuring citation performance:
The avoidance of citation (ratio) averages in field-normalization. Journal of Informetrics, 5(1),
228–230.
Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H. D. (2008). Citation counts for research evaluation:
Standards of good practice for analyzing bibliometric data and presenting and interpreting results.
Ethics in Science and Environmental Politics, 8(1), 93–102.
Boyack, K. W., & Klavans, R. (2011). Multiple dimensions of journal specificity: Why journals can’t be
assigned to disciplines. In E. Noyons, P. Ngulube & J. Leta (Eds.), The 13th Conference of the
International Society for Scientometrics and Informetrics (Vol. I, pp. 123–133). Durban, South Africa:
ISSI, Leiden University and the University of Zululand.
Garfield, E. (1979). Is citation analysis a legitimate evaluation tool? Scientometrics, 1(4), 359–375.
Gingras, Y., & Larivie`re, V. (2011). There are neither ‘‘king’’ nor ‘‘crown’’ in scientometrics: Comments on
a supposed ‘‘alternative’’ method of normalization. Journal of Informetrics, 5(1), 226–227.
Gla¨nzel, W. (2010). On reliability and robustness of scientometrics indicators based on stochastic models.
An evidence-based opinion paper. Journal of Informetrics, 4(3), 313–319.
Gla¨nzel, W., Thijs, B., Schubert, A., & Debackere, K. (2009). Subfield-specific normalized relative indi-
cators and a new generation of relational charts: Methodological foundations illustrated on the
assessment of institutional research performance. Scientometrics, 78(1), 165–188.
Leydesdorff, L. (1995). The challenge of scientometrics: The development, measurement, and self-organi-
zation of scientific communications. Leiden: DSWO Press, Leiden University. Retrieved from
A rejoinder on energy versus impact indicators 747
123
http://www.universal-publishers.com/book.php?method=ISBN&book=1581126816. Accessed 12 Sep
2011.
Leydesdorff, L. (1998). Theories of citation? Scientometrics, 43(1), 5–25.
Leydesdorff, L. (2006). Can scientific journals be classified in terms of aggregated journal–journal citation
relations using the journal citation reports? Journal of the American Society for Information Science
and Technology, 57(5), 601–613.
Leydesdorff, L. (2008). Caveats for the use of citation indicators in research and journal evaluation. Journal
of the American Society for Information Science and Technology, 59(2), 278–287.
Leydesdorff, L., & Amsterdamska, O. (1990). Dimensions of citation analysis. Science, Technology and
Human Values, 15(3), 305–335.
Leydesdorff, L., & Bornmann, L. (2011a). Integrated Impact Indicators (I3) compared with Impact Factors
(IFs): An alternative design with policy implications. Journal of the American Society for Information
Science and Technology (in press).
Leydesdorff, L., & Bornmann, L. (2011b). How fractional counting affects the Impact Factor: Normalization
in terms of differences in citation potentials among fields of science. Journal of the American Society
for Information Science and Technology, 62(2), 217–229.
Leydesdorff, L., & Opthof, T. (2010). Scopus’ Source Normalized Impact per Paper (SNIP) versus the
Journal Impact Factor based on fractional counting of citations. Journal of the American Society for
Information Science and Technology, 61(11), 2365–2396.
Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the tables in citation analysis one
more time: Principles for comparing sets of documents. Journal of the American Society for Infor-
mation Science and Technology, 62(7), 1370–1381.
Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics,
4(3), 265–277.
National Science Board. (2010). Science and engineering indicators. Washington, DC: National Science
Foundation. Retrieved from http://www.nsf.gov/statistics/seind10/. Accessed 12 Sep 2011.
Opthof, T., & Leydesdorff, L. (2010). Caveats for the journal and field normalizations in the CWTS
(‘‘Leiden’’) evaluations of research performance. Journal of Informetrics, 4(3), 423–430.
Opthof, T., & Leydesdorff, L. (2011). A comment to the paper by Waltman et al., Scientometrics, 87,
467–481, 2011. Scientometrics, 88(3), 1011–1016.
Prathap, G. (2011a). A comment to the papers by Opthof and Leydesdorff, Scientometrics, 88, 1011–1016,
2011 and Waltman et al., Scientometrics, 88, 1017–1022, 2011. Scientometrics. doi: 10.1007/
s11192-011-0500-0.
Prathap, G. (2011b). The Energy–Exergy–Entropy (or EEE) sequences in bibliometric assessment.
Scientometrics, 87(3), 515–524.
Pudovkin, A. I., & Garfield, E. (2002). Algorithmic procedure for finding semantically related journals.
Journal of the American Society for Information Science and Technology, 53(13), 1113–1119.
Rafols, I., & Leydesdorff, L. (2009). Content-based and algorithmic classifications of journals: Perspectives
on the dynamics of scientific communication and indexer effects. Journal of the American Society for
Information Science and Technology, 60(9), 1823–1835.
Rousseau, R. (2011). Percentile rank scores are congruous indicators of relative performance, or aren’t they?
Retrieved from http://arxiv.org/abs/1108.1860. Accessed 12 Sep 2011.
Schubert, A., & Braun, T. (1986). Relative indicators and relational charts for comparative assessment of
publication output and citation impact. Scientometrics, 9(5), 281–291.
Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science,
43(9), 628–638.
Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British
Medical Journal, 314, 498–502.
Theil, H. (1972). Statistical decomposition analysis. Amsterdam: North-Holland.
van Raan, A. F. J. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with
peer judgment for 147 chemistry research groups. Scientometrics, 67(3), 491–502.
van Raan, A. F. J., van Leeuwen, T. N., Visser, M. S., van Eck, N. J., & Waltman, L. (2010). Rivals for the
crown: Reply to Opthof and Leydesdorff. Journal of Informetrics, 4(3), 431–435.
Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. J. (2011). Towards a new
crown indicator: Some theoretical considerations. Journal of Informetrics, 5(1), 37–47.
748 L. Leydesdorff, T. Opthof
123


NARCIS 

A comment to the paper by Waltman et al.,

A comment to the papers by Opthof and Leydesdorff,

Algorithmic procedure for ﬁnding semantically related journals.

Can scientiﬁc journals be classiﬁed in terms of aggregated journal–journal citation relations using the journal citation reports?

Caveats for the journal and ﬁeld normalizations in the CWTS (‘‘Leiden’’) evaluations of research performance.

Caveats for the use of citation indicators in research and journal evaluation.

Citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results.

Citations: Indicators of signiﬁcance?

Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups.

Content-based and algorithmic classiﬁcations of journals: Perspectives on the dynamics of scientiﬁc communication and indexer effects.

Dimensions of citation analysis.

Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in ﬁeld-normalization.

How fractional counting affects the Impact Factor: Normalization in terms of differences in citation potentials among ﬁelds of science.

Integrated Impact Indicators (I3) compared with Impact Factors (IFs): An alternative design with policy implications.

Is citation analysis a legitimate evaluation tool?

Measuring contextual citation impact of scientiﬁc journals.

Multiple dimensions of journal speciﬁcity: Why journals can’t be assigned to disciplines. In

On reliability and robustness of scientometrics indicators based on stochastic models. An evidence-based opinion paper.

Percentile rank scores are congruous indicators of relative performance, or aren’t they? Retrieved from http://arxiv.org/abs/1108.1860.

Relative indicators and relational charts for comparative assessment of publication output and citation impact.

Rivals for the crown: Reply to Opthof and Leydesdorff.

Science and engineering indicators.

Scopus’ Source Normalized Impact per Paper (SNIP) versus the Journal Impact Factor based on fractional counting of citations.

Statistical decomposition analysis.

Subﬁeld-speciﬁc normalized relative indicators and a new generation of relational charts: Methodological foundations illustrated on the assessment of institutional research performance.

The challenge of scientometrics: The development, measurement, and self-organization of scientiﬁc communications.

The Energy–Exergy–Entropy (or EEE) sequences in bibliometric assessment.

The skewness of science.

Theories of citation?

There are neither ‘‘king’’ nor ‘‘crown’’ in scientometrics: Comments on a supposed ‘‘alternative’’ method of normalization.

Towards a new crown indicator: Some theoretical considerations.

Turning the tables in citation analysis one more time: Principles for comparing sets of documents.

Why the impact factor of journals should not be used for evaluating research.

A Rejoinder on Energy versus Impact Indicators

A Rejoinder on Energy versus Impact Indicators

Abstract

Similar works

Full text

Available Versions

Crossref

Springer - Publisher Connector

Search4Dev

International Migration, Integration and Social Cohesion online publications

AKJournals (Akadémiai Kiadó)

Springer - Publisher Connector

NARCIS

International Migration, Integration and Social Cohesion online publications