The quality of publications, approximated by the containing journal's quality indicator, is often the basis for hire and promotion in academic and research positions. Over the years a handful of ranking methods have been proposed. Discussing the most prominent methods we show that they are inherently biased against journals publishing short papers.quality ranking, paper length, impact factor, invariant method, LP method

Alexandru Nichifor

László Á. Kóczy

Martin Strobel

English

Research Papers in Economics

Article length bias in journal rankings
L aszl o  A. K oczyy Alexandru Nichiforz Martin Strobelx
Abstract
The quality of publications, approximated by the containing jour-
nal's quality indicator, is often the basis for hire and promotion in
academic and research positions. Over the years a handful of ranking
methods have been proposed. Discussing the most prominent methods
we show that they are inherently biased against journals publishing
short papers.
Keywords: quality ranking, paper length, impact factor, invariant
method, LP method
1 Introduction
The last decades saw an explosion in the number of journals. Libraries
face higher subscription fees and must allocate budgets in an ecient way.
Researchers nd it more and more dicult to keep up with the growing
literature even in narrow elds. Funding and promotion decisions are often
taken considering researchers' publications which are becoming increasingly
dicult to evaluate and compare. All these developments contribute to the
growing interest in objective measures of journal quality.
The ranking of journals has long been dominated by the Impact Factor
(Gareld, 1955) although numerous alternative measures have been intro-
duced, such as the invariant method (Pinski and Narin, 1976; Palacios-Huerta
We thank C a gatay Kay and Bettina Klaus for helpful comments. K oczy thanks
funding from OTKA under grant NF-72610; Nichifor thanks the Netherlands Organisation
for Scientic Research (NWO) for its support under grant VIDI-452-06-013; Strobel thanks
the funding by METEOR.
yKeleti Faculty of Economics, Budapest Tech, Tavaszmez} o u. 15-17., H-1084 Budapest
and Maastricht University. Email: koczy.laszlo@kgk.bmf.hu.
zDepartment of Economics, Maastricht University, P.O.Box 616, NL-6200MD Maas-
tricht. Email: a.nichifor@algec.unimaas.nl.
xDepartment of Economics, Maastricht University, P.O.Box 616, NL-6200MD Maas-
tricht. Emails: m.strobel@algec.unimaas.nl.
1and Volij, 2004), the LP method (Liebowitz and Palmer, 1984; Laband and
Piette, 1994; Kalaitzidakis, Mamuneas, and Stengos, 2003) and the H-index
(Hirsch, 2004; Braun, Gl anzel, and Schubert, 2005). These methods gener-
ated many variations, the best known being the PageRank (Brin and Page,
1998) algorithm which plays an important part in how Google ranks web
sites.
While most of these methods have been intuitively motivated, if at all,
Palacios-Huerta and Volij (2004) presented an axiomatization for the in-
variant method from a descriptive perspective. That is, they nd a set of
requirements that fully characterizes the invariant method.
Some of these properties are invariance properties that allow the compar-
ison of journals with dierent characteristics and from dierent elds. Yet,
a property that aects also journals rankings within the same discipline, in-
variance to article-splitting, requiring neutrality with respect to the length of
articles, has not been studied. We introduce this property and show that the
ranking methods above are not invariant to article-splitting, but are biased
against journals publishing short articles, exposition notes or letters.
2 Ranking Methods
Let J be the universe of journals and J be the set of all non-empty nite
subsets of J. Let J 2 J, j 2 J and let aj denote the number of articles
published in j. For each i;j 2 J, cij represents the number of citations to
journal i by journal j, that is, the number of references made by journal j to
journal i. Let us consider the citation matrix a jJjjJj nonnegative matrix,
C = fcijgi;j2J and let cj =
P
i2J cij be the total number of citations made
by j.
Denition 1. A ranking problem is a triple (J;a;C) consisting of a nite
set of journals J 2 J, a vector a 2 NjJj containing the number of published
articles and a citation matrix C = fcijgi;j2J.
Let R denote the set of ranking problems. A cardinal ranking of the
journals in J is given by a transposed valuations vector vT = (v1;v2;:::;vjJj),
where vi > 0.
Denition 2. A ranking method  maps a ranking problem (J;a;C) 2 R
to a vector of valuations v 2 R
jJj
++.
A ranking method induces a weak ordering of the journals via the ranking
rule i < j if and only if vi  vj. Ties, i.e., i < j and j < i, are allowed, but
only occur if vi = vj.
2Denition 3. The impact factor (Gareld, 1955) ranks according to IFj = P
i2J cji
aj , where vector a and matrix C contain data for a two-year period.
Before we present the next two ranking methods, we need an additional
assumption and some extra notation. We require the citation matrix C to
be irreducible: there is no partition of the set of journals J in two sets J1
and J2 such that [there are no inter partition citations] or [all inter partition
citations are unidirectional, say from journals in J2 to journals in J1]. Let A
and DC be diagonal matrices with fajgj2J and fcjgj2J along the diagonal.
Denition 4. The invariant method (Pinski and Narin, 1976) ranks accord-
ing to the vector v that solves
A
 1CD
 1
C Av = v: (1)
Denition 5. The LP method (Liebowitz and Palmer, 1984) ranks according
to the vector v that solves
A
 1Cv = v: (2)
It is known that both the invariant method and the LP method are well
dened for irreducible matrices (Palacios-Huerta and Volij, 2004).
3 Article-Splitting
Next we introduce a basic desirable property of a ranking method: invariance
to article-splitting. When we refer to an article's citations, we consider both
citations sent (that we interpret as an article's \consumption") and received
(that we interpret as an article's \production" which was consumed by other
articles). We do not measure the length of a paper in terms of pages or
characters. Length is indicated by the number of citations sent and received,
which, we assume are both evenly spread in a journal. In particular, we
assume that if a paper is split into k sub-papers, the citations are split
evenly among them.1 Our invariance property requires that having the split
ex-post or ex-ante should make no dierence. That is, splitting the citations
between the sections of a paper, or publishing sub-papers, leave the ranking
unchanged.
Formally, let j 2 R, j > 1, be a split factor and consider two ranking
problems f(J;a;C), (J;a0;C)g  R where for some journal j 2 J, a0
j = jaj
and for all other journals i 6= j, a0
i = ai.
1Conversely, if a set of k articles with no overlapping citations and published in the
same journal are merged into a single paper, like a survey or a book, then the resulting
publication collects all citations.
3Denition 6. A ranking method  is invariant to article-splitting if
(J;a;C) = (J;a
0;C):
Observe that the citation matrix is not aected: the number of citations does
not change, they are only distributed among more papers. The property thus
relates ranking problems implicating journals having similar citation patterns
but publishing dierent numbers of articles.
Denition 7. A ranking method  is biased against article-splitting if
j(J;a;C) > j(J;a
0;C):
In the following we show that none of the listed ranking methods is invariant
to article-splitting.
Proposition 1. The impact factor is not invariant to article-splitting.
Proof. Given journal j has a valuation IFj for problem (J;a;C) covering data
for the past two years, consider a modication of the problem where some of
j's articles are split and hence a0
j = jaj with j > 1, while a0
i = ai for all
i 6= j. Although this leaves the numerator unaected, the number of articles
goes up. IF
0
j =
aj
a0
jIFj = 1
jIFj, thus the impact factor decreases.
Proposition 2. The invariant method is not invariant to article-splitting.
Proof. Let vj be the valuation of journal j for problem (J;a;C). Then v
solves Equation 1. Now consider a modication of the problem where some
of j's articles are split and hence a0
j = jaj with j > 1, while a0
i = ai for
all i 6= j. Then v0 dened as v0
j = 1
jvj and v0
i = vi for i 6= j is the solution
of Equation 1, that is A0 1CD
 1
C A0v0 = v0. In order to see this, pre-multiply
by A0 and observe that A0v0 = Av. Finally observe that as j > 0, v0
j < vj,
while the valuations of other journals have not changed.
Proposition 3. The LP method is not invariant to article-splitting.
Proof. Using a similar technique as Roy, Saberi, and Wan (2008), we show
that an increase (decrease) in the number of articles of a journal will decrease
(increase) its valuation. All else equal, let journal j increase the number of
its articles per issue from aj to a0
j, i.e., a0
j = jaj with j > 1, while for all
i 6= j, a0
i = ai. Thus, Equation 2 becomes:
A
0 1Cv
0 = v
0 (3)
4Note that Equations 2 and 3 are algebraic eigenvalue problems. Let
  = A 1C and  0 = A0 1C. Let %( ) denote the spectral radius of   and let
v denote the eigenvectors associated with %( ). Similarly, we use %( 0),  0,
v0. Since the matrix   is irreducible,  0 is also irreducible and Equation 3 is
well dened. Using the newly introduced notation, we rewrite Equations 2
and 3 as  v = %( )v and  0v0 = %( 0)v0. Observe that   and  0 are obtained
from the same citation matrix C which is pre-multiplied by A 1 respectively
A0 1 and that for all i 2 J, a0
i  ai. Hence,  0 is weakly smaller in every
entry than   and therefore %( ) = %( 0) +  for some  > 0.
We scale v0 such that v0
j = vj and rewrite v0T as vT = vT   xT = [v1  
x1;:::;vj xj;:::;vi xi] where x 2 RjJj, such that xj = 0. Thus,  0v = %( 0)v
and replacing v and %( 0):
 
0v    
0x = %( )v   v   %( 
0)x (4)
Let v j, v j, x j be the valuation vectors except for journal j. Similarly,
let   j and  0
 j be the matrices   and  0 where we removed row and column
j and note that   j =  0
 j. Dropping the j'th equation from the system of
Equations 4 and using that  v = %( )v we obtain:
  jx j = v j + %( 
0)x j: (5)
which can be rearranged to (%( 0)I     j)x j =  v j. Minc (1988, Theo-
rem 5.1 and Theorem 5.2, Chapter 1) shows that the spectral radius of an
irreducible matrix is greater than the spectral radius of any of its subma-
trices. Hence, %( 0)  %( 0
 j) = %(  j). The matrix M = (%( 0)I     j)
is an M-matrix. Therefore, M 1 is positive. (For a reminder of M-matrices
and their properties see Minc, 1988, Denition 4.1, Theorem 4.5, Chapter 6).
Observing that v j is positive, x j has to be negative. Hence, vj = vj and
for all i 6= j, vi > vi. Rescaling v to v0, we have
v0
j
v0
i <
vj
vi.
Theorem 1. The impact factor, the invariant method and the LP method
are biased against article-splitting.
Proof. We have shown that for the impact factor and for the invariant method
the valuation of a journal j whose articles are split into j decreases by a
factor of 1
j. Note that for an appropriate choice of j it can be arbitrarily low.
In particular, it can be lower than the valuation of the journal ranked next,
thus changing also the relative order of the journals. For the LP method, we
have shown that an increase in the number of articles of journal j decreases
its relative weight that can lead to the same eect.
5The following example shows that the bias against article-splitting aect-
ing the ranking methods might also induce changes in the relative ordering
of the journals:
Example 1. Let J = fj1;j2;j3g, a = (2;2;3), a0 = (4;2;3) and dene C as:
C =
0
@
12 8 4
6 10 2
3 3 9
1
A
Let R = (J;a;C) and R0 = (J;a0;C) be two ranking problems and note that
the only dierence between R and R0 is that a0
1 = 1a1, with 1 = 2. For
each problem R and R0, the following table presents the vector of valuations
produced by each of the ranking methods investigated.2
IF(R) IF(R0) IM(R) IM(R0) LP(R) LP(R)
j1 0:46 0:30 0:48 0:31 0:50 0:29
j2 0:35 0:45 0:33 0:44 0:37 0:51
j3 0:19 0:25 0:19 0:25 0:13 0:20
Table 1: Article-splitting bias in ranking methods inducing changes in the
relative ordering of the journals.
Note that for R, all ranking methods induce the ordering j1  j2  j3, while
for R0, all ranking methods induce the ordering j2  j1  j3.
4 Conclusion
We nd that the most widely used methods to rank academic journals share
a common drawback as they cannot distinguish quality from quantity at the
paper level, which might create a bias against journals with briefer communi-
cations. One must therefore use and interpret those rankings with care when
estimating the quality of journals.
Not all ranking methods are subject to such bias. In the case of the H-
index (Hirsch, 2004; Braun, Gl anzel, and Schubert, 2005) it is easy to see
that article-splitting has an ambiguous eect, while the tournament method
(K oczy and Strobel, 2007) is by denition invariant to article-splitting. Fur-
ther work is due to understand more about the properties and interpretations
of dierent ranking methods.
2Since in this example we are interested only in relative values, for each method we
have normalized the vector of valuations so that they add up to 1.
6References
Braun, T., W. Gl anzel, and A. Schubert (2005): \A Hirsch-Type
Index for Journals," The Scientist, 19(22), 8.
Brin, S., and L. Page (1998): \The Anatomy of a Large-Scale Hypertex-
tual Web Search Engine," Computer Networks and ISDN Systems, 30(1{2),
107{117.
Garfield, E. (1955): \Citation Indexes to Science: A New Dimension in
Documentation through Association of Ideas," Science, 122(3159), 108{
111.
Hirsch, J. E. (2004): \An Index to Quantify an Individual's Scientic Re-
search Output," Proceedings of the National Academy of Sciences, 102(46),
16569{16572.
Kalaitzidakis, P., T. P. Mamuneas, and T. Stengos (2003): \Rank-
ings of Academic Journals and Institutions in Economics," Journal of the
European Economic Association, 1(6), 1346{1366.
K oczy, L.  A., and M. Strobel (2007): \Ranking Academic Journals by
Tournament Methods," Mimeo.
Laband, D. N., and M. J. Piette (1994): \The Relative Impacts of
Economics Journals: 1970-1990," Journal of Economic Literature, 32, 640{
666.
Liebowitz, S. J., and J. C. Palmer (1984): \Assessing the Relative
Impacts of Economics Journals," Journal of Economic Literature, 22(1),
77{88.
Minc, H. (1988): Nonnegative Matrices. John Wiley and Sons, New York.
Palacios-Huerta, I., and O. Volij (2004): \The Measurement of Intel-
lectual Inuence," Econometrica, 72(3), 963{977.
Pinski, G., and F. Narin (1976): \Citation Inuence for Journal Aggre-
gates of Scientic Publications: Theory, with Application to the Literature
of Physics," Information Processing & Management, 12(5), 297{312.
Roy, S., A. Saberi, and Y. Wan (2008): \Majorization for the Domi-
nant Eigenvector of a Nonnegative Matrix," American Control Conference,
Westin Seattle Hotel, Seattle, Washington, USA, pp. 1965{1966.
7

Article length bias in journal rankings

http://uni-obuda.hu/users/vecseya/RePEc/pkk/wpaper/0902.pdf

Article length bias in journal rankings

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics