We show that the invariant method (Pinski and Narin, 1976), recently axiomatised by Palacios-Huerta and Volij (2004), and used to quality-rank academic journals is subject to manipulation: a journal can boost its performance by making additional citations to other journals.Economics (Jel: A)

Kóczy, László Á.

Strobel Martin

English

Research Papers in Economics

László Á. Kóczy, Martin Strobel 
 
The Invariant Method is Manipulable 
 
RM/08/029 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Maastricht research school of Economics 
of TEchnology and ORganizations 
 
Universiteit Maastricht 
Faculty of Economics and Business Administration 
P.O. Box 616 
NL - 6200 MD Maastricht 
 
phone  : ++31 43 388 3830 
fax  : ++31 43 388 4873 
 
 
 
 The Invariant Method is Manipulable
L aszl o  A. K oczyyand Martin Strobelz
Abstract
We show that the invariant method (Pinski and Narin, 1976), re-
cently axiomatised by Palacios-Huerta and Volij (2004), and used to
quality-rank academic journals is subject to manipulation: a journal
can boost its performance by making additional citations to other
journals.
Keywords: quality ranking academic journals, invariant method,
manipulation
1 Introduction
In a recent surge of interest in ranking academic journals Palacios-Huerta
and Volij (2004) present an axiomatisation of the invariant method (Pinski
The authors thank the funding by METEOR; K oczy acknowledges the support of the
European Union by a Marie Curie Intra-European Fellowship (MEIF-CT-2004-011537).
yCorresponding author. Department of Economics, Maastricht University and Keleti
Faculty of Economics, Budapest Tech Tavaszmez} o 15-17, H-1084 Budapest. Email:
koczy.laszlo@kgk.bmf.hu
zDepartment of Economics, Maastricht University.
1and Narin, 1976) as the unique distribution vector that satises anonymity,
invariance to citation intensity, weak homogeneity, weak consistency and
invariance to splitting of journals. While many other ranking methods have
been proposed in the past, mostly without motivation. Palacios-Huerta and
Volij (2004) give an axiomatisation for the invariant method by a number of
attractive properties.
With journal rankings playing a more and more prominent role in aca-
demics, both in personnel and funding decisions, the incentives to improve
one's journal's rank, possibly by manipulation, is there. Indeed, for most
ranking methods the rank of a journal can be improved by changing its char-
acteristics or its citation pattern. We show that the invariant method is no
exception here.
In the following we elaborate on the idea of manipulation and prove our
claim by an example.
2 Manipulation
The invariant method and its properties highlight a number of problems in
comparing journals of dierent elds. While a bias in this sense is unfor-
tunate, such dierences can be accounted for. For instance, increasing a
journal's citation intensity would only be benecial if also citing journals
would increase theirs, such a concerted eort is, however, unlikely and comes
with incentives to free ride.
2On the other hand, possibilities to manipulate rankings, that is, to boost
a journal's ranking unilaterally we must view with concern. In their presence
a ranking re
ects quality, but also the strategic attitude of the publisher or
the editor.
Manipulation can take various forms. Since the impact factor (Gareld,
1955) { calculated as the cites per article ratio in a window of two years {
has become the standard to evaluate journal quality, publishers and editors
have inquired about the exact method of its computation (Jennings, 1999)
only to realise that it can be manipulated by classifying papers as \letters to
the editor" and not as \articles" (Didierjean, 2002). While this aects the
impact factor and rankings based on it, a more sophisticated ranking method
can account for such deviations and present a more correct ranking.
It is therefore more fortunate if the manipulation involves the citations
themselves. A citation refers to information that is not originally created in
the citing journal, but has appeared earlier in the cited journal. Receiving a
citation indicates that a result was found useful elsewhere. It is therefore not
surprising that citation analysis is used to evaluate the quality of journals; if
rankings can be manipulated by changing citation patterns the very meaning
of citations is questioned aecting science itself.
Citations can be manipulated by adding or removing citations. It is
important to stress that any ranking using citation analysis is subject to
manipulation if we allow for the latter { ad absurdum a journal not making
any citations could be ranked top by, for instance, the LP method (Liebowitz
3and Palmer, 1984). We can only assume that such a practice would swiftly
be rejected by the scientic community.
Adding self-citations has often the same eect: while the use of external
results is acknowledged, their importance is diminished by the more extended
use of results from the home journal. Self citations can manipulate the impact
factor (Smith, 1997), the LP- and the invariant method.
While, in a somewhat controversial step, these gratuitous citations can
be neutralised by ignoring self-citations altogether, manipulation by making
additional citations to other journals is shakes the foundations of citation
analysis.
That this kind of manipulation is possible is strange and almost paradox-
ical. An additional cite suggests that the results published are less original,
but taken from elsewhere. Such a cite should (weakly) improve the cited
journal's rank, while the journal in question should (weakly) drop in the
ranking. Should such manipulation be possible, the ranking method would
fail some natural monotonicity criterion. Non-manipulable ranking methods
exist (K oczy and Strobel, 2007).
3 An example
Proposition. The ranking based on the invariant method is manipulable by
making additional citations to other journals.
Proof. Consider the example with journals f1;2;3;4g each publishing 2 arti-
4cles (Using the notation of Palacios-Huerta and Volij (2004) a1 = a2 = a3 =
a4 = 2.) and a citation matrix given by
C =
0
B B B B B B
B B
@
0 1 1 1
1 0 0 2
1 1 0 0
1 0 1 0
1
C
C C C C C
C C
A
: (3.1)
The invariant method gives the vector v = (30;24;22;21)=97 ranking
journal 4 the lowest. Now suppose this journal makes 2 additional citations
to journal 1. The citation matrix is modied as follows:
C
0 =
0
B
B B B B B B
B
@
0 1 1 3
1 0 0 2
1 1 0 0
1 0 1 0
1
C
C C C C C C
C
A
; (3.2)
and the corresponding invariant vector is given by v0 = (54;32;34;35)=155.
In the ranking based on the new invariant vector journal 4 is ranked second,
overtaking journals 2 and 3.
The example is not very special, the problem is generic: Adding citations
to highly ranked journals diminishes the value of other citations giving other
cited journals a lower score.
That the invariant method can be manipulated by self-cites is fairly triv-
ial.
54 Conclusion
We have shown that the invariant method is subject to manipulation: a
journal's rank can be improved by making additional citations, possibly to
other journals.
We consider non-manipulability an important property, even more im-
portant than some of the properties used by (Palacios-Huerta and Volij,
2004), for instance invariance to citation-intensity. While journals in higher
citation-intensity elds are often ranked better, citation intensity is an exoge-
nous variable that can hardly be changed. A unilateral increase of citation-
intensity is unlikely to be benecial, while coordinated eorts in a eld are
subject to free riding. Manipulation, in contrast, allows for publishers and
editors to boost their indicators by unilateral steps. A manipulation-free
ranking should let them focus on genuine quality.
References
Didierjean, X. (2002): \\Editors! Check Your Impact Factor Data!" Let-
ters to Dermatology Count More towards the Calculation of the Impact
Factor than in Any Other Dermatology Journal," Dermatology, 205(4),
327{328.
Garfield, E. (1955): \Citation indexes to science: a new dimension in
documentation through association of ideas," Science, 122(3159), 108{111.
6Jennings, C. (1999): \Citation data: the wrong impact?," Neuroen-
docrinology Letters, 20(1{2), 7{10.
K oczy, L.  A., and M. Strobel (2007): \Ranking Academic Journals by
Tournament Methods," Mimeo.
Liebowitz, S. J., and J. C. Palmer (1984): \Assessing the Relative
Impacts of Economics Journals," Journal of Economic Literature, 22(1),
77{88.
Palacios-Huerta, I., and O. Volij (2004): \The Measurement of Intel-
lectual In
uence," Econometrica, 72(3), 963{977.
Pinski, G., and F. Narin (1976): \Citation In
uence for Journal Aggre-
gates of Scientic Publications: Theory, with Application to the Literature
of Physics," Information Processing & Management, 12(5), 297{312.
Smith, R. (1997): \Journal accused of manipulating impact factor," British
Medical Journal, 314(7079), 461.
7

Assessing the Relative Impacts of Economics Journals,&quot;

Citation data: the wrong impact?,&quot; Neuroendocrinology Letters,

Citation In uence for Journal Aggregates of Scienti Publications: Theory, with Application to the Literature of Physics,&quot;

Citation indexes to science: a new dimension in documentation through association of ideas,&quot;

Editors! Check Your Impact Factor Data!&quot; Letters to Dermatology Count More towards the Calculation of the Impact Factor than in Any Other Dermatology Journal,&quot; Dermatology,

The Invariant Method is Manipulable

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics