Roberto Serrano

English

Research Papers in Economics

The Measurement of Intellectual Inﬂuence:
the Views of a Sceptic∗
Roberto Serrano†
Working Paper No. 2004-02
January 2004
Abstract. In an extremely interesting paper, Palacios-Huerta and Volij (2004) [PV] in-
troduce the axiomatic method to the problem of how to rank academic journals on the
basis of their mutual citations. They characterize the invariant method as the only one
satisfying a list of ﬁve appealing properties. In this note, I show an impossibility result, by
identifying a sixth property that is violated by the invariant method. Further, I question
the appeal of the PV axioms, when applied over larger domains of problems that take into
account making distinctions among types of citations.
JEL classiﬁcation numbers: A0.
Keywords: journal rankings, invariant method, impossibility result, axiomatic method.
∗I am indebted to Pierre Menard, Ignacio Palacios-Huerta and Oscar Volij for introducing me to the
topic and for helpful comments. I also remember at this time those colleagues who make out of the art of
counting one of their main daily occupations. Research support from National Science Foundation under
grant SES-0133113 and Deutsche Bank is acknowledged.
†Brown University, roberto serrano@brown.edu, www.econ.brown.edu/faculty/serrano1 Introduction
In an extremely interesting paper, Palacios-Huerta and Volij (2004), PV in the sequel,
introduce the axiomatic method to the problem of ranking journals as a function of their
citations. They rightly criticize previous attempts to reduce all the information contained
in scholarly publications to a simple ranking. Indeed, they point out that all rankings
available prior to theirs violate at least one of the axioms they propose.
Their main result is the characterization of the invariant method as the only one that
satisﬁes a list of ﬁve reasonable axioms:
anonymity, i.e., the ranking should not depend on the names of the journals;
invariance to citation intensity, i.e., all other things equal, the ranking should be unaf-
fected by the length of the reference section in the papers published by a journal;
weak homogeneity, i.e., the relative ranking of any two journals should be a function of
their mutual citations;
weak consistency, i.e., the ranking method should be “consistent” when applied to prob-
lems involving diﬀerent numbers of journals; and
invariance to splitting of journals, i.e., if a journal is subdivided into two identical sub-
journals in terms of their citations, each of the two receives half the original weight
of the mother journal, while the valuations of all other journals are unaﬀected.
All these are reasonable principles in the domain of problems studied by PV. However,
when introducing axiomatics, some problems call for an impossibility result, and I believe
this is one such problem. Indeed, I propose a sixth reasonable property that is violated
by the invariant method, thereby obtaining an impossibility result. We also question the
appeal of the PV axioms when used over larger domains that take into account diﬀerent
kinds of citations.
In general, I agree with PV that neither the invariant method nor any other can sub-
stitute for the expert judgement and sound opinion of scholars. My analysis is therefore a
complement to that of PV.
2 The Model
Let J be a non-empty set of journals. Let J ⊂Jbe a subset of journals. A citation
matrix for J is a (|J|,|J|) non-negative matrix C =( cij), where for each i,j ∈ J, cij is
the citations to journal i by journal j (i.e., the number of times that articles published in
journal i are cited by articles published in journal j). It is assumed that C is irreducible.
Deﬁne D to be the diagonal matrix, whose (jj)-th term is
P
i∈J cij. Deﬁne A to be the
diagonal matrix, whose (jj)-th term is aj, the number of articles published by journal j.
1We shall refer to the triple R ≡h J,A,Ci as a ranking problem, where J ⊂J. Let R be
the set of all ranking problems.
A ranking method φ assigns to each ranking problem R = hJ,A,Ci∈Ra vector
φ(R) ∈ ∆J, where ∆J is the unit simplex over J. Thus, the vector φ(R) provides relative
valuations of the journals in J as a function of the information contained in A and C.
The invariant method is the following ranking method: φI(R) assigns to each ranking
problem R the vector v ∈ ∆J satisfying
(I − A
−1CD
−1A)v =0 .
PV argue forcefully in favor of the invariant method. In particular, they demonstrate
that it is the only ranking method that satisﬁes anonymity, invariance to citation intensity,
weak homogeneity, weak consistency and invariance to splitting of journals. All these
properties can be justiﬁed in the domain of problems under study, and PV successfully do
so.
3 An Impossibility Result
Our ﬁrst observation is that one could think of an additional appealing property that the
invariant method violates, thereby yielding an impossibility result.
Example 1 Consider the following ranking problem R = hJ,A,Ci, where J = {1,2},
A is such that aj = a for j =1 ,2, and C is the matrix with the constant k in every
entry. Clearly, the invariant method assigns the vector (1/2,1/2) to this problem, which is
reasonable. Consider now the addition of a third journal, which also publishes a articles,
and suppose that the new citation matrix C0 has the following entries:



k − 1 k 1
kk 0
10 2 k − 1


.
To make our point clear, the reader should think of k as being a large constant. One
can calculate that the invariant method assigns to the expanded problem the vector
(1/3,1/3,1/3), which does not appear reasonable. The same conclusion is reached if the
original set of journals consists of an arbitrary number, |J|, of journals, instead of only
two.
The example illustrates the poor behavior of the invariant method “near a reducible
problem.” The reason is that the method, as deﬁned, need not be continuous at reducible
points: for reducible matrices, there are multiple solutions to the stationary equation that
deﬁnes the invariant method, and therefore, the mapping of solutions to the equation is
2not a lower hemicontinuous correspondence at those points.1 We shall call journals like
the new journal in Example 1 “boundary journals.”
Furthermore, the example can be modiﬁed to get rid of the zeros, because of the
continuity of the invariant method on the domain of irreducible problems. This means
that there will exist an open ball of irreducible matrices around that of Example 1 in
which the invariant method weights will be far from measuring true inﬂuence.
consider the following axiom:
Limited inﬂuence of Boundary Journals:2 Let R ≡h J,A,Ci be a ranking problem where
A = aIJ and C is such that cij/
P
i∈J cij is in an ￿-neighborhood of 1/|J| for all i,j ∈ J
for some ￿>0. Consider a new ranking problem R0 ≡h J0,A 0,C0i, where J0 = J ∪{ j∗},
A0 = aIJ0, and C0 is such that c0
ij/
P
i∈J0 c0
ij is in the same ￿-neighborhood of 1/|J| for every
i,j ∈ J and
P
j∈J c0
j∗j/
P
i∈J0 c0
ij and
P
i∈J c0
ij∗/
P
i∈J0 c0
ij∗ are in the same ￿-neighborhood of
0. Then, φj∗(R0) must not be in the ￿-neighborhood of (1/|J0|).
That is, moving from a system of journals with a heavy ﬂow of mutual citations to
an expanded one in which the inﬂuence of a new journal on the system and the inﬂuence
of the system on the new journal are negligible, the weight assigned to the new journal
should not be too large.3
Theorem 1 There is no ranking method satisfying the PV axioms and limited inﬂuence
of boundary journals.
The proof is simply the combination of PV’s Theorem 1 and the above example.
4 The PV Axioms in Richer Domains
We make two ﬁnal points. The PV axioms, which look reasonable in the domain of ranking
problems proposed, may not be as appealing in other domains.
(i) Suppose that one can distinguish good citations (instances in which a paper is cited
like this: “we build our analysis on Smith (1776)” or “Smith (1776) is our motivation”)
1The implication is that if the new journal in Example 1 is ran by a lunatic editor, who chooses
to publish papers that cite only his journal (he only made one exception, publishing a paper that cited
something in journal 1, only because journal 1 published one article that cites his journal), he is generously
rewarded by the invariant method. Alternativley, Example 1 points to the importance of choosing the set
J wisely.
2This is to be distinguished from the more questionable “limited inﬂuence of journals at the frontier”.
3One could think of diﬀerent ways of phrasing this axiom. I chose this presentation because it is a very
weak requirement. Other versions appear in Serrano and Menard (2003), and I thank Pierre Menard for
his suggestion on how to appear inﬂuential. After all, if the current paper is published, Menard will have
a substantial impact in economics, if measured by the invariant method.
3from bad citations (“Menard (1776) makes the following egregious mistake” or “For the
literature to make any progress, the model in Menard (1776) must be abandoned).4
Example 2 Consider a two-journal problem, with aj = a for j =1 ,2 and with citation
matrix having the constant k in every entry. Suppose, however, that the k citations made
by journal 1 to journal 2 are good, while the ones made by journal 2 to journal 1 are bad.
It is then clear that a method that measures “good” inﬂuence should not satisfy weak
homogeneity, i.e., prescribe a relative valuation of the two journals solely on the basis of
mutual citations. A similar comment applies to weak consistency, because one would have
to be very careful deﬁning the appropriate reduced problem.
(ii) There are certain contributions that are so inﬂuential that do not make it to the
references section. For example, authors that work with Nash equilibrium or Arrow-Debreu
economies rarely refer to the original articles. For cases like these, that one could call
“textbook references” in the sense that they have already made it to the basic language of
a profession, no method based on citations alone would make them justice.
5 Concluding Remark
I have pointed out three deﬃciencies of the invariant method: its poor behavior near
reducible problems, its inability to distinguish between good and bad citations, and its
disregard for “textbook references.” Fortunately, one could think of ways to alleviate
these problems, and in all cases they entail a more careful reading of the works.
References
Borges, J. L. (1962), “Pierre Menard, Author of Don Quixote,” in Ficciones, English
translation, Grove Press Inc., from Spanish edition, 1956.
Menard, P. (1776), An Inquiry into the Nature and Causes of the Wealth of Nations,
virtual manuscript, Universit´ e Fictionale.
Palacios-Huerta, I. and O. Volij (2004), “The Measurement of Intellectual Inﬂuence,”
Econometrica 72, 963-978.
Serrano, R. and P. Menard (2003), “The Measurement of Intellectual Inﬂuence: the Views
of Two Sceptics,” mimeo.
4The above quotes are found verbatim in the literature, surprising as it may seem given that the text in
Menard is absolutely identical to Smith’s. The citing authors, though, rightly appreciated the important
diﬀerences. See also Borges (1962).
4Smith, A. (1776), An Inquiry into the Nature and Causes of the Wealth of Nations,
Edinburgh.
5

Pierre Menard, Author of Don Quixote,”

The Measurement of Intellectual Influence: the Views of a Sceptic

http://www.brown.edu/Departments/Economics/Papers/2004/2004-02_paper.pdf

The Measurement of Intellectual Influence: the Views of a Sceptic

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics