The last decade methods for quantifying the research output of individual researchers have become quite popular in academic policy making. The h- index (Hirsch, 2005) constitutes an interesting quality measure that has attracted a lot of attention recently. It is now a standard measure available for instance on theWeb of Science. In this paper we establish the asymptotic normality of the empirical h-index. The rate of convergence is non-standard: ph=(1 + nf(h)), where f is the density of the citation distribution and n the number of publications of a researcher. In case that the citations follow a Pareto-type or a Weibull-type distribution as defined in extreme value theory, our general result nicely specializes to results that are useful for constructing confidence intervals for the h-index.Asymptotic normality;confidence interval;extreme value theory;research output;scientometrics;tail empirical process.

Beirlant, J.

Einmahl, J.H.J.

English

Research Papers in Economics

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
No. 2007–86 
 
ASYMPTOTICS FOR THE HIRSCH INDEX 
 
By Jan Beirlant, John H.J. Einmahl 
 
 
 
October 2007 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ISSN 0924-7815 Asymptotics for the Hirsch index ¤
Jan Beirlant John H.J. Einmahl
October 25, 2007
Abstract
The last decade methods for quantifying the research output of individual
researchers have become quite popular in academic policy making. The h-
index (Hirsch, 2005) constitutes an interesting quality measure that has
attracted a lot of attention recently. It is now a standard measure available
for instance on the Web of Science. In this paper we establish the asymptotic
normality of the empirical h-index. The rate of convergence is non-standard: p
h=(1 + nf(h)), where f is the density of the citation distribution and n
the number of publications of a researcher. In case that the citations follow
a Pareto-type or a Weibull-type distribution as de¯ned in extreme value
theory, our general result nicely specializes to results that are useful for
constructing con¯dence intervals for the h-index.
JEL codes: C13, C14.
Key words: Asymptotic normality, con¯dence interval, extreme value theory,
research output, scientometrics, tail empirical process.
1 Introduction
Since its introduction in Hirsch (2005), the h-index has been used as a measure
to quantify the research output of individual scientists based on the distribution
¤Jan Beirlant is Professor, Department of Mathematics and Leuven Statistics Research Cen-
tre, Katholieke Universiteit Leuven, Celestijnenlaan 200B, B-3001 Leuven, Belgium (Email:
jan.beirlant@wis.kuleuven.be). John H.J. Einmahl is Professor, Department of Econometrics &
OR and CentER, Tilburg University, P.O. Box 90153, 5000 LE Tilburg, The Netherlands (Email:
j.h.j.einmahl@uvt.nl). The authors are very grateful to Goedele Dierckx for helping out with the
application section.
1of citation counts of the di®erent papers of the individual. It is now a standard
measure available in Citation Reports on the Web of Science. It can also be applied
to higher levels of aggregation such as the di®erent papers that appeared in a given
journal over a particular period of time (Braun et al., 2005). If a researcher has
n publications and each of the m most cited publications has at least m citations,
then the maximal m is the empirical h-index of this researcher. In recent papers
(GlÄ anzel, 2006 and Egghe and Rousseau, 2006) the ¯rst theoretical properties of
the h-index were derived. The underlying citation distribution was then assumed
to be of Pareto-type. Speci¯cally the dependence of the h-index on the basic
parameters of the distribution and on the sample size was discussed. However the
distribution of the empirical h-index as an estimator of a statistical functional was
not discussed yet.
In order to be more precise, let us introduce some notation. Let X1;:::;Xn be
i.i.d. random variables with common distribution function F. They denote the
numbers of citations of the n articles of a certain researcher (or from a particular
journal) with citation distribution F. Although the citation counts follow a discrete
distribution, we will henceforth assume that F is continuous. That is of course
mathematically convenient but, as shown below, it also allows to discover the
main characteristics of the asymptotics. Let x¤ denote the right endpoint of F:
x¤ = supfx 2 R : F(x) < 1g. We will assume x¤ = 1, since the case x¤ < 1 is
not of interest in scientometrics, see the monograph Egghe (2005). We will also
assume F(0) = 0, since this is natural in this context, but it is actually not needed.
From the de¯nition above it follows that the theoretical h-index h = hn of a
researcher or a journal is de¯ned by
1 ¡ F(h) =
h
n
;
see also GlÄ anzel (2006). Observe that h is unique and that h = hn ! 1 and
h=n ! 0, as n ! 1. Denoting the (right-continuous) empirical distribution
function of the Xi;i = 1;:::;n, by ^ F, the empirical Hirsch index ^ H is de¯ned by
1 ¡ ^ F( ^ H) ·
^ H
n
and 1 ¡ lim
x" ^ H
^ F(x) ¸
^ H
n
: (1)
These two inequalities indeed have a unique solution. In case ^ F puts only mass
at non-negative integer values this de¯nition coincides with the aforementioned
2de¯nition in Hirsch (2005). ^ H is a simple and comprehensible measure that works
at any given level of aggregation. It combines citation impact with publication
activity. Because of the Glivenko-Cantelli theorem we have that ^ H ! 1 a.s.
(n ! 1).
It is the aim of this paper to study the precise asymptotic behavior of ^ H in a
completely non-parametric setting. We will establish the expected asymptotic
normality, but the rate of convergence is rather non-standard. In Section 2 the
general asymptotic result will be stated and derived. In Section 3 we specify the
general result to the case of Pareto-type and Weibull-type distributions. This
allows to construct con¯dence intervals for h on the basis of ^ H. In Section 4 we
apply these results to the publication-citation record of two well-known scientists.
2 Main results
In this section we present our general, main results. The next proposition leads to
the proper consistency result and is as well the main step to asymptotic normality
of ^ H.
Proposition 1 If F is continuous and x¤ = 1, then
^ H ¡ h
p
h
+
n(F( ^ H) ¡ F(h))
p
h
d ! N(0;1) as n ! 1: (2)
Since the terms on the left in Proposition 1 have the same sign, we obtain con-
sistency of ^ H. Obviously the result has to be stated in a ratio-setting; the usual
consistency formulation is pointless since h = hn ! 1 as n ! 1.
Corollary 1 (Consistency) Under the assumptions of Proposition 1, we have
^ H
h
P ! 1:
We now state the main result, asymptotic normality of ^ H; note the unusual con-
vergence rate.
Theorem 1 (Asymptotic normality) Assume F is continuous, the density f = F 0
3exists in h, and x¤ = 1. Assume also the following condition holds: for all R > 0,
sup
¡R·r·R
¯
¯
¯
¯
¯
¯
n
³
F
³
h + r
p
h
nf(h)
´
¡ F(h)
´
r
p
h
¡ 1
¯
¯
¯
¯
¯
¯
! 0 as n ! 1: (3)
Then
1 + nf(h)
p
h
( ^ H ¡ h)
d ! N(0;1) as n ! 1:
Proof of Proposition 1 Let ®n be the uniform empirical process based on the
F(Xi);1 · i · n, so
p
n( ^ F ¡ F) = ®n ± F a.s.
We have, almost surely,
^ H ¡ h ¸ n(1 ¡ ^ F( ^ H)) ¡ n(1 ¡ F(h))
= n(F(h) ¡ ^ F( ^ H))
= ¡n( ^ F( ^ H) ¡ F( ^ H) + F( ^ H) ¡ F(h))
= ¡
p
n®n(F( ^ H)) ¡ n(F( ^ H) ¡ F(h)):
Hence
^ H ¡ h + n(F( ^ H) ¡ F(h)) ¸ ¡
p
n®n(F( ^ H)) a.s.
Similarly we obtain
^ H ¡ h + n(F( ^ H) ¡ F(h)) · ¡
p
n®n(F( ^ H)) + 1 a.s.
Hence
^ H ¡ h
p
h
+
n(F( ^ H) ¡ F(h))
p
h
= ¡
r
n
h
®n(F( ^ H)) + o(1) a.s. (4)
Let wn be the tail empirical process near 1, based on ®n, i.e.
wn(s) =
r
n
h
®n
µ
1 ¡
h
n
s
¶
; 0 · s ·
n
h
:
Write
Tn =
n(F( ^ H) ¡ F(h))
p
h
:
Using both these de¯nitions and the de¯nition of h, (4) translates to
^ H ¡ h
p
h
+ Tn = ¡wn
µ
1 ¡
Tn p
h
¶
+ o(1) a.s. (5)
4This yields
¯
¯
¯
¯
¯
^ H ¡ h
p
h(1 _ (1 ¡ Tn=
p
h))3=4 +
Tn
(1 _ (1 ¡ Tn=
p
h))3=4
¯
¯
¯
¯
¯
· sup
0·s·n=h
jwn(s)j
(1 _ s)3=4+o(1) a.s.
(6)
It is well-known that
sup
0·s·n=h
jwn(s)j
(1 _ s)3=4 = OP(1) (7)
see, e.g., Einmahl (1997). Recall that ^ H ¡ h and n(F( ^ H) ¡ F(h)) have the same
sign. Hence using (7), we obtain from (6) that
Tn
(1 _ (1 ¡ Tn=
p
h))3=4 = OP(1):
It follows that Tn1[Tn¸¡1] = OP(1). If Tn < ¡1, we have for large n
1 _ (1 ¡ Tn=
p
h) = 1 ¡ Tn=
p
h · 1 ¡ Tn · ¡2Tn:
Hence Tn1[Tn<¡1] = OP(1), so Tn = OP(1). This yields
1 ¡
Tn p
h
P ! 1:
Combining this with (5) and the weak convergence of wn on [0;2] to a standard
Wiener process (see, e.g., Einmahl (1997)) yields (2). 2
Proof of Theorem 1 From Proposition 1 we see that
Ã
1 +
n(F( ^ H) ¡ F(h))
^ H ¡ h
!
^ H ¡ h
p
h
d ! N(0;1):
So it su±ces to show that
1 + nf(h)
1 +
n(F( ^ H)¡F(h))
^ H¡h
P ! 1;
which is implied by
F( ^ H) ¡ F(h)
f(h)( ^ H ¡ h)
P ! 1: (8)
To prove (8), observe that it follows from Proposition 1 that F( ^ H) ¡ F(h) =
OP(
p
h=n), i.e. for " > 0 we can ¯nd an M" > 0 such that with probability greater
than 1 ¡ ", for large n
jF( ^ H) ¡ F(h)j ·
M"
p
h
n
: (9)
5De¯ne ^ r = nf(h)( ^ H ¡ h)=
p
h. Then (9) reads
¯
¯
¯
¯
¯
F
Ã
h +
^ r
p
h
nf(h)
!
¡ F(h)
¯
¯
¯
¯
¯
·
M"
p
h
n
: (10)
Assume j^ rj > 2M". Then from the condition, with R = 2M", it follows that
¯
¯
¯
¯
¯
F
Ã
h +
^ r
p
h
nf(h)
!
¡ F(h)
¯
¯
¯
¯
¯
¸
3
2
M"
p
h
n
:
This contradicts (10). Hence with probability greater than 1 ¡ ", for large n,
j^ rj · 2M". Now apply the condition again with R = 2M", then
1 ¡ " ·
n(F( ^ H) ¡ F(h))
^ r
p
h
· 1 + ":
This is (8). 2
3 Pareto-type and Weibull-type tails
From our main theorem two important corollaries follow, which immediately yield
a con¯dence interval for h. These corollaries also show that the condition in The-
orem 1 is appropriate and can be validated under popular, semiparametric models
encountered in extreme value theory and scientometrics.
First assume that the distribution function F satis¯es the von Mises condition for
heavy-tailedness, i.e. we have
lim
x!1
xf(x)
1 ¡ F(x)
= ®: (11)
with tail index ® = 1=° > 0, with ° the extreme value index. It is then immediate
that
lim
n!1
nf(h) = lim
n!1
hf(h)
1 ¡ F(h)
= ®:
Nevertheless, it is remarkable that under this often-used, semiparametric model,
the complicated quantity nf(h) in the asymptotic normality statement can be re-
placed by ®. For di®erentiable distribution functions, assumption (11) is equivalent
6to stating that the tail of the distribution is Pareto-type (or equivalently Zip¯an
or Lotkaian):
1 ¡ F(x) = x
¡®`(x)
where ` is a slowly varying function:
`(tx)
`(t)
! 1 as t ! 1 for every x > 0:
This assumption is of fundamental importance in the ¯eld of informetrics (Egghe,
2005). When `(x) = C(1 + o(1)) for some positive constant C, when x ! 1,
one easily veri¯es that h = (Cn)1=(1+®)(1 + o(1)). Further we need Karamata's
representation theorem (see Theorem 1.3.1 in Bingham et al., 1987):
`(x) = c(x)exp
µZ x
1
±(u)du=u
¶
(12)
with c(x) ! c0 > 0 and ±(x) ! 0, as x ! 1.
Let ^ ° be one of the many known consistent estimators of the extreme value index
° > 0 and set ^ ® = 1=^ °; see for instance Chapters 4 and 5 in Beirlant et al. (2004).
Corollary 2 When (11) and (12) hold with c ´ c0, we have
1 + ^ ®
p
^ H
( ^ H ¡ h)
d ! N(0;1):
Proof of Corollary 2 We only need to check condition (3) of Theorem 1. With
(12) it follows that
F
Ã
h +
r
p
h
nf(h)
!
¡ F(h)
= h
¡®`(h)
Ã
1 ¡
³
1 + r=(
p
hnf(h))
´¡® `(h + r
p
h=(nf(h)))
`(h)
!
:
When the function c is constant we have for large n
¯
¯
¯
¯
¯
log
Ã
`(h + r
p
h=(nf(h)))
`(h)
!¯
¯
¯
¯
¯
=
¯
¯
¯
¯
¯
Z h+r
p
h=(nf(h))
h
±(u)du=u
¯
¯
¯
¯
¯
·
2jrj
p
hnf(h)
sup
u¸h=2
j±(u)j:
7Hence, since nh¡®¡1`(h) = 1 by the de¯nition of h, we obtain after a straightfor-
ward calculation
sup
¡R·r·R
¯
¯
¯
¯
¯
¯
n
³
F
³
h + r
p
h
nf(h)
´
¡ F(h)
´
r
p
h
¡ 1
¯
¯
¯
¯
¯
¯
= nh
¡®¡1`(h)
µ¯
¯
¯
¯
®
nf(h)
(1 + o(1)) ¡ 1
¯
¯
¯
¯ +
2supu¸h=2 j±(u)j
nf(h)
(1 + o(1))
¶
! 0: 2
Similarly the main theorem can be speci¯ed for the class of Weibull-type distribu-
tions, de¯ned as
1 ¡ F(x) = exp(¡x
¿`(x)) (13)
with Weibull parameter ¿ > 0 and a slowly varying function `. Examples are the
gamma, normal and Weibull distributions. This subclass constitutes an important
subclass of the Gumbel domain of max-attraction, characterized by ° = 0. Con-
sistent estimators ^ ¿ of ¿ can be found for instance in Girard (2004) and Diebolt
et al. (2007). Remark that under (13) with `(x) = C(1 + o(1)) for some positive
constant C as x ! 1, we have hn = (C¡1 logn)1=¿(1 + o(1)) as n ! 1. The
following result now follows using similar calculations as for Corollary 2.
Corollary 3 When (13) holds with ` satisfying (12) with the function c constant,
we have
^ ¿ logn
p
^ H
( ^ H ¡ h)
d ! N(0;1):
It is interesting to note that for ¿ = 1=2 and `(x) = C(1 + o(1)) for some positive
constant C as x ! 1, this result simpli¯es to ^ H ¡ h
d ! N(0;4=C2).
4 Application
As an application we consider here the h-index of David R. Cox and of P¶ al Erd} os,
two really outstanding scientists who do not need any further introduction.
In Figure 1 we see the functions 1 ¡ ^ F(¢) and ¢=n, cf. (1), for the n = 450 papers
of Cox. We ¯nd an empirical Hirsch index ^ HCox = 57. Figure 2 shows the Pareto
QQ-plot, plotting on a log-log scale the ordered data against their theoretical
values under the standard Pareto model, jointly with four well-known estimators
8number of citations
0 200 400 600 800 1000
0.0
0.2
0.4
0.6
0.8
1.0 empirical tail function
H ^
Figure 1: The empirical Hirsch index for Cox.
of the extreme value index ° (see Chapters 4 and 5 in Beirlant et al. (2004)).
The QQ-plot is based on the 307 papers with non-zero citations. It shows linear
(a) (b)
0 1 2 3 4 5
0
2
4
6
8
10
0 50 100 150 200 250 300
0.0
0.5
1.0
1.5
2.0
Figure 2: F or the citation sizes of Co x: (a) P areto QQ-plot (b) Hill (full line),
Momen t (dotted line), ML (dashed-dotted), UH (dashed line) estimator.
b eha vior with p ositiv e slop e for the large citations n um b ers, indicating that the
citation distribution is hea vy tailed. In fact, the Hill estimator can b e considered
as the slop e of the QQ plot when using only the largest k observations. A careful
inspection of the plot of the estimators of the extreme value index leads to an
estimate ^ ° = 1:2; certainly for k running from 25 to 125 the four estimators show
stable behavior. Using ^ ® = 1=^ ° in Corollary 2, we obtain (50:2;63:8) an asymptotic
990%-con¯dence interval for hCox .
Based on the n = 519 papers of Erd} os, we ¯nd an empirical Hirsch index ^ HErd} os =
22. The Pareto QQ-plot (based on the 393 papers with non-zero citations) and
the four estimators of ° are shown in Figure 3. We see again linear behavior in the
(a) (b)
0 1 2 3 4 5 6
0
1
2
3
4
5
0 100 200 300 400
0.0
0.5
1.0
1.5
2.0
Figure 3: F or the citation sizes of Erd} os: (a) P areto QQ-plot (b) Hill (full line),
Momen t (dotted line), ML (dashed-dotted), UH (dashed line) estimator.
QQ-plot for the large citation n um b ers. The plot of the estimators of the extreme
v alue index leads to an estimate ^ ° = 0:6. This yields (19:1;24:9) as an asymptotic
90%-con¯dence interval for hErd} os .
References
[1] Beirlant, J., Goegebeur, Y., Segers, J. and Teugels, J. (2004). Statistics of Extremes,
Theory and Applications. Wiley, New York.
[2] Bingham, N.H., Goldie, C.M. and Teugels, J.L. (1987). Regular Variation. Encyclo-
pedia of Mathematics and its Applications, 27. Cambridge University Press.
[3] Braun, T., GlÄ anzel, W. and Schubert, A. (2005). A Hirsch-type index for journals.
The Scientist, 19, 8.
[4] Diebolt, J., Gardes, L., Girard S. and Guillou A. (2007). Bias-reduced estimators
of the Weibull tail-coe±cient, Test, to appear.
10[5] Egghe, L. (2005). Power Laws in the Information Production Process: Lotkaian
Informetrics. Wiley, New York.
[6] Egghe, L. and Rousseau, R. (2006). An informetric model for the h-index. Sciento-
metrics, 69, 121-129.
[7] Einmahl, J.H.J. (1997). Poisson and Gaussian approximation of weighted local em-
pirical processes. Stochastic Process. Appl., 70, 31-58.
[8] Girard, S. (2004). A Hill type estimate of the Weibull tail-coe±cient. Comm. in
Statist. Theory and Methods, 33, 205-234.
[9] GlÄ anzel, W. (2006). On the h-index - A mathematical approach to a new measure
of publication activity and citation impact. Scientometrics, 67, 315-321.
[10] Hirsch, J.E. (2005). An index to quantify an individual's scienti¯c research output.
Proceedings of the National Academy of Sciences of the United States of America,
102, 16569-16572.
11

A Hill type estimate of the Weibull tail-coe±cient.

A Hirsch-type index for journals.

An index to quantify an individual's scienti¯c research output.

An informetric model for the h-index.

Bias-reduced estimators of the Weibull tail-coe±cient, Test, to appear. 10[5] Egghe,

On the h-index - A mathematical approach to a new measure of publication activity and citation impact.

Poisson and Gaussian approximation of weighted local empirical processes. Stochastic Process.

Regular Variation. Encyclopedia of Mathematics and its Applications, 27.

Statistics of Extremes, Theory and Applications.

Asymptotics for the Hirsch Index

Asymptotics for the Hirsch Index

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics