When searching for a new resonance somewhere in a possible mass range, the
significance of observing a local excess of events must take into account the
probability of observing such an excess anywhere in the range. This is the so
called "look elsewhere effect". The effect can be quantified in terms of a
trial factor, which is the ratio between the probability of observing the
excess at some fixed mass point, to the probability of observing it anywhere in
the range. We propose a simple and fast procedure for estimating the trial
factor, based on earlier results by Davies. We show that asymptotically, the
trial factor grows linearly with the (fixed mass) significance

Eilam Gross

H. Chernoff

L. Demortier

L. Lyons

Ofer Vitells

R.B. Davies

S.S. Wilks

English

arXiv

Crossref

The European Physical Journal C

Trial factors for the look elsewhere effect in high energy physics

Springer

Springer - Publisher Connector

Eur. Phys. J. C (2010) 70: 525–530
DOI 10.1140/epjc/s10052-010-1470-8
Special Article - Tools for Experiment and Theory
Trial factors for the look elsewhere effect in high energy physics
Eilam Gross, Ofer Vitellsa
Weizmann Institute of Science, Rehovot 76100, Israel
Received: 1 June 2010 / Revised: 28 September 2010 / Published online: 15 October 2010
© The Author(s) 2010. This article is published with open access at Springerlink.com
Abstract When searching for a new resonance somewhere
in a possible mass range, the significance of observing a lo-
cal excess of events must take into account the probability of
observing such an excess anywhere in the range. This is the
so called “look elsewhere effect”. The effect can be quanti-
fied in terms of a trial factor, which is the ratio between the
probability of observing the excess at some fixed mass point,
to the probability of observing it anywhere in the range. We
propose a simple and fast procedure for estimating the trial
factor, based on earlier results by Davies. We show that as-
ymptotically, the trial factor grows linearly with the (fixed
mass) significance.
1 Introduction
The statistical significance that is associated to the observa-
tion of new phenomena is usually expressed using a p-value,
that is, the probability that a similar or more extreme effect
would be seen when the signal does not exist (a situation
usually referred to as the null or background-only hypoth-
esis). It is often the case that one does not a priori know
where the signal will appear within some possible range. In
that case, the significance calculation must take into account
the fact that an excess of events anywhere in the range could
equally be considered as a signal. This is known as the “look
elsewhere effect” [1, 2]. In the statistical literature this sit-
uation is usually referred to as an hypothesis test when a
nuisance parameter is present only under the alternative, for
which the standard regularity conditions do not apply. The
problem is however closely related to that of a level cross-
ings of a stochastic process, which has been studied exten-
sively and for which many relevant results exist [3, 4]. In
particular [4] provides an upper bound on the tail probability
of the maximum of a chi-squared process, which can there-
fore be applied to cases when the test statistic is the profile
a e-mail: ofer.vitells@weizmann.ac.il
likelihood ratio and the large-sample distribution at a fixed
point given by Wilks’ theorem [5] holds.
Of course, a straightforward way of quantifying the look-
elsewhere effect may be simply running many Monte-Carlo
simulations of background only experiments, and finding for
each one the largest fluctuation that resembles a signal.1 Ex-
amples for such an approach can be found e.g. in [6]. While
this procedure is simple and gives the correct answer, it is
also time and CPU consuming, as one would have to repeat
it O(107) times to get the p-value corresponding to a 5σ
significance. Approximations based on large sample limits
such as [4] can therefore be valuable.
To make the discussion concrete, let us first describe a
typical example of a ‘mass bump’ search. The statistical
model in this case consists of a background distribution b,
and a signal distribution s(m) with an unknown location pa-
rameter m (corresponding to the mass of a resonance). We
introduce a signal strength parameter μ, such that the model
is given by μs(m) + b, and we test H0 : μ = 0 against the
alternative, μ > 0.2 Both s and b may depend on additional
nuisance parameters which are omitted from the notation.
The mass m is another nuisance parameter, but it does not
exist under H0 since b does not depend on m. The testing
problem has therefore a nuisance parameter that is present
only under the alternative. When searching for a resonance
that can appear anywhere in the range, one will look for the
largest excess of events above the background. More pre-
cisely, if q(m) is a test statistic for a fixed mass, and large
values of q(m) indicate decreasing compatibility with the
null hypothesis, then the test statistic of the entire range
would be q(mˆ) = maxm[q(m)]. The problem is therefore in
1It is assumed that the signal can appear only in one location, therefore
when a search is conducted, one will look for the largest excess of
events and regard all others as background fluctuations.
2Strictly speaking, Wilks’ theorem does not apply here since the tested
value of μ is on the boundary of the allowed region. This difficulty
however can be easily alleviated by extending the allowed region of
μ to negative values and then reducing the p-value by half. A formal
generalization of this nature is given in [7].
526 Eur. Phys. J. C (2010) 70: 525–530
assessing the tail probability (p-value) of the maximum of
q(m) over the search range.
In Sect. 1 we review the main result of [4] and its appli-
cation to the present case. We propose a practical procedure
for estimating the p-value, which we then demonstrate with
a toy model simulation in Sect. 3.
2 Tail probabilities of the likelihood ratio test statistic
Suppose that a signal hypothesis depends on a nuisance pa-
rameter θ that does not exist under the null. We denote by
q(θ) the profile likelihood test statistic for some fixed θ and
we assume that it follows a χ2 distribution with s degrees of
freedom, as would be the case in the large sample limit when
Wilks’ theorem holds. We are interested in the tail probabil-
ity of the maximum of q(θ) over θ , which we denote by
q(θˆ). As shown in [4], this is bounded by:
P
(
q(θˆ) > c
) ≤ P (χ2s > c
) + 〈N(c)〉, (1)
where N(c) is the number of ‘upcrossings’ of the level c by
the process q(θ), with an expectation that is given by [4]:
〈
N(c)
〉 = c
(s−1)/2e−c/2√
π2s/2Γ (s/2 + 1/2)
∫ U
L
C(θ) dθ. (2)
Where [L,U ] is the range of possible values of θ , and
C(θ) is some function that depends on the details of the sta-
tistical model. To have the maximum of q(θ) above the level
c means that either the value at the lower threshold q(L) is
larger than c, or that there is at least one upcrossing, hence
the two terms in (1). Note that the bound is expected to be-
come an equality for large values of c, as the expected num-
ber of upcrossings will be dominated by the probability of
one upcrossing, that is when 〈N(c)〉  1.
The function C(θ) can in general be difficult to calculate.
Instead, we propose to estimate 〈N(c0)〉 at some low refer-
ence level c0 by simply counting the number of upcrossings
in a small set of background-only Monte Carlo simulations.
Equation (1) then becomes
P
(
q(θˆ) > c
) ≤ P (χ2s > c
) + 〈N(c0)
〉( c
c0
)(s−1)/2
e−(c−c0)/2
(3)
and so once 〈N(c0)〉 is determined, the p-value of a given
observation and the corresponding significance are readily
obtained.
Naturally, one would like to choose the reference level c0
in a way that will minimize the resulting uncertainty on the
bound. From (3), it can be seen that the statistical uncertainty
is proportional to σN/〈N〉 where σN is the standard devia-
tion of N . As long as the distribution of N is ‘well behaved’
in the sense that the relative uncertainty decreases with in-
creasing 〈N〉, then the optimal choice of c0 would be that
which maximizes the expected number of upcrossings (2),
that is c0 = s − 1. Note that in the case s = 1, the maximal
number of upcrossings occurs at c0 → 0, however the abil-
ity to reliably estimate 〈N(c0)〉 at very low levels depends on
the numerical resolution at which q(θ) is calculated. In this
case, one should therefore aim at having c0 as low as possi-
ble but still significantly larger than the numerical resolution
of q(θ), and the typical distance between upcrossings should
be kept significantly larger than the θ resolution. In the ex-
ample considered in the following section c0 = 0.5 satisfies
those conditions and proves to be a good choice.
It is further interesting to note that the dependence of
〈N(c)〉 on c in (2), is the same as the asymptotic form of
the cumulative distribution of a χ2 variable with s + 1 de-
grees of freedom,
P
(
χ2s+1 > c
) −−−→
c→∞
c
1
2 (s+1)−1e−c/2
2
1
2 (s+1)−1Γ ((s + 1)/2)
(4)
allowing us to write, for large c (c 	 s),
P
(
q(θˆ) > c
) ≈ P (χ2s > c
) + N P (χ2s+1 > c
)
, (5)
where
N = 1√
2π
∫ U
L
C(θ) dθ. (6)
And the bound has been replaced by a ‘≈’ sign since
we are dealing with the large c limit. The probability de-
scribed by (5) has a natural interpretation. It is the same
as one would have for a random variable that is the maxi-
mal of n independent χ2s+1 variates and one χ2s variate, with
E[n] = N . That is, if
y = max[x0, x1, . . . , xn] (7)
and
x0 ∼ χ2s , xi ∼ χ2s+1 i = 1, . . . , n, E[n] = N
then the tail probability P(y > c) for large c is given by the
right hand side of (5).
Intuitively, this suggests that we can view the range of θ
as being composed of several (on average N ) independent
regions , where in each one the likelihood fit involves an ex-
tra degree of freedom due to the variable mass, leading to a
χ2s+1 distribution. The χ2s term accounts for the possibility
of having a local maximum on the edge of the allowed re-
gion (assuming, from symmetry reasons, that each edge has
a probability 1/2 of being a local maximum). The number
N can therefore be interpreted as an ‘effective number’ of
independent search regions.
Eur. Phys. J. C (2010) 70: 525–530 527
We remark that exactly such an intuitive reasoning was
employed by Quayle [8] as a conjecture for the distribution
of q(θˆ). It was found that the distribution of a random vari-
able defined according to (7) reproduces, to a good approx-
imation, that of q(θˆ), and that the agreement is better at the
tail. As shown above, this behavior is expected and the con-
jecture is in fact a limiting case of Davies’ formula (1).
2.1 Trial factors
It is sometimes useful to describe the look-elsewhere effect
in terms of a trial factor, which is the ratio between the prob-
ability of observing the excess at some fixed mass point, to
the probability of observing it anywhere in the range. From
(5), we have
trial# = P(q(θˆ) > c)
P (q(θ) > c)
(8)
≈ 1 + N P(χ
2
s+1 > c)
P (χ2s > c)
(9)
≈ 1 + N
√
c
2
Γ (s/2)
Γ ((s + 1)/2) . (10)
For the case s = 1, √c is just the ‘fixed’ significance,
that is the quantile of a standard Gaussian corresponding to
a p-value of chi-square with s degrees of freedom. This also
holds asymptotically for s > 1, as Zfix = √c + O(s log(c)√c ).
We therefore have, for c 	 s,
trial# ≈ 1 + 1√
2
N Zfix Γ (s/2)
Γ ((s + 1)/2) (11)
which, for the common case of s = 1, is
trial#s=1 ≈ 1 +
√
π
2
N Zfix. (12)
The trial factor is thus asymptotically linear with both the
effective number of independent regions, and with the fixed-
mass significance.
3 Toy model simulations
We shall now illustrate the procedure described above with a
simple example. Our toy model consists of a Gaussian signal
(‘mass bump’) on top of a continuous background that fol-
lows a Rayleigh distribution, in a mass range [0,120]. The
width of the Gaussian increases linearly with the mass, rep-
resenting a case where the mass resolution changes with the
mass.
We assume that the background shape is known but its
normalization is not, so that it is a free parameter in the fit
(i.e. a nuisance parameter), together with the signal location
and normalization. We use a binned profile likelihood ratio
as our test statistic, where the number of events in each bin
is assumed to be Poisson distributed with an expected value
E(ni) = μsi(m) + βbi, (13)
where μ is the signal strength parameter, si(m) corresponds
to a Gaussian located at a mass m, β is the background nor-
malization and bi are fixed and given by the Rayleigh dis-
tribution. For simplicity of notation we will use in the fol-
lowing s = {si} and b = {βbi}. The hypothesis that no signal
exists, or equivalently that μ = 0, will be referred to as the
Fig. 1 (Top) An example
pseudo-experiment with
background only. The solid line
shows the best signal fit, while
the dotted line shows the
background fit. (Bottom) The
likelihood ratio test statistic
q(m). The dotted line marks the
reference level c0 with the
upcrossings marked by the dark
dots. Note the broadening of the
fluctuations as m increases,
reflecting the increase in the
signal Gaussian width
528 Eur. Phys. J. C (2010) 70: 525–530
Fig. 2 (Color online) (Top) Distribution of q(mˆ). (Bottom) Tail prob-
ability of q(mˆ). The solid line shows the result of the Monte Carlo
simulation, the dotted red line is the predicted bound (3) with the es-
timated 〈N(c0)〉 (see text). The yellow band represents the statistical
uncertainty due to the limited sample size
null hypothesis, H0. μˆ and bˆ will denote maximum likeli-
hood estimators while ˆˆb will denote the conditional maxi-
mum likelihood estimator of the background normalization
under the null hypothesis.
The test statistic q(m) is defined as:
q(m) = −2 ln L(bˆ)
L(μˆs(m) + bˆ) , (14)
where L is the likelihood function. An example background-
only pseudo-experiment is shown in Fig. 1, together with
q(m).
Fig. 3 (Color online) The trial factor estimated from toy Monte Carlo
simulations (solid line), with the upper bound of (3) (dotted black line)
and the asymptotic approximation of (12) (dotted red line). The yellow
band represents the statistical uncertainty due to the limited sample
size
We choose a reference level of c0 = 0.5 and estimate the
expected number of upcrossings, as demonstrated in Fig. 1,
from a set of 100 Monte Carlo simulations of background
experiments. This gives 〈N(c0)〉 = 4.34 ± 0.11, which cor-
responds to N = 5.58 ± 0.14. The distribution of q(mˆ) is
then estimated from a sample of ∼1 million background
simulations and we compare the tail probabilities to the pre-
diction of (3). The results are shown in Fig. 2. The bound
of (3) gives an excellent approximation to the observed p-
values for large c.
Figure 3 shows the corresponding trial factor, compared
to the bound calculated from (3) and the asymptotic approx-
imation of (12).
We consider in addition a case where the number of de-
grees of freedom is more than one. For this purpose, we
assume several independent channels, each identical to the
one described above, and where the signal normalizations
(μ1, . . . ,μs) are free parameters. (This could represent, for
example, a case where one is searching for a resonance in
several decay channels, with unknown branching ratios.)
The reference level is chosen to be c0 = s − 1 as discussed
in the previous section. The resulting distributions and trial
factors for s = 2,3 are shown in Figs. 4 and 5. As before,
the bound (3) agrees with the observed p-value, within sta-
tistical variation. The rate at which the asymptotic approx-
imation (11) converges to the bound becomes slower when
the number of degrees of freedom increases, making it less
accurate, however the trend of linear growth is evident.
Eur. Phys. J. C (2010) 70: 525–530 529
Fig. 4 (Color online) (Top) Distribution of q(mˆ) for s = 2,3. (Bottom)
Tail probability of q(mˆ). The solid lines shows the result of the Monte
Carlo simulation, the dotted red lines are the predicted bound (3) with
the estimated 〈N(c0)〉 (see text)
4 Conclusions
The look-elsewhere effect presents a case when the stan-
dard regularity conditions of Wilks’ theorem do not apply,
and so specialized methods are required for estimating tail
probabilities in the large sample limit. Such methods how-
ever exist, and as we have demonstrated, can provide accu-
rate results under fairly general conditions. The procedure
described in this paper consists of estimating the expected
number of upcrossings of the likelihood ratio at a low ref-
erence level using a small set of Monte Carlo simulations.
This can then be related to the expected number of upcross-
ings at a higher level using Davies’ result (2), providing a
bound on the probability of the likelihood ratio exceeding
this level, given by (3). The method is easy to implement
in practical situations, and the bound converges to the ac-
Fig. 5 (Color online) The trial factors estimated from toy Monte Carlo
simulations (solid line), with the upper bound of (3) (dotted black line)
and the asymptotic approximation of (11) (dotted red line). The yellow
band represents the statistical uncertainty due to the limited sample
size
tual tail probability when this probability becomes small. It
has further been shown that the trial factor is asymptotically
proportional to the effective number of independent search
regions and to the fixed-mass significance, allowing for a
simple interpretation of the effect as being the result of two
factors: the first one is the mere fact that there are more avail-
able distinct regions wherein fluctuations can occur, repre-
sented by the effective number of independent regions; the
second effect is that within each region we further maximize
the likelihood by fitting the mass in the neighborhood of the
fluctuation, which can be described by adding a degree of
freedom to the fit.
Acknowledgements We are indebted to Michael Woodroofe, Luc
Demortier and the referee of this paper, for pointing us to the work of
Davies [4] which became the leading thread of this work. We are grate-
ful for the valuable comments made by Luc Demortier in his summary
530 Eur. Phys. J. C (2010) 70: 525–530
talk in the 2010 Banff workshop [9], and to Michael Woodroofe who
spent valuable time in writing to us his impressions on the look else-
where effect. We also thank Bob Cousins for useful comments and
suggestions, BIRS and the organizers of the Banff workshop. One of
us (E.G.) is obliged to the Benoziyo center for High Energy Physics,
to the Israeli Science Foundation (ISF), the Minerva Gesellschaft, and
the German Israeli Foundation (GIF) for supporting this work.
Open Access This article is distributed under the terms of the Cre-
ative Commons Attribution Noncommercial License which permits
any noncommercial use, distribution, and reproduction in any medium,
provided the original author(s) and source are credited.
References
1. L. Lyons, Open statistical issues in particle physics. Ann. Appl.
Stat. 2(3), 887–915 (2008)
2. L. Demortier, P Values and nuisance parameters, in Proceedings of
PHYSTAT-LHC Workshop (2007). CERN–2008–001
3. R.B. Davies, Hypothesis testing when a nuisance parameter is
present only under the alternative. Biometrika 64, 247–254 (1977)
4. R.B. Davies, Hypothesis testing when a nuisance parameter is
present only under the alternative. Biometrika 74, 33–43 (1987)
5. S.S. Wilks, The large-sample distribution of the likelihood ratio for
testing composite hypotheses. Ann. Math. Stat. 9, 60–62 (1938)
6. CMS Physics TDR, Volume II, Appendix A.2. J. Phys. G, Nucl.
Part. Phys. 34, 995–1579 (2007)
7. H. Chernoff, On the distribution of the likelihood ratio. Ann. Math.
Stat. 25, 573–578 (1954)
8. W. Quayle, Combining channels with fits. Talk given at the
PHYSTAT-LHC Workshop (2007). Available at http://phystat-lhc.
web.cern.ch/phystat-lhc/program.html
9. Banff International Research Station Workshop on Statistical issues
relevant to significance of discovery claims (2010). Workshop site:
http://www.birs.ca/events/2010/5-day-workshops/10w5068. Luc
Demortier summary talk: http://www.stat.sfu.ca/lockhart/richard/
banff2010/demortier.pdf


file:///data/core-remote/dit/data/Springer-OA/pdf/c63/aHR0cDovL2xpbmsuc3ByaW5nZXIuY29tLzEwLjExNDAvZXBqYy9zMTAwNTItMDEwLTE0NzAtOC5wZGY=.pdf

Trial factors for the look elsewhere effect in high energy physics

Abstract

Similar works

Full text

Available Versions

Crossref

Springer

Springer - Publisher Connector

Springer - Publisher Connector