Methods for calculating the statistical significance of excess events and the interpretation of the formally derived values are discussed. It is argued that a simple formula for a conservative estimate should generally be used in order to provide a common understanding of quoted values

Staubert, R.

English

NASA Technical Reports Server

477
OG 9.5-10
ON THE STATISTICAL SIGNIFICANCE OF EXCESSEVENTS- REMARKSOF CAUTION
AND THE NEEDFORA STANDARDMETHODOF CALCULATION
R. Staubert
Astronomisches Institut der Universit_t TUbingen, TUbingen, Germany
ABSTRACT
Methods for calculating the statistical significance
of excess events and the interpretation of the
formally derived values are discussed. It is argued
that a simple formula for a conservative estimate
should generally be used in order to provide a common
understanding of quoted values.
1. Introduction. Substantial nonuniformity exists in the cosmic ray
literature with respect to how the statistical significance of features or
excess events is being calculated (e.g. point sources, spectral lines,
light curves). Consequently, there is no mutual understanding about what
the confidence in some result might really be when a number of 'standard
deviations' are being quoted. Some of the proposed procedures for
calculation need to be taken with caution. On the other hand, there is a
clear need for the adoption of a standard methoa to allow the reliable
intercomparison of quoted results and create a common understanding of the
associated confidence.
A number of methods and formulae have been proposed together with
sometimes extended mathematical derivation or justification (Ref. I-4_.
It has become clear however, that some of these methods need to,taken with
caution. On the other hand there is a very slmple formula which is being
widely used by X-ray astronomers providing a common understanding.
2. Statistical Siqnificance. An example for the statistical situation
which we like to discuss is given in Figure 1.
xi Numbers of events x_are plotted versus binnumber i = i .... n, corresponding to
intervals of some physical variable (e.g.
energy, phase, electric charge, time,
...). In the example given there seem to
be 'excess events' in bins 1 and 2 as
compared to the 'background' defined by
the other bihs. The excess is
J--l____. ON -_OFF,
when ON and OFF are the integrated countsin channels 1 to 2 and in channels 3 to n,
' ' ...... respectively and K is the ratio of the
1 21= _I corresponding number of bins, herelon off 2/(n-2).
Fig. I. Statistical example.
https://ntrs.nasa.gov/search.jsp?R=19850026674 2020-03-20T16:46:35+00:00Z
478
OG 9.5-10
The ge:,r_er"al ClUest:i. ons then are:
1.. Does 'the excess c.orre.spond t.o the presenr'.e _._ a physical
signal ?
2. What i.:!.'.. 'the 'sign:i. Ticance o.f the rs.ignal'?
It is important 'Lo d:i.!st.inguish between these two q,4est, ior_s,.
They correspond t(.': the assumption that one out o.F t.wo
all.emotive hypot.hesis is true:
.... t.he nL_l]. I'_ypothc, s:Ls HO "i.s,_ that. there really i_.s only
backgr"¢._und _
.... the hypothesis H1 is, that ,.'atrue signal ex'i. sts in
addition to backgr-oLu_d,
Whe:n a st.atement, is made about astat, ist:i, cal situat.ion_ it
should be clear uncler which hypcrthesis this statemerYt holds.
The .first o.f the two qu_estioI3S may be answered by giving the
pr'c_b,-._b:L].ity _or a chance occur'once ¢::)_ t'.he observed el.,cess by
a stati st :i.c:al T 1u.ctuati on (under HO)_ It is oT course
necessary 't.o use the. proper statistic (e.g. binomial
statistic Tor small numbers (mr: _-:vents). If a low probab:i.l:i, ty
.for' the chance cx::curance o.f 'excess events' is .fo,...,.nd_ it :i.s
t.hen u.sually concluded that the presenc:e oT a 'physical
signal" :i.s likely. From there on hypot.hesis one is advoc:ated
and all stat.ements m_:_d_.!.:shc)(lld r'e.fer to HI°
Only t.,.nder" H1 i'.he t.erm s:i. gni.fi,:::ance !._hou.ld be used. In
par't:i, cular the oTten used .for'mula (ON ... _(OFF)/ 0( _ is
usr'::l, ess (a._s are ,':_ number o..F or:her .formulae,j see e.g. (4)).
Also the probability which answers the Tirs'l: qute,:.ltion should
nat: be conve.rt'.ed into a s:i. gn:i..fic:ance (as is sometimes done
by using the integrated Gaussian distr:Lbutic)n,, even in cases
where the Gaussian statisti(: does not apply).
In answering the: se:c:ond question then the pre:sence o'f a
signal is assume.d (HI). The ':_igni.Fic:ance o.f the signal' k
can be deTined as the ratio oT the best estimate o.f 'the
s'i. gnal to its u.nc:_-.:r'l::a:Lnty. In 'the case o.f Po:i. ssoni an
c:ounting st.atistic: .for which the variance is equal "L:i) the
mean a stra:i, ghtTorwa.r'd er'ror i::)r'ol:)agat:i, on leads to the well
known Tormula (in t:erms o.f the above deTined variables):
ON .....O( OFF
'sigr_i.f:i.c:anc:e' I.:: :::: lOW + ¢XZOFF_ [:L_
in units 04: standard deviations 0" (see Re.f, (3.,5,6,7); note
'that in (3) t.his ,formula is interpreted incorrectly).
Formula [1] may. be also derived by using the mor_!
complicated maximum ].:Lkelyhood ratio (6).
4.79 OG 9.5-Z0
A general c:ri'Lic:ism o.f the _or"k oq (3) and to som(_': extent, o_:
(1) and (2) is given in (6). While-. _ it is v_,_::ry impor-t.an'L., nrJt.
t.o overestimate statistical s:i. gr_:_.ficar_ce, R(.:.:.H:. (3) does too
much,, leading to an underestimat, r_,
More recervL.].y, (.5) has contributed signi.ficantly to t.he
cor_.fLu!sion by t.r"ying to show t.hat, formula [:1.] is :i.ncc)rrect
and ,shou],d be r"ep:l._'_c;ed by another c-omplicat, ed .formula. The
main ar'gument, is that the new .formula ,f::i.ts much better L.o
Mont.e Car],o simulavL:i, ons t,han .formula [:1.} does. The whole
discussion is misleading and '..su.fqr_rs .from the 'fact that no
distinction bet.ween H1 and HC) is made_ while formula [1]
re,fers t.o H1 the Monte Carlo simu].ations as well as the new
,formula re.for t.o HO,_ so their d:i.s'L'ributions are necessarily
di q _ereni:. ,,
For _ the example given in Fig. 1 (with a u.n:Lt, oq 1 ,for the
scale oq counts x .) the two quesi:ions can be answered asl
.1.. The probab:i,l,it.y (L_nder HO) .{:or a c:hanc'.e occLu"anc:e o_ 1.4
events in bins 1 and 2. whii:h an aw_rage rate o.f 6 in two
bins is _ :L0-._;, using binomial st at.i st :i.c: (note that.
Pois,',_onian statistic: gives the somewhat larger probability
o,f ".3,,6 x 10-3) .
2. Ill one {eels th_':l._t the prc)bability o'f :L()-3 is low enoL%lh
to postulate the e×ist.enc:e oT a phys.i.r.:al signal (Hi.),, then
th@ !.signi_:icance oq thissi:i, gnal is
14 .... (2/10) 16
k =_4TI + (2/10) _ 16 i = 2,,,6st.andard devi.:,_tJ,ons.
To put it in other words again we consider Figure 2.
F(x)
/
I::'J. g u r e "._
F'e p r' ese.,n t. a t :Lo n
oq eveni: number
_ d J. str i b tJ.E j. Or'IS.
_ ,OFF ON
480 OG 9.5-10
If the signal ON ,.- (X OFF is compared to the standard
deviation o.f the background O(_one gets an estimate for'
the chance (_ccurance under the null hypothesis HO.
I.F,,(:inthe other hand, ON ". _OFF is c.:ompared to both the
standard deviations _.f:the background and the signal, as is
done by _ormula [i] under HI,, _Ine gets a different estimate.
This is related to the probability that a second measurement
(under identical conditions) _ill lead to a null result (ON
%< (_ OFF)., It. is this estimate that should be called
'signi.Fic.ance of the dete_:::tedsignal"
3. Final remarks
Val_.,.es(:H:signi-l:ic::ances in units o._ standard deviations are
usually quoted _hen the detection of some signal is claimed.
Consequently, a .{:ormula refering to HI (existence of a
signal) should be used.
Formula _I] has been widely adopted by X-ray astronomers
and has as such served suc:cessfully as a standard allowin 9
the reliable intercompari s(_n _:_f stated values of
signi.f:icance. It is up to the individual 'From what level of
signiFic._ance (inward one starts to 'believe' in some reported
result. Our pers(_nal view is that using formula [I_ a
minimum signi.6icanc:e o.F 3 standard deviations (better yet 5)
should be rea(:hed.
Re.{:erences
I. Hearn, D. 1969, Nuc:l.lnstr. Methods 79, 200
2. O'Mongain, E. 1973, Nature 241,, 376
3. Cherry,, M.L. et al. 1980, Ap.J. 242, 1257
4. Li, T.P. and Ma, Y.Q_ 1983, Ap.J. 272, 317
5. Li, T.P. and Ma, Y.Q. 1983, Proc.18th Int. Cosmic Ray
Conf., Bangalore, XG 3-7, p.76
6.. Nolan, P.L. 1982, preprint UCSD SP-82-15
7. Peterson., L. 1975, ARAA 13, 423


On the statistical significance of excess events: Remarks of caution and the need for a standard method of calculation

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server