We compare the ``unified approach'' for the estimation of upper limits with
an approach based on the Bayes theory, in the special case that no events are
observed. The ``unified approach'' predicts, in this case, an upper limit that
decreases with the increase in the expected level of background. This seems
absurd. On the other hand, the Bayesian approach leads to a result which is
background independent. An explanation of the Bayesian result is presented,
together with suggested reasons for the paradoxical result of the ``unified
approach''.Comment: Latex file (compiled with cernrep.cls included) 5 pages, no figures.
  To appear in the Proceed. of the workshop on ''Confidence limits'' (CERN, Jan
  2000). This paper and related papers also available at
  http://grwav3.roma1.infn.it

Astone, P.

Pizzella, G.

English

arXiv

We compare the ``unified approach'' for the estimation of upper limits withan approach based on the Bayes theory, in the special case that no events areobserved. The ``unified approach'' predicts, in this case, an upper limit thatdecreases with the increase in the expected level of background. This seemsabsurd. On the other hand, the Bayesian approach leads to a result which isbackground independent. An explanation of the Bayesian result is presented,together with suggested reasons for the paradoxical result of the ``unifiedapproach''

Astone, P

Pizzella, G

CERN Document Server

    UPPER LIMITS IN THE CASE THAT ZERO EVENTS AREOBSERVED: AN INTUITIVE SOLUTION TO THE BACKGROUNDDEPENDENCE PUZZLEP. Astone and G. PizzellaSezione INFN Roma 1 (“La Sapienza”), Universita` “Tor Vergata” and LNF Frascati. ItalyAbstractWe compare the “unified approach” for the estimation of upper limits withan approach based on the Bayes theory, in the special case that no events areobserved. The “unified approach” predicts, in this case, an upper limit thatdecreases with the increase in the expected level of background. This seemsabsurd. On the other hand, the Bayesian approach leads to a result which isbackground independent. An explanation of the Bayesian result is presented,together with suggested reasons for the paradoxical result of the “unified ap-proach”.1. INTRODUCTIONThe study of a new phenomenon in science often ends up in a null result. However it might be of greatimportance to set upper limits, as this will help our understanding by eliminating some of the theoriesproposed.The determination of upper limits is presently a hotly debated issue in several fields of physics.Many papers have been devoted to this problem and different solutions have been proposed. In particularthe problem has been discussed in paper [1] (“unified approach”) and, more recently, in papers [2, 3],based on the Bayes’ theory. The use of the “unified approach” (FC) to set upper limits or confidenceintervals is recommended by the PDG [4]. The “unified” and the Bayesian approaches are very different,not only in the sense that they lead to different numerical results but more radically in the meaningthey attribute to the quantities involved. These differences lead to intrinsic problems in any comparisonof their separate results. The purpose of this letter is to try to throw some light on this contentious andimportant issue. We shall show that the Bayesian approach is the correct one. If our argument is acceptedby the scientific community, many debates about upper limits will be clarified.2. THE BACKGROUND DEPENDENCE PUZZLEAccording to the (FC) “unified approach” the upper limit is calculated using a revised version of theclassical Neyman construction for confidence intervals. This approach is usually referred to as the “uni-fied approach to the classical statistical analysis”, and it aims to unify the treatment of upper limits andconfidence intervals. On the Bayes side, according to [2, 3], the upper limit may be calculated usinga function that is proportional to the likelihood. This function is called the ”relative belief updatingratio” and has already been used to analyse data in papers [5, 6]. The procedure has been extensivelydescribed by G. D’ Agostini in [7].Comparison between the two approaches is difficult for the general case. But we have noticed aspecial case which is easier to discuss. In this case the greater efficacy of one approach compared tothe other one seems clear. This case is when the experiment gave no events, even in the presence of abackground greater than zero.When there are zero counts, the predictions obtained with the two methods are different and bothare -intuitively- quite disturbing. Our intuition would, in fact, be satisfied by an upper limit that increaseswith the background level, and this is, in general, the case when the observation gives a number of events199 of the order of the background. However, when zero events are observed, the “unified approach” upperlimit decreases if the background increases (a noisier experiment puts a better upper limit than a less noisyone, which seems absurd) while the Bayesian approach leads to the predictions that a constant upper limitwill be found (the upper limit does not depend on the noise of the experiment). Various papers[8, 9, 10]have been devoted to the problem of solving some intrinsic difficulties with the ”unified” approach:in particular to solving the problem of ”enhancing the physical significance of frequentist confidenceintervals”[8], or to imposing ”stronger classical confidence limits”[9]. In this latter article the proposedmethod ”gives limits that do not depend on background in the case of no observed events” (that is theBayesian result !).In what follows we will give an explanation for the two results.We remind the reader that the physical quantity for which a limit must be found is the events rate(i.e. a gravitational wave burst rate)  . Here we will assume stationary working conditions. For a givenhypothesis  , the number of events which can be observed in the observation time  is described by aPoisson process which has an intensity equal to the sum of that due to background and that due to signal.In general, the main ingredients in our problem are that: we are practically sure about the expected rate of background events 	 but not about thenumber of events that will actually be observed (which will depend on the Poissonian statistics). is the observation time; we have observed a number  of events but, obviously, we do not know how many of these eventshave to be attributed to background and how many (if any) to true signals.Under the stated assumptions, the likelihood isﬀﬂﬁﬃ "!#$!%'&()+*,-.0/$12 (1)We will now concentrate on the solution given by the Bayesian approach.The “relative belief updating ratio”   is defined as: 4355.ﬀ6785697:<;85 (2)This function is proportional to the likelihood and it allows us to infer the probability that -signals will be observed for given priors (using the Bayes’s theorem).Under the hypothesis >=?; if   =<; ,   becomes @355.9ﬁﬃ!( A*/B1DC (3)The upper limit, or -more properly- ”standard sensitivity bound” [7], can then be calculated usingthe function: it is the value EFEF obtained when EFEF33535.ﬀ?;C;G (4)We remark that 5% does not represent a probability, but is a useful way to put a limit independentlyof the priors.Eq. 3 when no events are observed, that is, when  =0, becomes: 4ﬀﬁﬃ!5((5)Thus putting   H; in Eq. 4 we find EFEFIHJCLK8K, independently of the value of the background .200     We will not describe the well known (FC) procedure here, but we would just observe that, accord-ing to this procedure, for M<; and M?; , the upper limit is 3.09 (numerically almost identical to theBayes’ one) NFOQP it decreases as   increases (e.g. for   R; and   AG the upper (FC) limit at 95%CL is 1.47).In an attempt to understand such different behaviour we will now discuss some particular cases.Suppose we have   S; and UTS; . This certainly means that the number of accidentals, whoseaverage value can be determined with any desired accuracy, has undergone a fluctuation. The larger the values, the smaller is the VXWYZ[Y probability that such fluctuations will occur. Thus one could reasonthat it is less likely that a number \)] of real signals could have been associated with a large value of   ,since the observation gave   ?; .According to the Bayesian approach, instead, one cannot ignore the fact that the observation M; has already being made at the time the estimation of the upper limit comes to be calculated. TheBayesian approach requires that, given M?; and +T<; , one evaluates the ^_QV`^ﬁthat a number  \a]of signals exists. This ^_QVb9^ﬁof a possible signal is applied to the observation that has already beenmade.Suppose that we have estimated the average background with a high degree of accuracy, for ex-ample  =10. In the absence of signals, the a priori probability of observing zero events, due just to abackground fluctuation, is given by/M?;Q MA;89ﬁﬃ/%UcCGdA;ﬃe(6)Now, suppose that we have measured zero events, that is   =0. In general   ﬀ*f\a] . It isnow nonsense to ask what the probability that   =0 is, since the experiment has already been made andthe probability is 1.We may ask how the a priori probability would be changed if  \)] signals were added to thebackground. We getE/M?;Q >A;85\a]9ﬁﬃ /%g#/ihFj&(7)It is obvious thatE/can only decrease relative to/, since we are considering models in whichsignal events can only add to noise events1.The right answer is guaranteed if the question is well posed. Given all the previous comments,the most obvious question at this point is: what is that signal  \)] which would have reduced theprobability/by a constant factor, for example 0.05 ?E//d;C;8G+ﬁﬃ/%dﬁﬃ/ihgj (8)Using Eqs. 6, 7 and 8 the solution is:ﬁﬃ/ihFj?;C;G (9)that is:\)]k?JCLK8K (10)1In a gravitational wave experiment signals may add up to the noise with the same phase, thus increasing the energy of thecombined effect, or with a phase opposite to that of the noise, thus reducing the energy. They can in particular add up also tonoise events, even if we expect this to happen with a very low probability, as we know that the events due to the signal are very“rare” compared to the events due to the noise.Anyway, in principle, the presence of this fact will lead to the prediction of a signal rate that increases with the background:in fact the probability that one background event be cancelled by a signal event increases, as l%increases. Thus, if we, at leastin part, attribute the observation of l1=0 to a cancellation of background events due to the signal the final limit on m shouldincrease.In the modelling we usually, as reasonable, consider this effect be negligible. If this is not the case then it must be properlymodelled in the likelihood.201   Now suppose another situation,  =20, thus/nJCAdA;ﬃo. Repeating the previous reasoningwe still get the limit 2.99.The meaning of the Bayesian result is now clear: we do not care about the absolute value of thea priori probability of getting   R; in the presence of noise alone. The observation of   R; meansthat the background gave zero counts by chance. Even if the a priori probability is very small, its valuehas no meaning once it has happened. The fact that the single background measurement turned out tobe zero, either due to a zero average background or due to the observation of a low (a priori) probabilityevent, must not change our prediction concerning possible signals.For .n; we are certain that the number of events due to the background is zero. Clearly thisparticular situation gives more information about the possible signals. In the case   Tp; , instead, it isnot possible to distinguish between background and signal. The mathematical aspect of this is that thePoisson formula when   ?; reduces to the exponential term only, and thus it is possible to separate thetwo contributions, of the signal (unknown) and of the noise (known).We note that the different behaviour of the limit in the unified approach is due to the non-Bayesiancharacter of the reasoning. In such an approach an event that has already occurred is considered “im-probable”: given the observation of   ?; they still consider that the probabilityE/6  <;q 5\)]9ﬁﬃ /%g#/ihgj&(11)decreases as  increases. As a consequence they deduce that to a larger  corresponds a smaller upperlimit \)] .Given the previous considerations, we must now admit that our intuition to expect an upper limitthat increases with increasing background, even when   r; , was wrong. We should have expected topredict a constant signal rate, as a consequence of the observation of zero events, independently of thebackground level.3. CONCLUSIONWe have compared the upper limits obtained with the (FC) “unified” and with the Bayesian procedures,in the case of zero observed events.We believe that the greater efficacy of the Bayesian approach compared to the (FC) method,demonstrated for the case st; , is a strong indication that the Bayesian method -natural, simpleand intuitive- is the correct one. Thus we agree with the proposal in[7] that this method should beadopted by the scientific community for upper limit calculations (see, for example, [11] on upper limitsin gravitational wave experiments).References[1] G.J. Feldman and R.D. Cousins, Unified approach to the classical statistical analysis of smallsignal, Phys.Rev.D 57, 3873 (1998)[2] G. D’ Agostini, Contact interaction scale from Deep Inelastic scattering events- what do the datateach us?, ZEUS note 98-079, November 1998.[3] P. Astone and G. D’Agostini, Inferring the intensity of Poisson processes at the limit of the detectorsensitivity, CERN-EP/99-126 and hep-ex/9909047[4] C. Caso et al., Review of particle physics, Eur. Phys. J. C3 (1998) 1 (http://pdg.lbl.gov/)[5] ZEUS Collaboration, J. Breitweg et al., Search for contact interactions in Deep-Inelasticﬁ#Wvuﬁ#qwscattering at HERA, DESY Report 99-058, xByBz${ByB|$}B~B~B$BBB~ , May 1999, to be published inEur. Phys. J. C.202  [6] G. D’ Agostini and G. Degrassi, Constraints on the Higgs boson mass from direct searches andprecision measurements, internal report DFPD-99/TH/02, xByBzB{$zBxB}$~B~BB$BB . Feb. 1999, to be pub-lished in Eur. Phys. J. C.[7] G. D’Agostini, Confidence limits: what is the problem? is there the solution?, Proceedings of theWorkshop on “Confidence Limits”, CERN, 17-18/1/2000.http://www.cern.ch/CERN/Divisions/EP/Events/CLW/[8] C. Giunti, Enhancing the physical significance of frequentist confidence intervals, Proceedings ofthe Workshop on “Confidence Limits”, CERN, 17-18/1/2000.http://www.cern.ch/CERN/Divisions/EP/Events/CLW/[9] G. Punzi, Stronger classical confidence limits, Proceedings of the Workshop on “Confidence Lim-its”, CERN, 17-18/1/2000.http://www.cern.ch/CERN/Divisions/EP/Events/CLW/ and hep-ex/9912048[10] M. Woodroofe, On the problem of low counts in a signal plus noise model, Proceedings of theWorkshop on “Confidence Limits”, CERN, 17-18/1/2000.http://www.cern.ch/CERN/Divisions/EP/Events/CLW/[11] P. Astone and G. Pizzella, “On upper limits for gravitational radiation”, gr-qc/0001035. 12 Jan2000203  Discussion after talk of Pia Astone. Chairman Peter Igo-Kemenes.Don GroomIf you expect 10 background events and observe zero, wouldn’t you conclude simply that some-thing was broken?P. AstoneClearly the numbers I have used in the example are very high, but an event of very low probabilitymight have occurred. I simply wanted to force you to understand my point, even with a rather extremesituation. In any case it is very easy to get zero events, when the estimated background is, for example,two or three, and the meaning of my example is still the same. I work on gravitational wave detection,and we are now analysing the data of five detectors in coincidence.It really will not surprise me if, inthe presence of a background different from zero, we will measure zero coincidence events, as happenedmany times in the past.As a general comment, I want to say that I am assuming here that your estimation of the back-ground has been well done. It is obvious that we have to control and check the behaviour of our detectorand the procedure to estimate the background!Alex ReadYou said twice something about the expectation of an upper limit that increases with the averagebackground. Don’t you mean, decreases ?P. AstoneIt depends on the assumed working condition. Assume that the observation   is roughly equalto the background   (and this is the situation that usually happens), and that   and   are both muchhigher than the signal. In this case the upper limit increases as   , in the Bayesian approach. In thiscase we have assumed that   increases as the background increases. In contrast, if   is fixed, and thebackground increases, then I agree with you, the upper limit decreases.A. ReadYes, in fact the second case was the thing I was thinking about because you concentrated mostof your talk about zero counts and if you had zero counts and after the experiment you increase thebackground, the limit improves.C. GiuntiI don’t know if I understood well, but you said, for example, if you measure zero events, you wantthat the upper limit in the case of 10 or 20 expected background events is the same. This is what happensalso in the unified approach. If you increase the background the upper limit remains constant, practically.It goes down only in the beginning. For example, if you measure zero events, the upper limit decreasesfrom one to two expected background events, but above that it remains practically constant.P. AstoneIt increases, even if the increase is not very high. In the first transparency I gave a numerical result.For example, if the background is 15, the upper limit decreases from 3 to 1.47. Numerically it is really204  not so important, but I said that I am not interested in comparing the numbers. What is important is notto get a limit of 3 or 2 as we are interested in the order of magnitude of things, so from my point of view,3 or 2 is the same. But I tried to understand the meaning of these different behaviours and the reasoningthat lead to the two results. What I found out is that the difference is due to the non-Bayesian characterof the Feldman-Cousins reasoning: they consider still improbable, an event that has already occurred.On the contrary, using the Bayesian approach, the absolute value of the a priori noise probability is notimportant, but what is important is how it is rescaled once you suppose that the signals do exist.Gunter ZechI think this likelihood ratio is a rather rational approach, and it is neither Bayesian nor classical,but once you cut at a certain value this is somehow arbitrary because, I mean, if you have a long tailor not in the in the likelihood, and cut at 5%, it makes a difference. So all these kinds of approacheshave their problems. In the Poisson case it would be interesting to see if you get a different limit, if youintegrate the Bayesian way or if you cut at a corresponding value of the likelihood. Is there a difference?G. d’AgostiniPerhaps I can comment. First of all, in the cases we are interested in, the likelihood is really steepon the right side. Usually that is the case. As I always say, if it is not the case, publish the likelihood,our and so on (for example the log-likelihood). We have done an exercise to see what happens if youtry to integrate the likelihood in the sense that you assume a flat prior, and the order of magnitude is thesame. We have a table of comparison in our paper. We also give a justification of this uniform prior. It isthe prior that gives the same results - really they are almost identical results - that you would get from aprior which reflects the positive attitude of experimentalists, who are not losing time and money, but theydo research because they hope to see something. If you plug in these kinds of priors, you get essentiallythe same answer as a uniform prior.205

Upper Limits in the Case That Zero Events are Observed: An Intuitive Solution to the Background Dependence Puzzle

Abstract

Similar works

Full text

Available Versions

CERN Document Server