Nolty, R

CERN Document Server

    DRAWING SENSITIVITY CURVES: ASK A SILLY QUESTION, GETA SILLY ANSWERR. G. NoltyCalifornia Institute of TechnologyAbstractSensitivity plots are meant to answer the question, “If the experiment wereto see what is expected, what kind of confidence limits could the experimentbe expected to set?” In practice this may be done naively by assuming theexperiment will produce the data vector V which is the most likely data vectorpredicted for parameters X (for example, the parameters may be oscillationparameters, 	and the data vector may be the number of events inangular bins) and drawing the C.L. curve in parameter space for experimentalresult V. It is often the case that some other parameters  (that are “close” toX in parameter space) will predict a most likely data vector  that is extremelysimilar to V. In that case, if the experiment produced data vector V, the theorywould be excluded only at a very small C.L. (e.g., perhaps 5%). However, inreal experiments with limited statistics, if the true parameters of nature were X,the experiment would measure a data vector that is fluctuated from V, and thetheory would probably be excluded at a much higher C.L. (typically greaterthan 50%). Thus, the naive method does not do a good job of answering theessential question. I will present a simple algorithm that takes fluctuationsinto account when drawing sensitivity curves, and illustrate it with an examplefrom atmospheric neutrino oscillations.As far as I know, the concept of sensitivity was first formally defined in the 1998 paper of Feldmanand Cousins [1], where it was defined as “the average upper limit that would be obtained by an ensembleof experiments with the expected background and no true signal”. Informally, however, the definitionis broader and includes sensitivity to non-null hypotheses. Perhaps a good statement of the broaderinformal usage of the word sensitivity is “the limit that an experiment would set if it saw what wasexpected”, where the expectation may refer to a null or a non-null hypothesis. For example, a proposalfor a new neutrino oscillations experiment may include a plot labeled “sensitivity” which shows the limitthat would be set if the experiment saw the data predicted by the Super Kamiokande best fit parameters.There is also at least one case in which a paper presenting an experimental result [2] included a graphlabeled “sensitivity”, which showed the limit that would have been expected a priori assuming the trueparameters of nature were the best fit parameters which had been obtained by that very experiment. (Inthat case, the sensitivity was shown because the data vector that was actually obtained had a fairly poorfit to all hypotheses, including the best fit. The 90% C.L. curve was thus much more restrictive than thea priori expectation, and the experimenters felt it was only honest to point this out on the results plot.)The point of this paper is to point out a simple trap the experimentalist may fall into when com-puting so-called sensitivity curves for complicated experiments. Let us consider an atmospheric neutrinooscillations experiment in which  events are accumulated in a number of angular bins. Using a modelfor neutrino flux and cross sections, and for a particular oscillations hypothesis (i.e. a particular choice ofoscillations parameters), the experimentalist may predict the number of events that will show up in eachbin. Then she may say to herself, “Let me assume that the experiment measures exactly the predictednumber in each bin. What confidence limits curve would I then draw in parameter space?”Figure 1 shows a possible result. The figure is based on a rough approximation to the MACROexperiment, with a fairly short running period of just a couple of years. The “data” from which the graph167    :0/01/14   14.50-5-4-3-2-1010 0.2 0.4 0.6 0.8 1Fig. 1: Plot of exclusion levels using naive algorithm. The plot consists of a grid of points in parameter space. The x-axisis ﬁﬀﬃﬂ "!$#&%(' with intervals of 0.1. The y-axis is )+*-,.0/!$1324"' with intervals of 0.5. At each grid point, the size of the box isproportional to the confidence level at which that hypothesis is excluded (to be more precise, the lowest confidence level atwhich that hypothesis is not excluded). All of the black squares are greater than 90%, so all of those hypotheses are excluded at90% C.L. This graph was computed for an experiment which measured exactly the prediction for 13246587:9<;&= >?0@$ACBDE!ﬁ#"%F'G57IH 9was produced was the exact prediction for KJMLNPOQSR TU"VSW XYZJML[\N. The figure consists of a gridof test-point hypotheses. At each grid point, the size of the square is proportional to the confidence levelat which that hypothesis is excluded. The black squares are larger than 90%, and thus those hypothesesare excluded at the 90% C.L.While the 90% C.L. exclusion curve that can be estimated from this figure appears reasonable,I wish to draw your attention to the extremely small squares running along a curve from ]^JLNOQSR TUEVSW X_J`L[\Nto JaLNPbcU"VFW X_J`N[\d. For example, the hypothesis JaLNPb;VSW XYZJeN[\dis excluded only at the 5% C.L. This does not seem plausible for a real experiment.The source of the problem is made apparent in Figure 2. In this figure, the data bins (the predictionfor fJ^LNPOQSR TU"VSW XYgJhL[\N ) are given by the thick line, while the thin line with error bars givesthe prediction for a test hyptothesis ]JiLNPbcU"VSW XjJkN[\d. While a large statistics experimentcould distinguish between the two due to their different slopes, at the current statistics, a simple chi-squared evaluation would show the “data” agreeing with the test hypothesis much better than a datasetgenerated with fluctuations from the test hypothesis could be expected to. To be quantitative, only about5% of datasets generated from the test hypothesis with fluctuations score better than the “data” generatedwithout fluctuations from the default hypothesis ]KJeLNOQSR TUEVSW XZJeL[\N.However, it is not likely that the real experiment would produce data in such close agreementwith both predictions. If the default hypothesis were true, the experiment would produce data fluctuatedfrom the default prediction. This data will probably agree with the test hypothesis much less than theunfluctuated data does, and thus the test hypothesis will probably be excluded at a greater C.L. thanthe sensitivity calculation shows. Thus, our naive algorithm does a poor job of answering the informalquestion, “What limit would the experiment set if it saw what was expected?”Our naive experimenter has missed a point which Feldman and Cousins got right – a sensitivitycalculation should be based on an ensemble of (fluctuated) experiments. I propose an extended definition168    :0/01/14   19.460l1020m30n40o50p60q0l 1 2 3n 4 5p 6q 7 8r 9sFig. 2: The predicted bins for two hypotheses. The x-axis is bin-number for a histogram in tEuc@-!+%wv&x0ycz|{\}I' , with vertical to the left.The thick line is the prediction for the default hypothesis 1324~57:9<;P:= >E?0@ABDE!$#&%('657IH 9 . This was taken as the data vectormeasured by the experiment. The thin line with error bars is the prediction for a test point hypothesis 1324657&9 . ?0@$ACBDE!$#&%F'G59H . The error bars shown are  ﬂ only.of sensitivity, quantitatively slightly different from that of Feldman and Cousins but similar in spirit andeasier to calculate in a multi-dimensional parameter space. “The sensitivity of an experiment to a defaulthypothesis is given by computing, at all test point hypotheses in parameter space, the average (over anensemble of experiments fluctuated from the default hypothesis predictions) of the lowest confidencelevel at which the test point is not excluded.”The application of this prescription results in the exclusion plot in Figure 3. Qualitatively, formy example, the 90% C.L. curve is changed only a little. However, curves at lower confidence levelsare changed drastically. No hypothesis, including the default hypothesis, has an exclusion level of lessthan about 50% C.L. This makes sense; if you conduct the experiment once, the data vector you getwill typically agree with the default hypothesis better than about 50% of the data vectors that could begenerated from that hypothesis.In pseudocode, the naive algorithm can be represented as<Q<Q<QŁQ<e<e<<e<<6ŁQ<<6<Łw<Q<e<<<Q<<e<<Q<<<6<<Q<6ŁQe<w<Q<<Q<<Q<<Q<<< <Łw<<e<e<Q<Q<6e<Q<Q<<Q<<Q<e¡¢w<<Q<Q <<Q<e<<<e<Łw<<e£Z¤¥<<Łw<<Q<<e<<<e<6Q<e<Q<<<Q<< <<Q<< <ŁQ<Q£e<<e<<e<6Q<e<Q<<<Q<<  ¦¢QŁ<Q<e£¢Q¨§Q<<<Q<Q< <Ł<Q<eZ¤©Łw<<<Q<e¢ŁQ<<<Q<<Q<<e<w<<e<<e<Q¨<6QQZª8Q«<¡The score may be the likelihood of the data given the hypothesis, the Feldman-Cousins ratio oflikelihood to maximum likelihood, or some other measure.169      :0/01/14   17.41-5-4-3-2-10¬10¬ 0.2¬ 0.4¬ 0.6¬ 0.8¬ 1Fig. 3: Plot of exclusion levels using ensemble algorithm.My ensemble algorithm may be pseudocoded asQ¢©­¢®¤¯°<<Q<e<Q<QŁ<QQ±¤¥<<Łw<<Q<<e<<<e<Q<<<Q<Q<<Q<<QQ<6ŁQe<w<Q<<Q<<Q<<Q<<< <Łw<<e<QŁe<±©<<e<Q<6QQ<Q<<<Q<Q<e¡¢w<<Q<Q <<Q<e<<<e<Łw<<e£Z¤¥<<Łw<<Q<<e<<<e<6Q<e<Q<<<Q<< <<Q<< <ŁQ<Q£e<<e<<e<6Q<e<Q<<<Q<< < ¨­ ®¤E¯   ¦¢QŁ<<e<e£¢Q¨§<Q<<<e<Q<¢QŁ<<e<e<±6¤¨6Ł<Q<<<e¢Łw<<Q<<e<<Q<<e<w<<e<<e<Q¨<6QQZª8Q«Z²E¡³w¯´References[1] G.Feldman and R.Cousins, Phys. Rev. D57 (1998)3873.[2] MACRO Collaboration, M. Ambrosio et al., Phys. Lett. B434 (1998)451.170  Discussion after talk of Robert Nolty. Chairman: Peter Igo-Kemenes.Jacques BouchezDon’t you think it would be better to publish a median sensitivity curve rather than the mean? Themedian, that is 50% of the people would find a worse result and 50% a better result, because it avoidsthis problem of metric dependence.H. ProsperOf course, you’re quite right in saying that you should use an ensemble, but the result of coursedepends upon the ensemble that you’ve used, and depends what you assume to be random and what youassume to be fixed. So in this particular calculation, what did you assume to be fixed and what did youassume to be random?R. NoltyI don’t know how interesting this is, but I’ll do my best to answer that. I assumed that the os-cillation parameters were fixed at this value [points on the screen], and I assumed that the absolutenormalization was not known, and I treated it as if it were a Gaussian with the mean suggested by ourdefault cross-section calculation and the Bahcall neutrino fluxes, as if it had a Gaussian shape with theerrors quoted by the authors of those two models.J. LinnemannWere you using only chi-squared to distinguish the theory curves in your angular plot? There areother tests which are more sensitive to the slope, such as the Kolmogorov-Smirnov test.R. NoltyIn this case I used a complicated prescription that MACRO had come up with, but essentially itwas chi-squared. So, maybe other statistics would have done a better job of discriminating these two.171

Drawing sensitivity curves: ask a silly question, get a silly answer

Abstract

Similar works

Full text

Available Versions

CERN Document Server