A wish-list of statistics related issues, which were regarded by ATLAS and CMS as requiring a deeper understanding and perhaps the response of a professional statistician, is given

Gross, Eilam

CERN Document Server

ATLAS and CMS Statistics Wish-ListEilam GrossWeizmann Institute, Rehovot, IsraelAbstractA wish-list of statistics related issues, which were regarded by ATLAS andCMS as requiring a deeper understanding and perhaps the response of a pro-fessional statistician, is given.1 IntroductionThe rst Phystat meeting was a workshop at CERN on Condence Limits followed by a similar workshopat Fermilab. Fred James who organized the meeting with Louis Lyons presented then his personal wishlist titled: "What I would like to see" (see Figure 1). Fred wishes that physicists learn the vocabularyof statistics, all searches use Feldman and Cousins unied method [1] to derive condence intervals andBayesian methods are used only in policy decisions. When accepting upon myself to collect a wish listfrom Atlas and CMS my only experience was with LEP statistics [2] and the so called CLs method forderiving limits which was used at LEP. In the two months of preparation of this lecture I had a steeplearning curve during which I partially fullled the rst item in Fred’s wish list but found myself in milddebate with his other two points. However when enquiring around I discovered that most physicists aremainly concerned with old fashioned systematics issues and the majority have only a vague idea of themeaning of the term "Nuisance parameters" and the meaning and difference between the Feldman andCousins vs the prole likelihood methods etc. It became clear to me that my mission in this lecture isnot only to communicate to the statisticians our unsolved difculties but also to make sure that whenAtlas and CMS publish a combined limit or discovery signicance not only the few statisticians amongstthe Physicists (which I propose to call "Phystatisticians") will understand but also the majority of theHEP experimentalists. Therefore I decided to expand the contents of my talk to include also a pedestrianguide to LHC statistics. This guide which also provides the separate title to these proceedings [3]alsoprovides the reader with the statistics background required to understand the wish list. The statisticiansand phystatisticians are exempted from reading it.2 A Wish List of ATLAS and CMSThe wish-list is made of statistics related issues which were regarded by ATLAS and CMS as requiringa deeper understanding and perhaps a professional statistician response.2.1 Modeling of the underlying processThe raison d’etre of our meeting here are 109 Protons that will collide with 109 protons per secondin the 27 km long LHC tunnel. The Proton is made of partons which are Quarks and Gluons. Theunderlying process is a collision between two partons. To understand the process we need to know theParton Distribution Function fi(x;Q2, αs(Q2)) of parton i in a Proton. x is the fraction of momentumcarried by the parton and Q2 is the energy scale squared. To obtain the Parton Distribution Functions aglobal t analysis is performed[4], mainly the CTEQ and MRST, [5]. However some strange phenomenaoccur with these ts. For example, for most of the individual experiments from which the data is takenthe χ2/dof is around 1. However, in the global t, the CTEQ group set ∆χ2 ∼ 100 in order to getreasonable errors [6]. Stump [7] argues: "What we have are estimates on the uncertainties, not the trueones. The increase of χ2 if the estimators are biased or wrong might be bigger than 1. We nd thatalternate pdfs that would be unacceptable differ in χ2 by an amount of order 100". This is a very vaguestatement. Robert Thorne, the "S" in MRST concludes with a wish: "It would be nice to have a more71Fig. 1: Fred James personal wish-list presented in the first Phystat meeting, year 2000.systematic way of accounting for this, e.g. a modied denition of goodness of t to account for non-Gaussian nature of errors, a quantitative way of accounting for theoretical errors, etc..." [8]. Since Throneis elaborating the issue in this conference [6] I decided not to dwell anymore with it, though it is in theheart of proton-proton collisions and must be sorted out in order to reduce the systematics involved withall processes involved.2.2 Why 5σ?The null hypothesis is usually taken to be the background only hypothesis. The alternate hypothesis isthe signal+background hypothesis. When analyzing results in HEP (High Energy Physics) it became ahabit either to reject the signal hypothesis at the 95% Condence Level or announce an observation at the3σ level or a discovery at the 5σ level. Statistically speaking a 5σ discovery corresponds to a uctuationof a Gaussian distributed background expectation at the level of 5.4 · 10−7 (here we adopt the 2-sidedinterpretation). At that level one is probing tails of distributions. In order to make such statements oneneeds to understand the data and the detector response to that level. Nobody really knows where thishabit of 5σ was born. A back of the envelope calculation reveals a possible explanation which has to dowith the "look elsewhere effect". Suppose when searching for a new phenomena (Higgs boson...) oneis combining 100 search channels each with a discriminating variable distributed within 100 resolutionbins. The false discovery rate of 5σ will be 104 · 2.7 · 10−7 = 0.27%. This degrades the discoverysensitivity to 3σ. More examples and insights into this problem can be found in [9, 10, 11].So is there a way to clarify how many σs are needed for discovery? Is there a problem in seekingfor an effect at at tail of a pdf? How can we take a uctuation into account?2.3 Look Elsewhere Effect in TimeAn ongoing discussion in the LHC collaborations is the need and possibility to perform a blind analysis.Even if from the scientic integrity point of view the pros are clear, with each collaboration having over2000 physicists it is hard to believe such a habit can be adopted. Moreover, it will take years till theE. GROSS72detectors are understood, and understanding the detector necessitates looking at the data as it comes.However, that raises another issue, the issue of sequential analysis. Should a statistical stopping rulebe established? Is it possible that by adopting a stopping rule we would achieve a discovery with lessluminosity than needed with a blind analysis? Understanding stopping rules is a complicated issue. It isprobably more relevant in medical experiments where one would like to minimize the damage that mightbe referred to as patients trying new drugs. No doubt the HEP community must adopt very strict rules forlooking at the data and publishing results to minimize human bias. The HEP Physicist might be botheredby another related issue, which might be refereed to as a "look elsewhere in time" effect: How much thep-value is increased as a result of the fact that we have already looked at the data a few times before andgot no satisfactory signicance?2.4 Estimating SystematicsThere are two issues related to systematics. Classifying and estimating them and implementing them inthe analysis. Implementation of the systematics in the statistical hypothesis tests is discussed at length in[3]. Knowing the type of error one is dealing with is very important and make its estimation clearer. Sin-ervo [12] classied the sytematics into three types: Class I: Statistics-like uncertainties that are reducedwith increasing statistics. Example: Calibration constants for a detector whose precision of (auxiliary)measurement is statistics limited. Class II: Systematic uncertainties that arise from one’s limited knowl-edge of some data features and cannot be constrained by auxiliary measurements. One has to make someassumptions. Example: Background uncertainties due to fakes, isolation criteria in QCD events, shapeuncertainties. These uncertainties do not normally scale down with increasing statistics. Class III: The"Bayesian" kind. The theoretically motivated ones, uncertainties in the model, Parton Distribution Func-tions, Hadronization Models. The most accurate way to communicate the systematic error is to separateone type from another and quote them separately. Some bad habits should cease. For example addingall sorts of systematics in quadrature and quote only the nal result.Another unfortunate habit in estimating systematics is when Physicists do not differentiate be-tween cross-checks and identifying the sources of the systematic uncertainty. For example we shift cutsaround and measure the effect on the observable. Very often the observed variation is dominated simplyby the statistical uncertainty in the measurement [13].2.5 Reference Priors in Demand of a CodeAnalytical derivation of reference priors might be technically complicated. Bernardo [14] proposes analgorithm (pseudo code) to obtain a numerical approximation to the reference prior in the simple caseof a one parameter model. The pseudo code should work for any number of parameters (of interest andnuisance) provided you make non informative priors for ALL! If the code could be extended to multipleparameters, including some with informative priors, it would be more useful for the HEP community.Another complication is that the order of the parameters matter. This should be further investigated andclaried. The wish is to have a generalized routine (REAL CODE) to numerically calculate referencepriors for parameters {θ} given the Likelihood L({θ}) as an input.2.6 Subsequent InferenceOften the background distribution is tted with a polynomial,∑ni=0 aixi with the degree n determinedwith a stepwise test. However, the tted coefcients ai were obtained as if we know a priori the degreeof the polynomial. How does one take the prior test into account? Perhaps the degree was wrong to startwith?ATLAS AND CMS STATISTICS WISH-LIST73Table 1: Hypothesis Test Methods. The columns indicate if the method obeys the Likelihood Principle (LP), if ithas a coverage and if it uses priors.LP Coverage Priors Comments1 F&C No Yes No Pioneering in HEP2 F&H&C2 No No Yes3 Prole Likelihood (PL) Yes Asymptotic No  	4 PL F&C Construction No Satisfactory No5 PL Full Construction No Yes No Cumbersome6 Bayesian Yes No Yes Choose priors with care7 CLs with C&H Yes Partial Yes For upper limits only2.7 Multivariate AnalysisThe number of Physicists objecting to MultiVariate (MV) analyses (like ANN, Decision Trees) is gettingsmaller as the average year of birth of the active physicists go up. Evaluating the systematics with MVanalyzes is very unclear. Many physicists have the habit of changing the input parameter by what theybelieve is a standard deviation, do it one at a time or randomly with all of them together. There mustbe a better way to do it. Can the community come up with good gures of merit for the robustnessoptimization of a MV analysis (and not only for the signicance )?Note that the articles by Linnemann (’A pitfall in estimating systematic errors’) and by Reid(’Some aspects of experiment design’) in these proceedings also deal with this issue of problems withchanging one variable at a time.2.8 Telling Between Multi HypothesesIs the scalar particle we have just discovered a Standard Model one, a CP-odd SUSY one or a CP-evenSUSY one [15]? Here are three hypotheses regarding the nature of the Higgs Boson. The NeymanPearson lemma tell us the best test statistic to tell between two simple hypotheses. In case of more thanone equivalent alternate hypotheses, what is the best test statistics to use besides testing them one againstthe other? Is there anyway to do it without a Bayesian assumption that all hypotheses have an a prioridegree of belief?2.9 Hypothesis TestTesting a preferred hypothesis includes the estimation and incorporation of systematic errors. The resultis then interpreted in terms of exclusion, measurement or discovery. Hypothesis testing is a science byitself. The LEP collaboration has chosen the CLs method integrating out the systematics using the C&Hmethod. This was one possibility out of many. In the years since LEP we have grown up to understandmany more methods. The frequently used ones in HEP are introduced in [3] and compared in Table 1.Since each method has its pros and cons the ATLAS and CMS combined statistics forum has expressedits wish that the analyses be interpreted in a few of them. The frequentist based Prole Likelihood and aBayesian method are highly recommended. CLs will certainly be practised for exclusion (the power ofa habit...). It is also recommended to try one of the Neyman construction methods. If all methods agreea trust in the result will be established, however, if one method gives completely different inference fromothers, this should be further investigated.3 Conclusion and a Personal WishSome of the statistical issues raised in this paper are quite picky. The level of the discussions on hy-pothesis tests is quite advanced, which indicates that the HEP community’s understanding of statisticshas matured since the days of LEP (thanks in large measure to the Phystat meetings). It is therefore myE. GROSS74personal wish that when the real data analysis phase arrives (one hopes soon) every physicist will makethe effort to become a Phystatistician to some degree, so he or she understands what is a p-value, what isProle Likelihood, a Condence Limit, a Condence Interval etc.4 AcknowledgementsThis work would have never been possible without the patience and amazing support of Bob Cousins,Kyle Cranmer and Glen Cowan. They helped me to become as close as possible to a Phystatistician intwo months.I would also like to thank Alex Read, Bill Quayle, Luc Demortier and my student Ofer Vitells foralways being there to answer my questions and requests. Last but not least, applause to Louis Lyons forkeeping the Phystat meetings going. These meetings are the ultimate source of statistics for the HighEnergy Physicists. Long may these meetings live! Toda Louis. I am obliged to the Benoziyo center forHigh Energy Physics for their support of this work. This work was also supported by the Israeli Sci-ence Foundation(ISF), by the Minerva Gesellschaft and by the Federal Ministry of Education, Science,Research and Technology (BMBF) within the framework of the German-Israeli Project Cooperation inFuture-Oriented Topics(DIP).References[1] Gary J. Feldman and Robert D. Cousins. A unied approach to the classical statistical analysis ofsmall signals. Phys. Rev., D57:3873-3889, 1998.[2] Eilam Gross and Amit Klier, Higgs Statistics for Pedestrians, hep-ex/0211058.[3] Eilam Gross, LHC Statistics for pedestrians, these proceedings.[4] A nice introduction can be found in John Pumplin’s talk: Light FrontPhysics and Parton Distributions (Talk at LC2006 Minneapolis May 17, 2006);  		    	   [5] A CTEQ and MRST Parton Distribution Functions generator can be found in  		ﬀ   	 	 		ﬁﬂﬃ  [6] Robert Throne in these proceedings.[7] Daniel Stump, Uncertainties of Parton Distribution Functions, Phystat2003.[8] Private Communication with Robert Throne.[9] See talk by Kyle Cranmer in these proceedings.[10] Gary Feldman, Concluding Remarks: Phystat 2005[11] 95% CL limits and 5σ discoveries in CMS TDR Appendix A. See also talk by Alexey Drozdetskiy,this conference.[12] P. Sinervo, Denition and treatment of systematic uncertainties in high energy physics and astro-physics PhyStat2003.[13] See an excellent discussion on estimating sysyematics in the 2000 SLUO Lec-tures on Statistics and Numerical Methods in HEP, lecture number 5, Roger Barlow,  	 !ﬀ 	  	" 	ﬂ  	#  	$  $%  &ﬃ[14] Berger, J. O. and Bernardo, J. M. (1989). Estimating a product of means: Bayesian analysis withreference priors. J. American Statistical Association 84, 200-207; Berger, J. O. and Bernardo, J. M.(1992). On the development of reference priors. Bayesian Statistics 4 (J. M. Bernardo, J. O. Berger,D. V. Lindley and A. F. M. Smith, eds). Oxford: Oxford University Press, 61-77 (with discussion).See also Luc Demortier, Bayesian Reference Analysis for Particle Physics, Phystat05.[15] D. Rainwater, D. Zeppenfeld and K. Hagiwara, Phys . Rev. D 59, 014037 (1999); T. Plehn, D.Rainwater and D. Zeppenfeld, Phys. Lett. B 454, 297 (1999); Phys. Rev. D 61, 093005 (2000).ATLAS AND CMS STATISTICS WISH-LIST75

ATLAS and CMS Statistics Wish-List

Abstract

Similar works

Full text

Available Versions

CERN Document Server