We provide brief comments on some common threads arising from the `wishlists' set out in some of the other papers in this volume. The discussion is necessarily incomplete: in particular we have dealt only with points for which a reasonably compact answer seems possible

Cox, D R

Reid, N

CERN Document Server

The Wish-lists: Some CommentsD.R. Cox and N. ReidNufeld College, Oxford and University of Toronto, CanadaAbstractWe provide brief comments on some common threads arising from the ‘wish-lists’ set out in some of the other papers in this volume. The discussion isnecessarily incomplete: in particular we have dealt only with points for whicha reasonably compact answer seems possible.1 IntroductionThe wish-lists are wide-ranging and raise issues of varying difculties, ranging up to the seeminglyimpossible. The following comments concern just some of the topics raised.2 Combination of independent sets of dataIn the simplest situation there are m independent sets of data from each of which a common parameter θcan be estimated, representing for example some constant of interest. Separate analyses of the individualsets give estimates t1, . . . , tm with uncorrelated estimates of the variances s21, . . . , s2m. For an initialdiscussion ignore errors in the s2j .If there are no additional sources of variation and the studies do indeed estimate the same unknown,there is the implicit modeltj = θ + ²j,where the ²j are independent Gaussian errors of zero mean and variances estimated by s2j . The parameterθ is estimated by weighted least squares or equivalently by ordinary least squares applied to a modiedversion of the model, namelytj/sj = θ/sj + ²j/sj,where now the errors have unit variance. The estimate isθ˜ =Σtj/s2jΣ1/s2j,withvar(θ˜) =1Σ1/s2j.Importantly also the residual sum of squares from the modied model, namelyΣ(tj/sj − θ˜/sj)2 = Σt2j/s2j − θ˜2Σ1/s2j ,has under the model a chi-squared distribution with m− 1 degrees of freedom.The argument can be rened, essentially by an empirical Bayes approach, to allow for errorsin estimating the variances. The main practical point is that there can be major drawbacks to givingrelatively high weight to individual estimates that have very small values of s2j arising by chance.Suppose now that the chi-squared test indicates clear heterogeneity, that is, the tj vary too much.There are a number of possible explanations: the internal estimates of variance are unrealistically small119 there may be a small number of anomalous values there may be characteristics of the different studies which if entered into a regression equation forthe tj account for the additional variation.If none of these is applicable and provided m is not too small, for example is at least, say, 10 itmay be reasonable to suppose that there is an additional source of random error producing inter-studyvariation and to replace the starting model bytj = θ + ηj + ²j ,where the ²j are as before and the ηj are independent random variables of zero mean and unknownvariance σ2η .If in fact that variance were known, the least squares estimate of θ isΣtj/(s2j + σ2η)Σ1/(s2j + σ2η)a value intermediate between the simple weighted mean θ˜ and the unweighted mean Σtj/m. The com-ponent of variance σ2η can be estimated by maximum likelihood or, slightly less efciently, by equatinga sum of squares to its expectation. The assumptions involved in this formulation are quite strong andestimation of σ2η is fragile if m is small. When the estimates are based on Poisson-distributed counts,the variances are functions of the counts themselves and this involves some changes in any more renedformulae. The additive representation for additional variation may be better replaced by a multiplicativeform.These issues are treated in more depth in [2].3 Comparison of fit of a small number of modelsSuppose rst there are just two models neither of which is nested within the other. Two broad approachesmight be considered. There are formidable practical difculties in most situations with a Bayesian dis-cussion, not so much connected with specifying the prior probabilities of the two models as with theconditional densities of the (different) parameters within each model. Unless these priors can be speci-ed at least approximately on external evidence there is difculty in computing the posterior probabilitiesrequired for model assessment.A frequentist approach is to test model 1 for departures in the direction of model 2 and compute ap-value. Then switch the roles of the two models. There results information about whether both modelsgive an adequate t in the respect tested, whether one but not the other ts or whether neither model isadequate. In the last case, further analysis to develop an improved model would normally be required.Note that such a possibility cannot be directly obtained from the formal Bayesian approach.With three or more models the best procedure is usually to test model 1, say, in turn against model2 and then model 3 and to take the smaller p-value adjusted for selection as an assessment of model 1,and so on.4 SystematicsMost statistical analysis focuses on random errors, it being assumed that the impact of systematic errorshas been eliminated by design, that is by arranging that the effects of interest are estimated by com-parisons of groups of data equally affected by systematic errors. There is also a substantial literatureon estimating sources of variability in complex measurement systems intended, in particular, to aid thestandardization of measurement techniques.D.R. COX AND N. REID120In the present context these methods are largely not applicable and explicit consideration of sys-tematic errors seems unavoidable. A common approach seems to be to estimate the effect of an estimatedphysical constant on the nal result of interest by re-computing this nal result with the physical constantchanged by plus and minus one standard error. Half the difference between these two resulting valuesis then approximately the derivative of that quantity with respect to the physical constant. This could becombined with other estimated sources of error in a propogation of errors formula, but it is essential tonote that the errors in estimating the constant must be independent of errors from other sources. A lessformal approach would be to investigate the sensitivity to the result of interest to errors in the physicalconstant by re-computing the results over a range of plausible values for the constant.If there are k sources of systematic error and these can be given bounds, taken without loss ofgenerality as say (−∆j,∆j) for j = 1, . . . , k, a very cautious approach is to do 2k possible analysesbased on the set of extreme possibilities, each with its condence limits for the effect of interest and totake the union of these intervals as the basis of inference. Assumptions that the ∆j are random variablesmay be reasonable but the key issue will often concern the independence assumptions involved whichmay have very strong implications. Ref. [3] has given a careful account of these issues from a Bayesianperspective.Systematic errors that are essentially nuisance parameters in a model that is fully specied, oreven partially specied, can be eliminated from the full likelihood by either maximizing over them orby integrating over them, with respect to a weight function. The integration approach is emphasized in[4]. . The maximization approach results in a prole likelihood, discussed in [5], and is implementedas MINOS in MINUIT. The limiting distribution of statistics based on the prole likelihood is the sameas that for a simple likelihood. However the approximations given by this limiting theory, such as theχ2 approximation to twice the log-(prole) likelihood ratio, can be quite inaccurate, especially if thereare large numbers of nuisance parameters. Several adjustments to prole likelihood have been suggestedin the statistical literature (see for example [1], Chs. 2 and 3), to take account of the uncertainty inestimating the nuisance parameters. These adjustments are implicit in the weight function applied inthe integration approach, although the weight function is best thought of as a prior distribution on thenuisance parameters. The choice of the prior is important, and a large body of evidence now indicatesthat at priors on the nuisance parameters are not appropriate, and can lead to very poorly calibratedinference, especially if there are large numbers of nuisance parameters. In some applications it may bepossible to construct an empirical prior distribution from previous observations or from simulations.5 Comparison of alternative test statisticsTests are conventionally assessed by the power curve. In the simplest case of testing a null hypothesis H0that a single parameter θ is equal to θ0 against alternatives θ > θ0, the power curve shows the probabilitythat the test rejects H0 at level α as a function of θ. Equivalently the power curve shows the probabilityof a p-value less than α versus θ. If correctly calibrated the curve should pass through (θ0, α). It is oftena good idea to plot Φ−1(power) against θ, where Φ(x) =∫ x∞{1/√(2pi)} exp(−y2/2)dy is the standardGaussian distribution function. This produces a series of roughly parallel curves, or even approximatelylines, for different α. In comparing two tests the steeper the curves the better.More mathematically for test statistics that are approximately normally distributed we may denethe efcacy of a test T as{ ∂E(T ; θ)∂θ∣∣∣∣θ=θ0}2/var(T ; θ0).This measures the sensitivity of the expectation of T near the null hypothesis relative to the variance.For two test statistics T1, T2 of the same null hypothesis the ratio of their efcacies is the asymp-totic relative efciency (ARE) of T1 relative to T2. Because efcacy usually scales as sample size, theARE compares the sample sizes needed to achieve the same power with the two tests. Thus for testingTHE WISH-LISTS: SOME COMMENTS121the mean of a Gaussian distribution the ARE of the median relative to the mean is 2/pi so that tests basedon the median of n observations and on the mean of 0.63n observations have about the same power.These ideas may be useful even if the properties of the tests are studied primarily by simulation.6 p-values and limitsThe CLs or CLs+b methods combine size and power in a very ad hoc way and are unlikely to havesatisfactory statistical properties. As is emphasized in Neal [4], upper and lower one-sided condencelimits should replace condence intervals, and a full plot of the log-likelihood function is better still.A related point is that the construction of a p-value for discoveries, i.e. for conrming the existenceof a particular effect, should be treated as a separate problem from the establishment of limits on themagnitude of a well-established effect. When there are several parameters of interest, a decision is neededabout whether they can be assessed separately, treating the other parameters as nuisance parameters foreach of these assessments, or whether it is physically more relevant to consider two (or more) of theparameters as forming a single vector. In the latter case approximate p-values can be computed usingthe usual asymptotic theory of likelihood, or a more rened version, but the construction of condenceregions is considerably more difcult and often not very illuminating.References[1] A.R. Brazzale, A.C.Davison, N. Reid, Applied Asymptotics., Cambridge University Press, 2007.[2] D.R. Cox, Encyclopedia of Statistical Science N.L. Johnson and S.Kotz, eds. New York: Wiley 2,45-53, 1982.[3] S. Greenland, Journal of the Royal Statistical Society: Series A 168, 267-306, 2005.[4] R. Neal, in this volume.[5] N. Reid and D.A.S. Fraser, in Proceedings of PHYSTAT2003, L. Lyons, R. Mount, R. Reitmeyer,eds. SLAC e-Conf C030908, 265271, 2003.D.R. COX AND N. REID122

The Wish-lists: some Comments

Abstract

Similar works

Full text

Available Versions

CERN Document Server