Discussion of "Statistical Inference: The Big Picture" by R. E. Kass
[arXiv:1106.2895]Comment: Published in at http://dx.doi.org/10.1214/11-STS337C the Statistical
  Science (http://www.imstat.org/sts/) by the Institute of Mathematical
  Statistics (http://www.imstat.org

Gelman, Andrew

English

arXiv

I agree with Rob Kass’ point that we can and should make use of statistical methods developed under different philosophies, and I am happy to take the opportunity to elaborate on some of his arguments

Gelman, Andrew E.

Columbia University Academic Commons

Statistical Science2011, Vol. 26, No. 1, 10–11DOI: 10.1214/11-STS337CMain article DOI: 10.1214/10-STS337© Institute of Mathematical Statistics, 2011Bayesian Statistical PragmatismAndrew GelmanI agree with Rob Kass’ point that we can and shouldmake use of statistical methods developed under differ-ent philosophies, and I am happy to take the opportu-nity to elaborate on some of his arguments.FOUNDATIONS OF PROBABILITYKass describes probability theory as anchored uponphysical randomization (coin flips, die rolls and thelike) but being useful more generally as a mathematicalmodel. I completely agree but would also add anotheranchoring point: calibration. Calibration of probabil-ity assessments is an objective, not subjective process,although some subjectivity (or scientific judgment) isnecessarily involved in the choice of events used inthe calibration. In that way, Bayesian probability cal-ibration is closely connected to frequentist probabilitystatements, in that both are conditional on “referencesets” of comparable events. We discuss these issuesfurther in Chapter 1 of Bayesian Data Analysis, featur-ing examples from sports betting and record linkage.CONFIDENCE INTERVALS AND HYPOTHESISTESTSI agree with Kass that confidence and statisticalsignificance are “valuable inferential tools.” They aretreated differently in classical and Bayesian statistics,however. In the Neyman–Pearson theory of inference,confidence and statistical significance are two sides ofthe same coin, with a confidence interval being the setof parameter values not rejected by a significance test.Unfortunately, this approach falls apart (or, at the veryleast, is extremely difficult) in problems with high-dimensional parameter spaces that are characteristic ofmy own applied work in social science and environ-mental health.In a modern Bayesian approach, confidence intervalsand hypothesis testing are both important but are notisomorphic; they represent two different steps of infer-ence. Confidence statements, or posterior intervals, areAndrew Gelman is Professor, Department of Statistics andDepartment of Political Science, Columbia University, NewYork, New York 10027, USA (e-mail:gelman@stat.columbia.edu).summaries of inference about parameters conditionalon an assumed model. Hypothesis testing—or, moregenerally, model checking—is the process of compar-ing observed data to replications under the model if itwere true. Statistically significance in a hypothesis testcorresponds to some aspect of the data which wouldbe unexpected under the model. For Bayesians, as forother statistical researchers, both these steps of infer-ences are important: we want to make use of the mathe-matics of probability to make conditionally valid state-ments about unobserved quantities, and we also wantto make use of this same probability theory to revealareas in which our models do not fit the data.SAMPLINGKass discusses the role of sampling as a modelfor understanding statistical inference. But sampling ismore than a metaphor; it is crucial in many aspects ofstatistics. This is evident in analysis of public opinionand health, where analyses rely on random-sample na-tional surveys, and in environmental statistics, wherecontinuous physical variables are studied using space-time samples. But even in areas where sampling is lessapparent, it can be important. Consider medical exper-iments, where the object invariably is inference for thegeneral population, not merely for the patients in thestudy. Similarly, the goal of Kass and his colleagues intheir neuroscience research is to learn about general as-pects of human and animal brains, not merely to studythe particular creatures on which they have data. Ul-timately, sample is just another word for subset, andin both Bayesian and classical inference, appropriategeneralization from sample to population depends on amodel for the sampling or selection process. I have noproblem with Kass’ use of sampling as a frameworkfor inference, and I think this will work even better ifhe emphasizes the generalization from real samples toreal populations—not just mathematical constructs—that are central to so much of our applied inferences.SUBJECTIVITY AND BELIEFThe only two statements in Kass’ article that I clearlydisagree with are the following two claims: “the onlysolid foundation for Bayesianism is subjective,” and10DISCUSSION 11“the most fundamental belief of any scientist is that thetheoretical and real worlds are aligned.” I will discussthe two statements in turn.Claims of the subjectivity of Bayesian inferencehave been much debated, and I am under no illusionthat I can resolve them here. But I will repeat my pointmade at the outset of this discussion that Bayesianprobability, like frequentist probability, is except in thesimplest of examples a model-based activity that ismathematically anchored by physical randomization atone end and calibration to a reference set at the other.I will also repeat the familiar, but true, argument1 thatmost of the power of a Bayesian inference typicallycomes from the likelihood, not the prior, and a personwho is really worried about subjective model-buildingmight profitably spend more effort thinking about as-sumptions inherent in additive models, logistic regres-sions, proportional hazards models, and the like. Eventhe Wilcoxon test is based on assumptions! To put it an-other way, I will accept the idea of subjective Bayesian-ism when this same subjectivity is acknowledged forother methods of inference. Until that point, I prefer tospeak not of “subjectivity” but of “assumptions” and“scientific judgment.” I agree with Kass that scientistsand statisticians can and should feel free to make as-sumptions without falling into a “solipsistic quagmire.”Finally, I am surprised to see Kass write that sci-entists believe that the theoretical and real worlds arealigned. It is from acknowledging the discrepancies be-tween these worlds that we can (a) feel free to makeassumptions without being paralyzed by fear of mak-ing mistakes, and (b) feel free to check the fit of ourmodels (those hypothesis tests again! Although I pre-fer graphical model checks, supplanted by p-values asnecessary). All models are false, etc.I assume that Kass is using the word “aligned” in aloose sense, to imply that scientists believe that theirmodels are appropriate to reality even if not fully cor-rect. But I would not even want to go that far. Often1As a friend remarked to me in tenth-grade English class, “I don’tknow why they don’t want us to use clichés. These sayings areclichés because they’re true!”in my own applied work I have used models that haveclear flaws, models that are at best “phenomenologi-cal” in the sense of fitting the data rather than corre-sponding to underlying processes of interest—and of-ten such models do not fit the data so well either.2 Butthese models can still be useful: they are still a part ofstatistics and even a part of science (to the extent thatscience includes data collection and description as wellas deep theories).DIFFERENT SCHOOLS OF STATISTICSLike Kass, I believe that philosophical debates canbe a good thing, if they motivate us to think carefullyabout our unexamined assumptions. Perhaps even theexistence of subfields that rarely communicate witheach other has been a source of progress in allowingdifferent strands of research to be developed in a plural-istic environment, in a way that might not have been soeasily done if statistical communication had been dom-inated by any single intolerant group. Ideas of sam-pling, inference, and model checking are important inmany different statistical traditions and we are luckyto have so many different ideas on which to draw forinspiration in our applied and methodological research.ACKNOWLEDGMENTSThis work was supported in part by Institute of Ed-ucation Sciences, Department of Energy, National Sci-ence Foundation and National Security Agency.2In the annals of hack literature, it is sometimes said that if youaim to write best-selling crap, all you will end up with is crap. Totruly produce best-selling crap, you have to have a conviction, per-haps misplaced, that your writing has integrity. Whether or not thisis a good generalization about writing, I have seen an analogousphenomenon in statistics: If you try to do nothing but model thedata, you can be in for a wild and unpleasant ride: real data alwaysseem to have one more twist beyond our ability to model (von Neu-mann’s elephant’s trunk notwithstanding). But if you model the un-derlying process, sometimes your model can fit surprisingly well aswell as inviting openings for future research progress.

Bayesian Statistical Pragmatism

Abstract

Similar works

Full text

Available Versions

Columbia University Academic Commons

Crossref