Measures of information and surprise, such as the Shannon information value (S value), quantify the signal present in a stream of noisy data. We illustrate the use of such information measures in the context of interpreting P values as compatibility indices. S values help communicate the limited information supplied by conventional statistics and cast a critical light on cutoffs used to judge and construct those statistics. Misinterpretations of statistics may be reduced by interpreting P values and interval estimates using compatibility concepts and S values instead of "significance"and "confidence.

Cole, S.R.

Edwards, J.K.

Greenland, S.

Carolina Digital Repository

Surprise!Stephen R. Cole∗, Jessie K. Edwards, and Sander Greenland∗ Correspondence to Dr. Stephen R. Cole, Department of Epidemiology, Gillings School of Global Public Health,University of North Carolina at Chapel Hill, Campus Box 7435, Chapel Hill, NC 27599-7435 (e-mail: cole@unc.edu).Initially submitted April 3, 2020; accepted for publication July 7, 2020.Measures of information and surprise, such as the Shannon information value (S value), quantify the signalpresent in a stream of noisy data. We illustrate the use of such information measures in the context of interpretingP values as compatibility indices. S values help communicate the limited information supplied by conventionalstatistics and cast a critical light on cutoffs used to judge and construct those statistics. Misinterpretations ofstatistics may be reduced by interpreting P values and interval estimates using compatibility concepts and Svalues instead of “significance” and “confidence.”compatibility; confidence intervals; information; P value; random error; S value; significance tests; statisticalinferenceEditor’s note: The opinions expressed in this article arethose of the authors and do not necessarily reflect the viewsof the American Journal of Epidemiology. A response to thiscommentary appears on page 194.Measures of information and surprise have a long history(see Good (1), chapter 16) but have seen little use outsidethe fields of engineering and mathematical statistics. Suchmeasures of information and surprise attempt to quantify thesignal present in a stream of noisy data. One such measureis the Shannon information, which, when seeing an event ofprobability p, is defined as s = log2(1/p) = −log2(p) and isalso known as the binary surprise index or surprisal (2). It hasbeen argued that this measure could aid interpretation of Pvalues and interval estimates, especially when the latter areviewed as showing compatibility of data with hypotheses,rather than stronger notions of significance or confidence(3–5). Here we briefly illustrate these ideas in the contextof interpreting P values as compatibility indices.THE P VALUE AS A COMPATIBILITY INDEXA P value represents the chance of observing a datasummary (test statistic) as extreme as or more extremethan what was seen, under a test hypothesis and auxiliary(background) assumptions. Perhaps the most common aux-iliary assumptions are that the observed data are randomlysampled or treatment is randomly assigned within observedcovariate levels, and that measurement error is negligible(6); regression models add further assumptions. Typicallythe test hypothesis is that a parameter is 0 (2-sided null) oris no greater than 0 (1-sided null), but other values can andshould be tested besides 0 (3–5, 7; also see Rothman et al.(8), chapter 10).A P value is valid if it would have a uniform distributionwhen sampling data under the tested hypothesis given theauxiliary assumptions used to compute it. Such a P value canbe interpreted as giving the percentile at which the observeddata fell in this distribution. The P value can thus be taken asan index of compatibility between the data and the parametervalues specified by the tested hypothesis given the auxiliaryassumptions, ranging from p = 0 (data flatly contradict thehypothesis) to p = 1 (data are exactly as expected under thehypothesis) (3–5). A valid 95% confidence interval can beconstructed as the set of all parameter values with p > 0.05(see Rothman et al. (8), chapter 10). Therefore, the values ofa 95% confidence interval have a compatibility index of 0.05and above, and they comprise a 5%-or-more compatibilityinterval (3–5), which can also, like the confidence interval,be abbreviated using “CI.” (Some authors define a P value asa random variable P that is uniform under the test hypothesisand auxiliary assumptions, with p being the value of P in theobserved data).Consider a recent randomized trial of lopinavir and riton-avir, versus standard care, in the treatment of severe corona-virus disease 2019 (9) which reported 19 and 25 deathsamong 99 and 100 patients, respectively. The authors statedthat “no benefit was observed with lopinavir-ritonavir treat-ment beyond standard care” (9, p. 1787), despite observinga 28-day mortality risk difference of −5.8% (i.e., 19.2%− 25.0%), with a 95% compatibility (“confidence”) inter-val ranging from −17.3% to 5.7%. This interval includesrisk differences ranging from −17.3%, which representsa tremendous mortality benefit, to 5.7%, which representsa nontrivial increase in mortality. The statistics leave thehypothesis of no benefit (i.e., a causal risk difference ≥ 0) asreasonably compatible with the data, with 1-sided p = 0.16(from a z score of 0.9780 (−0.0581/0.0587), where 0.0587(0.0587 = [0.057 − (−0.173)]/3.92) is an approximate stan-dard error). But benefits up to a risk difference of −11.6%are even more compatible with the data, in that they haveeven higher P values than does no benefit.THE S VALUEThe S value provides a reinterpretation of the P valueusing a familiar mechanical framework for calibratingintuitions, one that is simpler and less abstract than effectestimation from statistical models. Envision a coin-tossingsetup that we want to check for bias toward heads (as wemight be advised to do if we were going to wager on tailsfrom this setup). We check by tossing the coin s times. Ifwe observe heads on every toss, the exact P value for thehypothesis of no bias toward heads is 0.5s, a special case ofthe fact that, for m heads in n tosses, the exact P value for the1-sided hypothesis that “the probability of heads is nogreater than μ” isn∑k=m( nk)μk(1 − μ)(n−k). The Shannonmeasure of the information against this hypothesis is thenthe binary surprisal −log2(0.5s) = s, the number of headsin a row observed. Because s is computed using base-2 logs,its units are said to be bits (binary digits) of information (2,p. 32); other base units are possible (3).A key benefit of the S value is that it provides a simplecoin-tossing framework for interpretation of P values andconfidence intervals. Returning to the coronavirus example,the 1-sided P value of 0.16 for the no-benefit hypothesisyields an S value of 2.6 (−log2(0.16) = 2.6). To place thisresult into our coin-tossing framework, a result of all headsin 3 fair tosses has a 0.125 (1 in 8) chance of occurringand thus does not seem terribly surprising (albeit it is moresurprising than 2 heads in a row, where p = 0.25, andless surprising than 4 heads in a row, where p = 0.0625).Therefore we say that, if there is no treatment benefit, theobserved p = 0.16 is less surprising than seeing 3 heads ina row in 3 fair tosses (because 2.6 < 3).The P value for a benefit of 11.6% is equal to theP value for the no-benefit hypothesis, meaning that thedata are equally compatible with (and would be equallysurprising under) both hypotheses. These data would beeven less surprising under risk differences between 0 and11.6%. Viewing the compatibility interval of –17.3% toFigure 1. The S value as a function of the P value.5.7%, the data supply at most 4.3 bits of information(−log2(0.05) = 4.3) against treatment effects ranging froma 17.3% reduction to a 5.7% increase in mortality, and allrisk differences in this interval make the data about the sameas or less surprising than seeing 4 heads (−log2(0.05) ≈ 4)in 4 fair tosses.Now consider P values of 0.10, 0.05, 0.01, and 0.005.The corresponding S values are 3.3, 4.3, 6.6, and 7.6, sowith rough rounding, these P values should seem about assurprising as seeing 3, 4, 7, or 8 heads in a row from faircoin-tossing. Figure 1 provides the mapping from P valuesto S values. One may feel that p > 0.05 is unsurprising ifthe test hypothesis is correct, given s < 4.3. That judgmentis fine; nonetheless, effect sizes with higher P values thanthe test hypothesis exhibit more compatibility with the dataand have less information against them than does the testhypothesis. Thus, p > 0.05 is not a sufficient basis forclaiming or acting as if the results support the test hypothesisor do not support alternatives, since such dichotomizationsmask important distinctions.The S value is based on the same assumptions as thoseused to compute its source P value, and thus introduces nonew technical or validity issues. While the computationsare objectively determined by data and assumptions, theirinterpretations are subject to the limitations of human cog-nition. One should expect an event with chance 1 in 10 tohappen in one-tenth of our observations, on average. If onehypothesizes that the event is as likely as not (i.e., chance 1in 2), then one ought to feel no surprise if one sees 1 eventin 2 tries (2-sided p = 1, s = 0). The extent of our surpriseought to grow, as does the S value, as the data diverge fromthe hypothesis. Specifically, the S value grows by a unit forevery halving of the P value. Being a continuum, there isno particular S value cutpoint above which one ought to be“surprised.” Use of the P value or S value as a continuum isnot as arbitrary as making a dichotomous comparison, sayP < 0.05. A key point here is that the S value maps directlyonto a standard game of coin-tossing, providing the highlyheterogeneous set of human observers with an easily taughtreference system, to help gauge the information content ofstudies.In conclusion, we advise that misinterpretations whichremain standard in the medical literature can be reducedby reinterpreting P values and confidence intervals as indi-cators of compatibility with data (rather than as indicatingsignificance, confidence, or support). In the above example(9), the authors used confidence intervals as significancetests, concluding that “no benefit was observed” becausethe 95% confidence interval contained the null value (equiv-alent to a null p > 0.05). But interpreting the P valuesand confidence intervals as compatibility values and inter-vals instead of significance tests shows that the results are1) most compatible with a modest benefit and 2) impreciseand therefore highly compatible with a wide range of effects.We thus conclude that compatibility interpretations and Svalues can help communicate the limited information sup-plied by conventional statistics and can cast a critical lighton the cutoffs used to judge and construct those statistics.ACKNOWLEDGMENTSAuthor affiliations: Department of Epidemiology,Gillings School of Global Public Health, University ofNorth Carolina at Chapel Hill, Chapel Hill, North Carolina(Stephen R. Cole, Jessie K. Edwards); Department ofEpidemiology, Fielding School of Public Health, Universityof California, Los Angeles, Los Angeles, California(Sander Greenland); and Department of Statistics, Collegeof Physical Sciences, University of California, LosAngeles, Los Angeles, California (Sander Greenland).This work was supported in part by National Institutes ofHealth grants K01 AI125087 and R01 AI157758.Conflict of interest: none declared.REFERENCES1. Good IJ. Good Thinking. Minneapolis, MN: University ofMinneapolis Press; 1983.2. Shannon CE. A mathematical theory of communication. BellSyst Tech J. 1948;27:379–424, 623–656.3. Greenland S. Valid P-values behave exactly as they should:some misleading criticisms of P-values and their resolutionwith S-values. Am Stat. 2019;73(suppl 1):106–114.4. Amrhein V, Trafimow D, Greenland S. Inferential statistics asdescriptive statistics: there is no replication crisis if we don’texpect replication. Am Stat. 2019;73(suppl 1):262–270.5. Rafi Z, Greenland S. Semantic and cognitive tools to aidstatistical science: replace confidence and significance bycompatibility and surprise. BMC Med Res Methodol. 2020;20:Article 244.6. Greenland S. Randomization, statistics, and causal inference.Epidemiology. 1990;1(6):421–429.7. Poole C. Beyond the confidence interval. Am J Public Health.1987;77(2):195–199.8. Rothman KJ, Greenland S, Lash T. Modern Epidemiology.3rd ed. New York, NY: Lippincott-Raven; 2008.9. Cao B, Wang Y, Wen D, et al. A trial of lopinavir-ritonavir inadults hospitalized with severe Covid-19. N Engl J Med. 2020;382(19):1787–1799.

Surprise!

http://cdr.lib.unc.edu/downloads/js956s35n

Surprise!

Abstract

Similar works

Full text

Available Versions

Carolina Digital Repository