Jarman, Alex

Smith, Leonard A.

English

LSE Research Online

Small-number statistics, Common Sense, and Profit:Challenges and Non-challenges for Hurricane ForecastingAlex Jarman and Leonard A. SmithCentre for the Analysis of Time Series, London School of EconomicsEmail: a.s.jarman@lse.ac.ukAbstractWhen making only one forecast per year, or perdecade, it can take some time to establish statisti-cal confidence in the skill of a given forecast scheme.Must a risk tolerant decision maker wait decades un-til skill is “proven” if that decision maker believes thesystem to have value? What of a risk neutral decisionmaker? A methodology is illustrated to demonstratethere are imperfect forecast systems which almost cer-tainly have nontrivial value long before one might es-tablish that their skill was statistically significant.1 Identifying skill with small datasetsNumber of Atlantic Basin hurricanes 1960-20091960 1965 1970 1975 1980 1985 1990 1995 2000 2005 Year051015NumberFigure 1: HURDAT data: Number of Atlantic Basin hurricanes from1960 to 2008 [3])The forecasting of US hurricanes has become a high pro-file endeavour over recent years largely due to its potentialapplications for the insurance industry, and the search forclimate change signals in hurricane activity data. Estab-lishing out-of-sample skill in an annual hurricane forecast-ing system poses a challenge on decadal timescales due tothe slow rate new information is gathered with which toverify forecasts. The range of uncertainty in a sample offorecast model evaluations increases with decreasing timeduration, and thus genuine skill cannot be reliably ascribedto a forecast model which is verified with a small dataset.It has been argued that to robustly assess the predictive skillof a hurricane forecasting system, it would need to sustainan accurate enough performance over at least a period ofseveral decades [4]. Otherwise, there is no way of knowingwhether any skilful predictions made by the model are at-tributable to the quality of the model or to just chance alone.This raises two interesting questions: 1) would it truly takeseveral decades to establish skill in practice? And 2) evenif so, should the lack of established skill deter a decision-maker from using a forecast they believe to be valuable?The second question is investigated in this poster, and weargue that the answer to this question is “no”.2 Does a decision-maker need towait?If demonstrating genuine skill with limited datasets is notpossible, then should a decision-maker wait for proof of skillin a model before using it? Might they be forgoing the op-portunity to benefit from forecast information whilst seekingstatistical reassurance? We examine the cost of waiting. Ifthe decision-maker believes in the skill of their model, theymight rationally choose to begin implementing it and willbegin to receive value before those who choose to delay. Thechance to profit before proving can be conceptualised in thecontext of what is called the “Swindled Statistician Scam”(see Box 1). System CDF contours with phase0 5 10 15 20Phi (years)0123456789101112Hurricane Number (X_j)90th percentile70th percentileMedian30th percentile10th percentileFigure 2: Contours of the cumulative distribution function showinghow the system probability distribution changes as a function of thephase of the 24 year cycle. Note that occurrences of higher hurricanenumbers are more likely in years with phase 5, 6 or 7 than those withphase 17, 18 or 19.3 Profit or Proof: how should adecision-maker decide?Consider a toy hurricane system in which the mean num-ber of storms follows a 24 year cycle, while the number ofstorms in any given year is determined at random. To illus-trate that structural model error does not preclude value toa decision-maker, consider an imperfect model of that sys-tem with the same cycle period, but where the probabilitydistribution function used is incorrect in shape, not merelyin parameter. This model will then be used in games ofHurricane Roulette, where the imperfect (but time depen-dent) model probabilities are used to place bets against oddsset by a house using the correct (but not time dependent)climatological probability distribution. The results can bereported in either bits of information or as an expected an-nual return (see [1]). System, Climatology and Forecast Model PDFs0 1 2 3 4 5 6 7 8 9 10 11 12 13Hurricane Number (x_j)0.00.10.20.30.4Prob(X_j)System (phi=12)ClimatologyForecast ModelFigure 3: This figure shows the probability distributions for the system(black), and an imperfect model (green) for phase year 12 of the 24year cycle. The climatological distribution (computed over all valuesof phase) is also shown in blue. Note that one need not know thetrue system PDF in order to outperform climatology. In this case theimperfect model PDF is expected to have a better ignorance score thanclimatology; 0.066 bits on average (corresponding to a 1.047% per yearinterest rate).Box 1. The Swindled StatisticianScam: A wily underwriter approaches a non-Floridian statistician with a business deal: thestatistician will produce a probability forecastof the number of destructive events in the com-ing year, the underwriter will use her marketcontacts to bet on the forecast. As soon as thestatistician can prove the forecast really doeshave skill, the underwriter will pay royalties.Will this leave the statistician swindled out of asmall fortune?The system is defined as a Poisson process, X ∼Pois(λ(t)), where X is the number of hurricanes in a givenyear and has a sinusoidal time-dependent mean determinedby the equation λ(t) = 2.5sin(2pit/T +φ)+5.0. The prob-ability distribution of the system over each of the 24 phasesis illustrated in Figs 2 and 3 which describe the cumulativedistribution function for all phases, and the probability den-sity function corresponding to phase year 12 respectively.The forecast model is defined by a squared Gaussian dis-tribution i.e. if V ∼ N (µ, σ2) then the random variableY = bV 2 + 0.5c represents the distribution of annual fore-cast hurricane numbers. In addition, the model parameterσ has been fitted to each phase of the 24-year system cycleby minimising the expected ignorance of the forecast.Likelihood of Profit with Lead Time0 5 10 15 20 25 30Time-50050100150200Wealth90th percentile70th percentileMedian30th percentile10th percentileFigure 4: Frequency distribution of wealth of 2048 decision-makerswho bet using the imperfect hurricane forecast model, as a function ofthe duration of time over which they bet. The dotted line shows theirinitial wealth (set equal to one unit), so the fraction of those above thisline reflects the likelihood of being in profit by using the model. Notethat the 10th percentile (orange) crosses and stays clearly in profit afterabout year 15: this is long before one could establish that the modelhad statistically significant skill.Likelihood of Profit with Lead Time0 20 40 60 80 100 120Time10-5100105101010151020Wealth99th percentile90th percentile70th percentileMedian30th percentile10th percentile1st percentileFigure 5: Frequency distribution of wealth as in Fig. 4, but shownover a longer time period and on logarithmic scales. Note that on thetime scales on which one could establish statistically significant skill bytime series analysis given a model with a period of 24 years, most of thedecision-makers have made a nontrivial profit.Hurricane Roulette proceeds as follows: at the start ofeach annual hurricane season a punter, or in this case thedecision-maker, is offered odds defined by the climatologyPDF (equally-weighted sum of the 24 system phase PDFs).She then places her bet by distributing all of her currentwealth (based on the Kelly betting strategy [2]) according tothe forecast probabilities assigned to each possible hurricanenumber outcome (Xj). The actual outcome determines thepay-off on each annual bet.The results of a sample of 2048 realisations (or worlds)of this game of Hurricane Roulette demonstrate that thedecision-maker would be very likely to have made a non-trivial profit even before two system cycles have completed- which is much shorter than the time it would take to reacha sufficient level of statistical significance (NB: the phase,φ, is selected at random for each realisation to avoid bias).This is evident in Fig 4 which shows the frequency distri-bution of wealth of 2048 betting decision-makers.If the game of Hurricane Roulette is extended out to 128years, the likelihood of profit significantly increases and wefind that over 99% of decision-makers will have made a non-trivial profit before 50 years have passed. This is shown inFigs 5 and 6. Even though this is an idealised situation, itdoes demonstrate why a decision-maker might consider andpotentially benefit from the use of a forecasting system theybelieve to be skilful. The results above illustrate a case inwhich the fleeced statistician would be kicking himself for along time before the first royalty payment came in.Percentage of decision-makers who are likely to profit0 20 40 60 80 100 120Time5060708090100Percentage of forecastersFigure 6: Percentage of decision-makers expected to make a profit withtime when betting against climatology using the imperfect model in agame of hurricane roulette; the sample size is 2048 worlds.Box 2. Discussion Points• Establishing Forecast Skill on Systems withlong time scales poses a challenge• A decision-maker may accept risk in order togain profit, rather than first wait to establishstatistical confidence in the forecast. In suchcases, any difficulty in establishing forecastskill is a non-challenge.•Ways of benefiting from an imperfect modelcan be demonstrated through the use ofgames like Hurricane RouletteReferences[1] Renate Hagedorn and Leonard A. Smith. Communicating the value of probabilistic forecastswith weather roulette. Meteorological Applications, 16(2):143–155, October 2009.[2] J. L. Kelly. A new interpretation of information rate. Bell Systems Tech., 35:917–926, 1956.[3] National Oceanic and Atmospheric Administration. Atlantic hurricane database re-analysisproject. source: http://www.aoml.noaa.gov/hrd/datasub/re\verbanal.html.[4] R. A. Pielke. United states hurricane landfalls and damages: Can one- to five-year predictionsbeat climatology? Environmental Hazards, 8:187–200(14), September 2009.Acknowledgements: The support of EQUIP, of the Economic and SocialResearch Council (ESRC), and of Munich Re is gratefully acknowledged.

Small-number statistics, common sense, and profit: challenges and non-challenges for hurricane forecasting

Abstract

Similar works

Full text

Available Versions

LSE Research Online