Data-driven risk analysis involves the inference of probability distributions
from measured or simulated data. In the case of a highly reliable system, such
as the electricity grid, the amount of relevant data is often exceedingly
limited, but the impact of estimation errors may be very large. This paper
presents a robust nonparametric Bayesian method to infer possible underlying
distributions. The method obtains rigorous error bounds even for small samples
taken from ill-behaved distributions. The approach taken has a natural
interpretation in terms of the intervals between ordered observations, where
allocation of probability mass across intervals is well-specified, but the
location of that mass within each interval is unconstrained. This formulation
gives rise to a straightforward computational resampling method: Bayesian
Interval Sampling. In a comparison with common alternative approaches, it is
shown to satisfy strict error bounds even for ill-behaved distributions.Comment: 13 pages, 3 figures; supplementary information provided. A revised
  version of this manuscript has been accepted for publication in Philosophical
  Transactions of the Royal Society A: Mathematical, Physical and Engineering
  Science

Strbac, Goran

Tindemans, Simon H.

English

arXiv

Data-driven risk analysis involves the inference of probability distributions from measured or simulated data. In the case of a highly reliable system, such as the electricity grid, the amount of relevant data is often exceedingly limited, but the impact of estimation errors may be very large. This paper presents a robust non-parametric Bayesian method to infer possible underlying distributions. The method obtains rigorous error bounds even for small samples taken from ill-behaved distributions. The approach taken has a natural interpretation in terms of the intervals between ordered observations, where allocation of probability mass across intervals is well specified, but the location of that mass within each interval is unconstrained. This formulation gives rise to a straightforward computational resampling method: Bayesian interval sampling. In a comparison with common alternative approaches, it is shown to satisfy strict error bounds even for ill-behaved distributions.
          This article is part of the themed issue ‘Energy management: flexibility, risk and optimization’.</jats:p

Simon H. Tindemans

Goran Strbac

Crossref

Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences

Robust estimation of risks from small samples

Data-driven risk analysis involves the inference of probability distributions from measured or simulated data. In the case of a highly reliable system, such as the electricity grid, the amount of relevant data is often exceedingly limited, but the impact of estimation errors may be very large. This paper presents a robust non-parametric Bayesian method to infer possible underlying distributions. The method obtains rigorous error bounds even for small samples taken from ill-behaved distributions. The approach taken has a natural interpretation in terms of the intervals between ordered observations, where allocation of probability mass across intervals is well specified, but the location of that mass within each interval is unconstrained. This formulation gives rise to a straightforward computational resampling method: Bayesian interval sampling. In a comparison with common alternative approaches, it is shown to satisfy strict error bounds even for ill-behaved distributions

Tindemans, S

Strbac, G

Spiral - Imperial College Digital Repository

Supplementary information‘Robust estimation of risks from small samples’Simon H. TindemansGoran StrbacImperial College Londons.tindemans@imperial.ac.ukContentsS1 Bayesian interval sampling with multiplicity 1S2 Further examples 2S3 Connections to other methods 3S3.1 Conditional distribution for x . . . . . . . . . . . . . . . . . . . . 3S3.2 Purely probabilistic approach . . . . . . . . . . . . . . . . . . . . 4S3.3 Stochastic processes and NPI . . . . . . . . . . . . . . . . . . . . 5S4 Posterior Dirichlet processes 6S1 Bayesian interval sampling with multiplicityThe Bayesian Interval Sampling algorithm requires the generation of nresample×(n+1) random weights from the Dirichlet distribution. When the sample size nis large the resampling procedure may require a significant amount of CPU time.This is true in particular for rare event simulations in which a large majorityof observations does not contribute to the observable (e.g. the cost of systemmalfunctions in a very reliable system, which is 0 for the overwhelming majorityof cases). For such cases the required sample size n can be in the millions.However, the inference process can be made considerable more efficient if thedata set contains duplicate observations. Let us consider the extended set oforder statistics {x(0):(n+1)} with x(0) = xL and x(n+1) = xR, and assume thatx(k) to x(k+m) are m+ 1 identical observations. We define the reduced data set{z0:n−m+2} = {x(0):(k), x(k+m):(n+1)} (1)1in which the duplicate observation occurs only twice. We may then use theproperty (assuming i < j without loss of generality){P1:i−1,Pi + Pj , Pi+1:j−1, Pj+1:k} ∼ Dir[α1:i−1,αi + αj , αi+1:j−1, αj+1:k](2)to merge overlapping point intervals inFn =[n+1∑i=1WiHx(i) ,n+1∑i=1WiHx(i−1)](3)with {W1:n+1} ∼ Dir [1, . . . , 1]. This results inFn =[n−m+2∑i=1W̃iHzi(x),n−m+2∑i=1W̃iHzi−1(x)](4)with the adjusted weight vector{W̃1:n−m+2} ∼ Dir [1, . . . , 1,m, 1, . . . , 1] , (5)where a weightm has been assigned to the k+1 position. With this modification,only (n−m + 2) random samples are required to sample a random realisationof Fn. When n−m n this results in a correspondingly large speedup.S2 Further examplesThis section presents the computation of c-credible intervals for two additionalquantities of interest, both applied to data from the log-normal distributionwith log-mean 0 and log-standard deviation 1.We first consider the truncated mean, which discards one or both extremesof the inferred distribution. Figure S1 shows the probability box for a truncatedmean µ99%, discarding the largest 1% of sampled distribution for each resam-pling step. This can be considered the counterpart to the conditional valueat risk (CVaR99%), which reports the mean of the remaining tail contribution:µ = 0.99µ99% + 0.01CVaR99%. Because the sample size n = 15 is insufficientto meaningfully compute a 99% quantile, we used 1000 data samples from the(0,1)-log-normal distribution instead. Note that the resulting upper and lowerdistributions are very similar, implying that the resulting uncertainty is pre-dominantly probabilistic in nature. Estimates of the truncated mean are hardlyaffected by unknown features of the distribution.Finally, we estimate the location for the 99% quantile q99% (also known asthe 0.99 value-at-risk), which is also the cutoff point for the truncated meanµ99%. The resulting probability box and interval are shown in Fig. S2. TableS1 summarises all results, including those from Figure 3 in the main text.21.2 1.4 1.6 1.8 2.0x0.20.40.60.81.099% truncated mean99% credible intervalFigure S1: Probability box for the truncated mean µ99%, considering only the99% probability mass associated with the smallest values. Based on 1000 sam-ples from a (0,1)-log-normal distribution and nresample = 10, 000. The black dotdenotes the true value of the median of the originating distribution (≈ 1.51).5 10 15 20x0.20.40.60.81.099% VaR99% credible intervalFigure S2: Probability box for the 99% quantile q99%. Based on 1000 samplesfrom a (0,1)-log-normal distribution and nresample = 10, 000. The black dotdenotes the true value of the median of the originating distribution (≈ 10.2).Table S1: Interval estimatesNsample parameter Nresample credibility result15 median 1000 90 % [0.34, 3.60]15 mean 1000 90 % [1.21,∞)1000 99% quantile 10,000 99 % [7.76, 14.2]1000 99% truncated mean 10,000 99 % [1.38, 1.70]3S3 Connections to other methodsIn the following we discuss various limiting cases of the robust posterior priorFn described in the paper, and we make connections to other inference methodsdescribed in the literature.S3.1 Conditional distribution for xIt is illustrative to consider the restriction of the random imprecise distributionFn to a given position x, assuming x /∈ {x1:n}. Repeated application of (2) to(3) and {W1:n+1} ∼ Dir [1, . . . , 1] results in the random probability intervalFn(x) =[W−x , 1−W+x], (6a)with{W−x ,W 1x ,W+x } ∼ Dir[n−(x), 1, n+(x)]. (6b)Here n−(x) is the number of observations smaller than x and n+(x) is thenumber of observations larger than x.For a Dirichlet distribution Dir[α1:n], the probability distribution of a com-pound event is the beta distribution P (A) ∼ β[∑i∈A αi,∑j /∈A αj ]. Therefore,the lower and upper bounds can be described as:Fn(x) ∼ β[n−(x), n+(x) + 1], (7)Fn(x) ∼ β[n−(x) + 1, n+(x)]. (8)These random variables are inferred lower and upper bound distributions ex-pressing our state of knowledge regarding the value F ∗X(x) = P (X ≤ x). Theresults are consistent with Bayesian estimators from Bernoulli process data,using the extreme priors limε→0 β[ε, 1 − ε] (F0(x) ≈ 0) and limε→0 β[1 − ε, ε](F0(x) ≈ 1), respectively. In the language of Bayesian sensitivity analysis thesepriors may be considered bounds for the family of priors Px = {β[p, 1− p] : p ∈[0, 1]}.S3.2 Purely probabilistic approachWe now consider the restriction of Fn to a purely probabilistic description ofuncertainty, starting from the definitionFn = Un+1 ◦ F n , with (9)F n =1n+ 1F 0 +nn+ 1F̂n, (10)F 0 = [HxR , HxL ]. (11)A possible uninformative choice is to replace the vacuous prior F 0 by the averageof its upper and lower distributions:F 0 →12HxL +12HxR , (12)4which results inḞn =Un+1 ◦( 12HxL +12HxR +∑ni=1Hxin+ 1). (13)The random variable Ḟn(x) at x ∈ I;x 6= xi isḞn(x) ∼ β[n−(x) +12, n+(x) +12]. (14)As in the previous case, we can interpret this result as the Bayesian estima-tor for the probability F ∗X(x) = P (X ≤ x) given n = n+ + n− independentmeasurements. This time, the prior is β( 12 ,12 ), an uninformative prior which isboth the Jeffreys prior and reference prior for this problem [1]. This correspon-dence retrospectively justifies the choice (13) as a minimally informative purelyprobabilistic contraction of Fn.S3.3 Stochastic processes and NPIBesides the intuitive interpretation as a distribution of distributions, the Dirich-let process may also be interpreted as a stochastic process in which each gen-erated sample modifies the probability distribution for subsequent samples ina rich-get-richer scheme [2]. The observations in the resulting infinite sequenceare dependent but exchangeable. The distribution and process viewpoints arelinked by De Finetti’s theorem, which states that distributions of the obser-vations in (infinite) sequences generated in this way are distributed accordingto a measure representing a latent variable. In this case, that measure is thedistribution-of-distributions representation of the Dirichlet process [3].The robust posterior distribution Fn has the property that, on average, itassigns a probability mass 1/(n + 1) to the interval between measurements.Similarly, Hill [4] has proposed the ‘An assumption’ for a stepwise inferenceprocess, stating that “conditional upon the observations X1, . . . , Xn, the nextobservation Xn+1 is equally likely to fall in any of the open intervals betweensuccessive order statistics of the given sample.” The An assumption is thenused to generate bounds on successive samples, each time adding the (possible)generated samples to the observations to be used in An+1.In a 1993 paper [5], Hill has made use of the concept of adherent massto assign the two halves of the probability associated with each interval to itslower and upper boundaries. In combination with limits at −∞ and +∞ it wasshown that the resulting process is equivalent to the Dirichlet process definedin Eq. (13). The purely probabilistic projection Ḟn of Fn is therefore the DeFinetti measure of an An process.Hill’s An assumption forms the basis of the Nonparametric Predictive Infer-ence (NPI) method (see e.g. Coolen [6]). In this approach, successive predictiveobservations are generated using minimally informative interval probabilities(probability boxes) constrained by An. The first inference step of the NPI pro-5cess is given by the imprecise predictive distributionFNPIn+1 =[n+1∑i=1Hx(i)n+ 1,n∑i=0Hx(i)n+ 1], (15)where we make use of the extended set of order statistics {x(0):(n+1)}. Thisimprecise distribution is equal to the imprecise input distribution F N (10). Thisis consistent, because our best (non-random) estimate for the (n+ 1)-predictivedistribution is the expected probability box E[Fn] = F n .Furthermore, we can invoke Hill’s adherent mass argument [5] to analyse theNPI process in the case of a monotonic population parameter q. In this case, weassign the adherent mass consistently to either the lower or upper bounds of eachinterval to estimate the lower and upper bounds of q, respectively. Analogousto Hill’s result for Ḟn we findQminNPI =q[CDP[∑ni=1Hxi +HxLn+ 1]], (16)QmaxNPI =q[CDP[∑ni=1Hxi +HxRn+ 1]]. (17)This result is identical to the result [Qmin, Qmax] =[q[Fn], q[Fn]from the maintext. In other words, NPI for infinite random sequences produces bounds thatare identical to those derived using the robust posterior distribution. Thereforewe postulate that the robust posterior distribution may be considered the DeFinetti measure of the NPI process.Even though the results of the NPI process converge to those obtained usingour method, they are conceptually quite different. NPI has a process-based def-inition that generates a series of predictive samples. The first sample is alwaysdrawn from F n = E[Fn], and subsequent samples are generated from subsequentsample-dependent distributions. An infinite sequence of samples obtained in thisway is equivalent to computing a single realisation of Fn. The direct formula-tion developed in this paper results in a significant computational advantage,especially for resampling approaches such as Bayesian Interval Sampling, whichrequire many realisations of Fn.S4 Posterior Dirichlet processesIn this section we demonstrate that F(s)n =[F (s)n ,F(s)n]is a posterior dis-tribution corresponding corresponding to the vacuous prior distribution F 0 =[HxR , HxL ] with concentration parameter s and observations x1:n. This is donein two steps. First, a partial ordering is introduced on the space of cumulativeDirichlet processes (CDPs). Second, the degenerate CDPs are expressed as lim-its of non-degenerate CDPs. This enables us to provide a limiting definition ofthe posterior CDPs.6Definition. We define a partial ordering on CDPs F and G as follows:F  G : F(x) equals or dominates G(x),∀x ∈ R, (18)where dominance is defined as first order stochastic dominance [7]. In otherwords, F  G if for all x each quantile of F(x) equals or exceeds the corre-sponding quantile of G(x).Lemma S4.1. Let F and G be CDPs with concentration parameter s:F (s) ∼ CDP[sF ], (19)G(s) ∼ CDP[sG], (20)where F and G cumulative probability distributions. Then the following impli-cation holds(F (x) ≥ G(x),∀x ∈ R)⇒ F (s)  G(s). (21)Proof. It follows from the properties of the Dirichlet process that the randomvariable F(x) is distributed according to the beta distributionF (s)(x) ∼ β[sF (x), s(1− F (x))]. (22)Therefore,F (x) > G(x)⇒ F (s)(x) dominates G(s)(x). (23)This holds for all x, thus proving the statement (21).The implication of this lemma is that for a given value of the concentrationparameter s the cumulative Dirichlet process preserves the partial ordering ofthe shape parameter (the ‘input’ distribution) in a probabilistic sense.Definition. We define the family of ε-contaminated priors that are furtherparametrised by the concentration parameter s and a distribution function Fon I.F (s,ε)0 [F ] ∼CDP[s ((1− ε)HxR + εF )] (24a)F (s,ε)0 [F ] ∼CDP[s ((1− ε)HxL + εF )] (24b)When the implied support of F covers that of F ∗X (the target distribution),these Dirichlet processes have the desirable property that they almost surelyhave a nonzero probability mass associated with the local neighbourhood ofthe observed values {x1:n} and. Specifically, this is the case for the choicesF = uniform(I) and F = F ∗X . The ε-contaminated priors also have the limitslimε↓0F (s,ε)0 [F ] = F(s)0 ∼ CDP[sHxR ] (25a)limε↓0F (s,ε)0 [F ] = F(s)0 ∼ CDP[sHxL ]. (25b)7Property (21) proves that these limits exist for the Cumulative Dirichlet Pro-cess, by virtue of their existence in the regular distribution space. The p-box[F (s)0 ,F(s)0 ] can therefore be considered the envelope of the set of proper priorson I with concentration parameter s.For F (s,ε)0 [F ] and F(s,ε)0 [F ] with ε > 0 the Bayesian posterior distributionsconditioned on the observations {x1:n} is given byF (s,ε)n [F ] ∼ CDP[s ((1− ε)HxR + εF ) +N∑i=1Hxi], (26a)F (s,ε)n [F ] ∼ CDP[s ((1− ε)HxL + εF ) +N∑i=1Hxi]. (26b)Analogous to Eqs. (25) we can now define the posterior envelope distributionsasF (s)n = limε↓0F (s,ε)n [F ] = Un+s ◦(sn+ sHxR +nn+ sF̂n)(27a)F (s)n = limε↓0F (s,ε)n [F ] = Un+s ◦(sn+ sHxL +nn+ sF̂n). (27b)We conclude that F(s)n =[F (s)n ,F(s)n]is indeed a tight envelope of the properprior CDPs with concentration parameter s bounded by the vacuous prior F 0 =[HxR , HxL ].References[1] Bernardo JM. Reference Analysis. In: Handbook of Statistics, Volume 25:Bayesian Thinking Modeling and Computation. vol. 25 of Handbook ofStatistics. Elsevier; 2005.[2] Ferguson TS. A Bayesian Analysis of Some Nonparametric Problems. TheAnnals of Statistics. 1973;1(2):209–230.[3] Lijoi A, Prünster I. Models beyond the Dirichlet process. In: BayesianNonparametrics. Cambridge; 2010. p. 80–135.[4] Hill BM. Posterior Distribution of Percentiles: Bayes’ Theorem for Sam-pling from a Population. Journal of the American Statistical Association.1968;63(322):677–691.[5] Hill BM. Parametric Models for An: Splitting Processes and Mixtures.Journal of the Royal Statistical Society Series B ( Methodological ).1993;55(2):423–433.8[6] Coolen FPA. On Nonparametric Predictive Inference and ObjectiveBayesianism. Journal of Logic, Language and Information. 2006;15(1-2):21–47.[7] Hadar J, Russell WR. Rules for Ordering Uncertain Prospects. The Amer-ican Economic Review. 1969;59(1):25–34.9

Robust estimation of risks from small samples

Abstract

Similar works

Full text

Available Versions

Crossref

Spiral - Imperial College Digital Repository