A method for constructing consonant predictive beliefs for multivariate datasets is presented. We make use of recent results in scenario theory to construct a family of enclosing sets that are associated with a predictive lower probability of new data falling in each given set. We show that the sequence of lower bounds indexed by enclosing set yields a consonant belief function. The presented method does not rely on the construction of a likelihood function, therefore possibility distributions can be obtained without the need for normalization. We present a practical example in two dimensions for the sake of visualization, to demonstrate the practical procedure of obtaining the sequence of nested sets

De Angelis, Marco

Ferson, Scott

Gray, Ander

Rocchetta, Roberto

DeAngelis, Marco

University of Liverpool Repository

Constructing Consonant Predictive Beliefs from Data with Scenario Theory

de Angelis, Marco

University of Strathclyde Institutional Repository

Proceedings of Machine Learning Research 147:357–360, 2021 ISIPTA 2021Constructing Consonant Predictive Beliefs from Data with Scenario TheoryMarco DeAngelis MARCO.DE-ANGELIS@LIVERPOOL.AC.UKInstitute for Risk and Uncertainty, University of Liverpool, United KingdomRoberto Rocchetta R.ROCCHETTA@TUE.NLDepartment of Mathematics and Computer Science, Eindhoven, The NetherlandsAnder Gray ANDER.GRAY@LIVERPOOL.AC.UKScott Ferson SCOTT.FERSON@LIVERPOOL.AC.UKInstitute for Risk and Uncertainty, University of Liverpool, United KingdomAbstractA method for constructing consonant predictive beliefsfor multivariate datasets is presented. We make use ofrecent results in scenario theory to construct a familyof enclosing sets that are associated with a predictivelower probability of new data falling in each given set.We show that the sequence of lower bounds indexed byenclosing set yields a consonant belief function. Thepresented method does not rely on the construction of alikelihood function, therefore possibility distributionscan be obtained without the need for normalization.We present a practical example in two dimensions forthe sake of visualization, to demonstrate the practicalprocedure of obtaining the sequence of nested sets.Keywords: Predictive beliefs; Consonant random sets;Generalization error; Imprecise probability; Evidencetheory.1. IntroductionA very common problem in engineering science is thatof computing with multivariate sample sets making theleast amount of assumptions. Multivariate sample sets aredifficult to deal with because even when it is possible tolearn a joint probability distribution, it is often very chal-lenging to characterize the uncertainty associated with anyparticular choice. We propose a method to characterize theuncertainty associated with constructing a joint distributiondirectly from the data using scenario theory. We show thatthe mechanism that constructs such joint distribution isdesigned to yield coherent lower predictive probabilitiesnicely fitting within the imprecise probability paradigm.We present a way to construct consonant random sets–or possibility measures–that is rooted in scenario theory[5]. We resort to the generalization property of scenariotheory whereby given a set of independent observations,the bounds on the predictive probability of a future obser-vation can be computed. Recent works including [8] haveelaborated on the complexity and the feasibility of suchcomputations in highly multi-dimensional datasets. A re-cent overview on scenario theory can be found here [14].This paper is also inspired by [12], whereby the problem ofinferential uncertainty is cast within the imprecise probabil-ity paradigm, see also [10, 11]. Other papers on this topicare [9, 1, 13, 7]. This work focuses on two main aspects: (i)we propose a method to systematically obtain a sequence ofenclosing sets with assigned probability bounds by meansof scenario theory, (ii) we show that the lower predictiveprobability–also known as predictive belief –associated tothe enclosing sets obtained as such, constitutes a possibilitydistribution, thus is a coherent lower probability. The advan-tage of resorting to scenario optimization to construct thesesets resides in the flexibility of choosing a parametric modelfor the shape of the enclosing sets that are guaranteed to ef-ficiently contain all the data. Moreover this method enablesthe encoding of both aleatory and epistemic dependencewithin the same framework.2. Problem StatementLet X be a random variable on a domain X with unknownprobability distribution PX . We would like to quantify thebeliefs held by an agent about a random future realization ofX from past observations X1, ...,Xn randomly drawn fromthe same distribution and mutually independent (iid). Thisquestion is typically addressed within probability theoryeither using Bayesian inference or with classical statisticalinference, however when the sample size is limited nei-ther approach yields a unique answer, without additionalmodelling assumptions, especially in the multivariate case.With scenario theory we aim to characterize the uncertaintyabout the distribution PX using a consonant predictive be-lief function given some random data (scenarios).2.1. Predictive Beliefs and Belief FunctionsGiven iid random samples X1, ...,Xn, with parent probabil-ity distribution PX , we want to produce a belief function,denoted by BelX := Bel(·;X1, · · · ,Xn), in such a way thatthe inequality BelX ≤ PX , will hold at least in 100(1−β )%of the cases. Which properties should BelX satisfy? Asmore and more data are gathered, it seems reasonable to© 2021 M. DeAngelis, R. Rocchetta, A. Gray & S. Ferson.CONSONANT PREDICTIVE BELIEFS WITH SCENARIO THEORYpostulate that BelX = PX , from a frequentist perspective[7]. A sample set of infinite size should be equivalent toknowledge of the distribution of X , hence the belief func-tion should asymptotically become identical to PX . Forfinite n, it seems natural to impose that BelX will be lesscommitted than PX , which leads to BelX ≤ PX . However,because of limited knowledge, the inequality is often toostringent to hold in all cases. In other terms, assumingthe experiment is repeated indefinitely, we demand BelXto be less committed than PX most of the time, with atleast some prescribed long run frequency 1− β , whereβ ∈ (0,1) is a small number. In summary, the followingtwo requirements are typically imposed to the belief func-tion: (1) ∀A ⊆X ,BelX (A)→ PX (A), as n→ ∞, and (2)Pn(BelX ≤ PX )≥ 1−β . A belief function satisfying thesetwo requirements is called a predictive belief function atconfidence level 1− β . Several methods have been pro-posed to construct these belief functions, like in [7], usingmultinomial confidence intervals, and then further extendedto the continuous case [1]. In this work, we restrict our con-struction to sets that are mutually nested, and resort toscenario theory to compute the belief measure. The conso-nant belief function obtained as such will therefore satisfythe second requirement, but not the first requirement. Let usrecall now that belief functions are coherent lower probabil-ities. Given a power set 2X , the function m : 2X → [0,1]is called a basic belief assignment, if the following twoproperties hold: (i) m( /0) = 0, (ii) ∑A∈2X m(A) = 1. Themass m(A) can be interpreted as the part of the agent’sbelief allocated exactly to the hypothesis that X takes somevalue in A [15]. The subsets A ⊆X such that m(A) > 0are called the focal sets of m. For any focal set A ∈ 2X , itsbelief can be obtained summing up the masses of the focalelements B ∈ 2X contained in it,BelX (A) = ∑B:B⊆Am(B). (1)When the focal sets are nested, BelX is said to be consonant.Consonance has recently been used to build confidence pre-serving structures [3]. Eq. 1 constitutes a lower bound onthe true yet unknown probability PX (A) to the hypothesisthat X takes some value in A. Thus the belief function in Eq.1 falls in a subclass of coherent lower probabilities [2, 6].We propose an algorithm to construct recursively enclosingsets of predictive beliefs, which are also known as necessi-ties, directly from data. The resulting consonant structureprovides a lower bound on the probability of observing newdata in any given enclosing set. Note that by limiting thisapproach to consonant beliefs, the second reequirementcannot be reached for all events, even when n→ ∞. Thisis because the consonant belief structure is effectively apossibility distribution, which always corresponds to a mul-titude of probability distributions, thus never converges toa single probability distribution [16].3. Scenario TheoryConsider a closed bounded set B ⊆ Rm, like one of therectangles in Figure 1 where m = 2. Let us consider avector of parameters z ∈Z ⊆ Rd that uniquely identify B,e.g. its size and location. Given a convex cost function f (z)e.g. the area of the rectangle, a family of convex constraintsz ∈ ZX ⊆ Rd , parametrized by the data X , and a sampleset X1, ...,Xn with d < n, of iid realizations, the followingoptimization programminz∈Zf (z)subject to: z ∈⋂i=1,...,nZXi ,(2)determines the optimal set B, while X1, ...,Xn are calledscenarios, which have to be interpreted as observationsfrom which one wants to make a decision, i.e. select anoptimal value of z. Eq. 2 is a standard convex problemthat can be numerically solved with common software.From this point on, the optimization of Eq.2 will only beused to identify the number of scenarios k that are activeconstraints, i.e. those points that prevent the cost functionfrom further improving.Definition 1 (Enclosing set of degree k) The optimal setB⊆Rm, that strictly contains n−k observations is referredto as an enclosing set of degree k, and it is denoted by Bk,where k is the number of observations strictly not containedin Bk, i.e. either on the borders or outside Bk.The observations lying exactly on the border of Bk arealso known as active scenarios. Note that k is a randomvariable because it is a function of the data, while Bk isa realization of the random set expressed by the belieffunction. Given a parametric set B, e.g. a hyper-box or anellipsoid, the optimization program in Eq.2 is tasked toidentify the number of active scenarios for a given subsetof observations that are to be enclosed. If the optimizationproblem is convex the optimal set is unique and can becomputed exactly.Definition 2 (Lower bound probability of degree k)The precise predictive probability of a given enclosing setof degree k: PX (Bk), has a lower bound pk, with assignedone-sided coverage probability. For a fixed and arbitrarilysmall β ∈ (0,1), and a number k ∈ Z+ of active scenariosthe coverage probability of the given set Bk isPn(pk ≤ PX (Bk))≥ 1−β , (3)Where Pn is the product probability (X1, ...,Xn) due to in-dependence of the samples. The (one-sided) lower boundpk is a random variable of the probability space of the datagenerating mechanism and can be computed using the for-mula and code provided in [4]. A code for the two-sided358CONSONANT PREDICTIVE BELIEFS WITH SCENARIO THEORYcase for both lower and upper bounds is provided in [8].The solution tk to the polynomial equation,βn+1n∑m=k(mk)tm−k−(nk)tn−k = 0, (4)is unique in the interval (0,1). Under the two assumptionsof uniqueness and non-degeneracy, it holds that pk = tk,see Theorem 2 in [4]. To compute tk a simple bisectionalgorithm can be used. More about the two assumptionscan also be found in [4]. The positive integer k can beinterpreted in multiple ways: (i) as a parameter defininghow complex is our decision z; (ii) the minimum numberof scenarios/samples that are necessary to reconstruct theoptimized z; (iii) an over-fitting parameter which constitutesthe key for constructing consonant sets as described in thenext section.4. Method for Constructing ConsonantPredictive Beliefs-15 -10 -5 0 5 10 15 20X1-50050100150200250300X2Figure 1: Consonant box-shaped enclosing sets.The proposed method for constructing sequentially en-closing sets is very straightforward. Let us consider theobservations Xi ∈R2 with i = 1, · · · ,n shown as scatter plotin Figure 1. A simple shape for the enclosing set is forexample a rectangle. The rectangle with the smallest areaenclosing the dataset Xi is uniquely determined by at mostk observations, which are the so called active scenarios.We denote this rectangle by Bk. Note that in the case of arectangle the number of support scenarios is at least two.Also note that the value of k is data dependent (random) andis affected by the functional form of B. Given a fixed and ar-bitrarily small β ∈ (0,1) the predictive belief pk associatedwith Bk can be computed finding the roots of the polyno-mial in Eq.3, which is independent on the shape of the set.the base of the Aztec pyramid shown from above in Figure1 corresponds to the rectangle-shaped set enclosing all theobservations. This set is associated with the largest pre-dictive belief. The algorithm that constructs the sequenceof nested rectangles proceeds by cumulating the numberof active scenarios k until all the observations have beenconverted to support scenarios. In particular we propose touse a simple arithmetic progression with common differ-ence equal to the number of active scenarios at the giveniteration, to encourage symmetry in the resulting sequenceof nested sets. The second iteration determines a rectanglewhose support scenarios are sought in the dataset strippedof the observations that were identified as active scenariosat the previous iteration. This procedure is repeated until nomore observations can be found in a given set, thus guaran-teeing consonance of the constructed rectangles. In Figure2 the two marginal plausibility distributions that result fromthe procedure are shown. Because of Eq.3, any sequenceof active scenarios verifying 0 = k0 < k1 < · · · < kn = n,implies1≥ pk0 ≥ pk1 ≥ ·· · ≥ pkn = 0, (5)X ⊇ Bk0 ⊇ Bk1 ⊇ ·· · ⊇ Bkn = /0. (6)-15 -10 -5 0 5 10 15 20X100.20.40.60.811-BelX-50 0 50 100 150 200 250 300X200.20.40.60.811-BelXFigure 2: Plausibility contours obtained from Eqs.5 and 6,projecting onto the coordinate axes.4.1. The Lower Bound pk Is a Predictive BeliefWe are ready to state the main result of this paper:Theorem 3 The lower bound pk is a sequence of validpredictive beliefs for any k ∈ Z+ such that 0 = k0 < k1 <· · ·< kn = n.Proof Eq.3 guarantees that the lower bound pk is alwaysnon-increasing with k, see also Eq. 5. Two sets obtainedfrom the same scenario optimization program, but withdifferent number of active scenarios are always weaklynested to one another , see Eq. 6. This suffices to provide astrong condition for the sequence of lower bounds pk to bevalid beliefs.359CONSONANT PREDICTIVE BELIEFS WITH SCENARIO THEORYAcknowledgementsThe authors would like to thank Dominik Hose and Alexan-der Wimbush for the fruitful discussions on possibilitydistributions and likelihood functions; and Ryan Martin forthe insightful review of the manuscript. The work was sup-ported by the “ITEA3-2018-17030-DayTiMe” project andby the UKRI Engineering & Physical Sciences ResearchCouncil with grant no. EP/R006768/1.References[1] A. Aregui and T. Denœux. Constructing predictivebelief functions from continuous sample data usingconfidence bands. In 5th International Symposiumon Imprecise Probability: Theories and Applications,2007.[2] T. Augustin, F.P.A. Coolen, G. De Cooman, andM.C.M. Troffaes. Introduction to imprecise prob-abilities. John Wiley & Sons, 2014.[3] M. S. Balch. Mathematical foundations for a theoryof confidence structures. International Journal ofApproximate Reasoning, 53(7):1003–1019, 2012.[4] M. C. Campi and S. Garatti. Wait-and-judge scenariooptimization. Mathematical Programming, 167(1):155–189, Jan 2018. ISSN 1436-4646. doi: 10.1007/s10107-016-1056-9.[5] M. C. Campi, S. Garatti, and F. A. Ramponi. A gen-eral scenario theory for nonconvex optimization anddecision making. IEEE Transactions on AutomaticControl, 63(12):4067–4078, Dec 2018. ISSN 0018-9286. doi: 10.1109/TAC.2018.2808446.[6] T. Denœux. Reasoning with imprecise belief struc-tures. International Journal of Approximate Reason-ing, 20(1):79–111, 1999. ISSN 0888-613X. doi:https://doi.org/10.1016/S0888-613X(00)88944-6.[7] T. Denœux. Constructing belief functions from sam-ple data using multinomial confidence regions. In-ternational Journal of Approximate Reasoning, 42(3):228–252, 2006. ISSN 0888-613X. doi: https://doi.org/10.1016/j.ijar.2006.01.001.[8] S. Garatti and M. C. Campi. Risk and complexity inscenario optimization. Mathematical Programming,pages 1–37, November 2019. ISSN 1436-4646. doi:10.1007/s10107-019-01446-4.[9] D. Hose and M. Hanss. On data-based estimation ofpossibility distributions. Fuzzy Sets and Systems, 399:77–94, 2020. ISSN 0165-0114. doi: https://doi.org/10.1016/j.fss.2020.03.017. Fuzzy Intervals.[10] C. Liu and Ryan M. Inferential models and possibilitymeasures, 2020.[11] R. Martin. An imprecise-probabilistic characteriza-tion of frequentist statistical inference, 2021.[12] R. Martin and C. Liu. Inferential Models: Reason-ing with Uncertainty. Chapman Hall/CRC Mono-graphs on Statistics and Applied Probability. CRCPress, 2015. ISBN 9781439886519.[13] M.H. Masson and T. Denœux. Inferring a possibil-ity distribution from empirical data. Fuzzy sets andsystems, 157(3):319–340, 2006.[14] R. Rocchetta, Q. Gao, and M. Petkovic. Soft-constrained interval predictor models and epistemicreliability intervals: A new tool for uncertainty quan-tification with limited experimental data. Mechan-ical Systems and Signal Processing, 161:107973,2021. ISSN 0888-3270. doi: https://doi.org/10.1016/j.ymssp.2021.107973.[15] G. Shafer. A Mathematical Theory of Evidence. Lim-ited paperback editions. Princeton University Press,1976. ISBN 9780691100425.[16] M. Troffaes and S. Destercke. Probability boxes ontotally preordered spaces for multivariate modelling.International Journal of Approximate Reasoning, 52(6):767–791, 2011. ISSN 0888-613X. doi: https://doi.org/10.1016/j.ijar.2011.02.001.360

Constructing consonant predictive beliefs from data with scenario theory

Proceedings of Machine Learning Research 147:362-365, 2021 ISIPTA 2021Constructing Consonant Predictive Beliefs from Data with Scenario TheoryMarco DeAngelis MARCO.DE-ANGELIS@LIVERPOOL.AC.UKInstitute for Risk and Uncertainty, University of Liverpool, United KingdomRoberto Rocchetta R.ROCCHETTA@TUE.NLDepartment of Mathematics and Computer Science, Eindhoven, The NetherlandsAnder Gray ANDER.GRAY@LIVERPOOL.AC.UKScott Ferson SCOTT.FERSON@LIVERPOOL.AC.UKInstitute for Risk and Uncertainty, University of Liverpool, United KingdomAbstractA method for constructing consonant predictive beliefsfor multivariate datasets is presented. We make use ofrecent results in scenario theory to construct a familyof enclosing sets that are associated with a predictivelower probability of new data falling in each given set.We show that the sequence of lower bounds indexed byenclosing set yields a consonant belief function. Thepresented method does not rely on the construction of alikelihood function, therefore possibility distributionscan be obtained without the need for normalization.We present a practical example in two dimensions forthe sake of visualization, to demonstrate the practicalprocedure of obtaining the sequence of nested sets.Keywords: Predictive beliefs; Consonant random sets;Generalization error; Imprecise probability; Evidencetheory.1. IntroductionA very common problem in engineering science is thatof computing with multivariate sample sets making theleast amount of assumptions. Multivariate sample sets aredifficult to deal with because even when it is possible tolearn a joint probability distribution, it is often very chal-lenging to characterize the uncertainty associated with anyparticular choice. We propose a method to characterize theuncertainty associated with constructing a joint distributiondirectly from the data using scenario theory. We show thatthe mechanism that constructs such joint distribution isdesigned to yield coherent lower predictive probabilitiesnicely fitting within the imprecise probability paradigm.We present a way to construct consonant random sets–or possibility measures–that is rooted in scenario theory[5]. We resort to the generalization property of scenariotheory whereby given a set of independent observations,the bounds on the predictive probability of a future obser-vation can be computed. Recent works including [8] haveelaborated on the complexity and the feasibility of suchcomputations in highly multi-dimensional datasets. A re-cent overview on scenario theory can be found here [14].This paper is also inspired by [12], whereby the problem ofinferential uncertainty is cast within the imprecise probabil-ity paradigm, see also [10, 11]. Other papers on this topicare [9, 1, 13, 7]. This work focuses on two main aspects: (i)we propose a method to systematically obtain a sequence ofenclosing sets with assigned probability bounds by meansof scenario theory, (ii) we show that the lower predictiveprobability–also known as predictive belief –associated tothe enclosing sets obtained as such, constitutes a possibilitydistribution, thus is a coherent lower probability. The advan-tage of resorting to scenario optimization to construct thesesets resides in the flexibility of choosing a parametric modelfor the shape of the enclosing sets that are guaranteed to ef-ficiently contain all the data. Moreover this method enablesthe encoding of both aleatory and epistemic dependencewithin the same framework.2. Problem StatementLet X be a random variable on a domain X with unknownprobability distribution PX . We would like to quantify thebeliefs held by an agent about a random future realization ofX from past observations X1, ...,Xn randomly drawn fromthe same distribution and mutually independent (iid). Thisquestion is typically addressed within probability theoryeither using Bayesian inference or with classical statisticalinference, however when the sample size is limited nei-ther approach yields a unique answer, without additionalmodelling assumptions, especially in the multivariate case.With scenario theory we aim to characterize the uncertaintyabout the distribution PX using a consonant predictive be-lief function given some random data (scenarios).2.1. Predictive beliefs and belief functionsGiven iid random samples X1, ...,Xn, with parent probabil-ity distribution PX , we want to produce a belief function,denoted by BelX := Bel(·;X1, · · · ,Xn), in such a way thatthe inequality BelX ≤ PX , will hold at least in 100(1−β )%of the cases. Which properties should BelX satisfy? Asmore and more data are gathered, it seems reasonable topostulate that BelX = PX , from a frequentist perspective© 2021 M. DeAngelis, R. Rocchetta, A. Gray & S. Ferson.CONSONANT PREDICTIVE BELIEFS WITH SCENARIO THEORY[7]. A sample set of infinite size should be equivalent toknowledge of the distribution of X , hence the belief func-tion should asymptotically become identical to PX . Forfinite n, it seems natural to impose that BelX will be lesscommitted than PX , which leads to BelX ≤ PX . However,because of limited knowledge, the inequality is often toostringent to hold in all cases. In other terms, assumingthe experiment is repeated indefinitely, we demand BelXto be less committed than PX most of the time, with atleast some prescribed long run frequency 1− β , whereβ ∈ (0,1) is a small number. In summary, the followingtwo requirements are typically imposed to the belief func-tion: (1) ∀A ⊆X ,BelX (A)→ PX (A), as n→ ∞, and (2)Pn(BelX ≤ PX )≥ 1−β . A belief function satisfying thesetwo requirements is called a predictive belief function atconfidence level 1− β . Several methods have been pro-posed to construct these belief functions, like in [7], usingmultinomial confidence intervals, and then further extendedto the continuous case [1]. In this work, we restrict our con-struction to sets that are mutually nested, and resort toscenario theory to compute the belief measure. The conso-nant belief function obtained as such will therefore satisfythe second requirement, but not the first requirement. Let usrecall now that belief functions are coherent lower probabil-ities. Given a power set 2X , the function m : 2X → [0,1]is called a basic belief assignment, if the following twoproperties hold: (i) m( /0) = 0, (ii) ∑A∈2X m(A) = 1. Themass m(A) can be interpreted as the part of the agent’sbelief allocated exactly to the hypothesis that X takes somevalue in A [15]. The subsets A ⊆X such that m(A) > 0are called the focal sets of m. For any focal set A ∈ 2X , itsbelief can be obtained summing up the masses of the focalelements B ∈ 2X contained in it,BelX (A) = ∑B:B⊆Am(B). (1)When the focal sets are nested, BelX is said to be consonant.Consonance has recently been used to build confidence pre-serving structures [3]. Eq. 1 constitutes a lower bound onthe true yet unknown probability PX (A) to the hypothesisthat X takes some value in A. Thus the belief function in Eq.1 falls in a subclass of coherent lower probabilities [2, 6].We propose an algorithm to construct recursively enclosingsets of predictive beliefs, which are also known as necessi-ties, directly from data. The resulting consonant structureprovides a lower bound on the probability of observing newdata in any given enclosing set. Note that by limiting thisapproach to consonant beliefs, the second reequirementcannot be reached for all events, even when n→ ∞. Thisis because the consonant belief structure is effectively apossibility distribution, which always corresponds to a mul-titude of probability distributions, thus never converges toa single probability distribution [16].3. Scenario theoryConsider a closed bounded set B ⊆ Rm, like one of therectangles in Figure 1 where m = 2. Let us consider avector of parameters z ∈Z ⊆ Rd that uniquely identify B,e.g. its size and location. Given a convex cost function f (z)e.g. the area of the rectangle, a family of convex constraintsz ∈ ZX ⊆ Rd , parametrized by the data X , and a sampleset X1, ...,Xn with d < n, of iid realizations, the followingoptimization programminz∈Zf (z)subject to: z ∈⋂i=1,...,nZXi ,(2)determines the optimal set B, while X1, ...,Xn are calledscenarios, which have to be interpreted as observationsfrom which one wants to make a decision, i.e. select anoptimal value of z. Eq. 2 is a standard convex problemthat can be numerically solved with common software.From this point on, the optimization of Eq.2 will only beused to identify the number of scenarios k that are activeconstraints, i.e. those points that prevent the cost functionfrom further improving.Definition 1 (Enclosing set of degree k) The optimal setB⊆Rm, that strictly contains n−k observations is referredto as an enclosing set of degree k, and it is denoted by Bk,where k is the number of observations strictly not containedin Bk, i.e. either on the borders or outside Bk.The observations lying exactly on the border of Bk arealso known as active scenarios. Note that k is a randomvariable because it is a function of the data, while Bk isa realization of the random set expressed by the belieffunction. Given a parametric set B, e.g. a hyper-box or anellipsoid, the optimization program in Eq.2 is tasked toidentify the number of active scenarios for a given subsetof observations that are to be enclosed. If the optimizationproblem is convex the optimal set is unique and can becomputed exactly.Definition 2 (Lower bound probability of degree k)The precise predictive probability of a given enclosing setof degree k: PX (Bk), has a lower bound pk, with assignedone-sided coverage probability. For a fixed and arbitrarilysmall β ∈ (0,1), and a number k ∈ Z+ of active scenariosthe coverage probability of the given set Bk isPn(pk ≤ PX (Bk))≥ 1−β , (3)Where Pn is the product probability (X1, ...,Xn) due to in-dependence of the samples. The (one-sided) lower boundpk is a random variable of the probability space of the datagenerating mechanism and can be computed using the for-mula and code provided in [4]. A code for the two-sided363CONSONANT PREDICTIVE BELIEFS WITH SCENARIO THEORYcase for both lower and upper bounds is provided in [8].The solution tk to the polynomial equation,βn+1n∑m=k(mk)tm−k−(nk)tn−k = 0, (4)is unique in the interval (0,1). Under the two assumptionsof uniqueness and non-degeneracy, it holds that pk = tk,see Theorem 2 in [4]. To compute tk a simple bisectionalgorithm can be used. More about the two assumptionscan also be found in [4]. The positive integer k can beinterpreted in multiple ways: (i) as a parameter defininghow complex is our decision z; (ii) the minimum numberof scenarios/samples that are necessary to reconstruct theoptimized z; (iii) an over-fitting parameter which constitutesthe key for constructing consonant sets as described in thenext section.4. Method for constructing consonantpredictive beliefs-15 -10 -5 0 5 10 15 20X1-50050100150200250300X2Figure 1: Consonant box-shaped enclosing sets.The proposed method for constructing sequentially en-closing sets is very straightforward. Let us consider theobservations Xi ∈R2 with i = 1, · · · ,n shown as scatter plotin Figure 1. A simple shape for the enclosing set is forexample a rectangle. The rectangle with the smallest areaenclosing the dataset Xi is uniquely determined by at mostk observations, which are the so called active scenarios.We denote this rectangle by Bk. Note that in the case of arectangle the number of support scenarios is at least two.Also note that the value of k is data dependent (random) andis affected by the functional form of B. Given a fixed and ar-bitrarily small β ∈ (0,1) the predictive belief pk associatedwith Bk can be computed finding the roots of the polyno-mial in Eq.3, which is independent on the shape of the set.the base of the Aztec pyramid shown from above in Figure1 corresponds to the rectangle-shaped set enclosing all theobservations. This set is associated with the largest pre-dictive belief. The algorithm that constructs the sequenceof nested rectangles proceeds by cumulating the numberof active scenarios k until all the observations have beenconverted to support scenarios. In particular we propose touse a simple arithmetic progression with common differ-ence equal to the number of active scenarios at the giveniteration, to encourage symmetry in the resulting sequenceof nested sets. The second iteration determines a rectanglewhose support scenarios are sought in the dataset strippedof the observations that were identified as active scenariosat the previous iteration. This procedure is repeated until nomore observations can be found in a given set, thus guaran-teeing consonance of the constructed rectangles. In Figure2 the two marginal plausibility distributions that result fromthe procedure are shown. Because of Eq.3, any sequenceof active scenarios verifying 0 = k0 < k1 < · · · < kn = n,implies1≥ pk0 ≥ pk1 ≥ ·· · ≥ pkn = 0, (5)X ⊇ Bk0 ⊇ Bk1 ⊇ ·· · ⊇ Bkn = /0. (6)-15 -10 -5 0 5 10 15 20X100.20.40.60.811-BelX-50 0 50 100 150 200 250 300X200.20.40.60.811-BelXFigure 2: Plausibility contours obtained from Eqs.5 and 6,projecting onto the coordinate axes.4.1. The lower bound pk is a predictive beliefWe are ready to state the main result of this paper:Theorem 3 The lower bound pk is a sequence of validpredictive beliefs for any k ∈ Z+ such that 0 = k0 < k1 <· · ·< kn = n.Proof Eq.3 guarantees that the lower bound pk is alwaysnon-increasing with k, see also Eq. 5. Two sets obtainedfrom the same scenario optimization program, but withdifferent number of active scenarios are always weaklynested to one another , see Eq. 6. This suffices to provide astrong condition for the sequence of lower bounds pk to bevalid beliefs.364CONSONANT PREDICTIVE BELIEFS WITH SCENARIO THEORYAcknowledgementsThe authors would like to thank Dominik Hose and Alexan-der Wimbush for the fruitful discussions on possibilitydistributions and likelihood functions; and Ryan Martin forthe insightful review of the manuscript. The work was sup-ported by the “ITEA3-2018-17030-DayTiMe” project andby the UKRI Engineering & Physical Sciences ResearchCouncil with grant no. EP/R006768/1.References[1] A. Aregui and T. Denœux. Constructing predictivebelief functions from continuous sample data usingconfidence bands. In 5th International Symposiumon Imprecise Probability: Theories and Applications,2007.[2] T. Augustin, F.P.A. Coolen, G. De Cooman, andM.C.M. Troffaes. Introduction to imprecise prob-abilities. John Wiley & Sons, 2014.[3] M. S. Balch. Mathematical foundations for a theoryof confidence structures. International Journal ofApproximate Reasoning, 53(7):1003–1019, 2012.[4] M. C. Campi and S. Garatti. Wait-and-judge scenariooptimization. Mathematical Programming, 167(1):155–189, Jan 2018. ISSN 1436-4646. doi: 10.1007/s10107-016-1056-9.[5] M. C. Campi, S. Garatti, and F. A. Ramponi. A gen-eral scenario theory for nonconvex optimization anddecision making. IEEE Transactions on AutomaticControl, 63(12):4067–4078, Dec 2018. ISSN 0018-9286. doi: 10.1109/TAC.2018.2808446.[6] T. Denœux. Reasoning with imprecise belief struc-tures. International Journal of Approximate Reason-ing, 20(1):79–111, 1999. ISSN 0888-613X. doi:https://doi.org/10.1016/S0888-613X(00)88944-6.[7] T. Denœux. Constructing belief functions from sam-ple data using multinomial confidence regions. In-ternational Journal of Approximate Reasoning, 42(3):228–252, 2006. ISSN 0888-613X. doi: https://doi.org/10.1016/j.ijar.2006.01.001.[8] S. Garatti and M. C. Campi. Risk and complexity inscenario optimization. Mathematical Programming,pages 1–37, November 2019. ISSN 1436-4646. doi:10.1007/s10107-019-01446-4.[9] D. Hose and M. Hanss. On data-based estimation ofpossibility distributions. Fuzzy Sets and Systems, 399:77–94, 2020. ISSN 0165-0114. doi: https://doi.org/10.1016/j.fss.2020.03.017. Fuzzy Intervals.[10] C. Liu and Ryan M. Inferential models and possibilitymeasures, 2020.[11] R. Martin. An imprecise-probabilistic characteriza-tion of frequentist statistical inference, 2021.[12] R. Martin and C. Liu. Inferential Models: Reason-ing with Uncertainty. Chapman Hall/CRC Mono-graphs on Statistics and Applied Probability. CRCPress, 2015. ISBN 9781439886519.[13] M.H. Masson and T. Denœux. Inferring a possibil-ity distribution from empirical data. Fuzzy sets andsystems, 157(3):319–340, 2006.[14] R. Rocchetta, Q. Gao, and M. Petkovic. Soft-constrained interval predictor models and epistemicreliability intervals: A new tool for uncertainty quan-tification with limited experimental data. Mechan-ical Systems and Signal Processing, 161:107973,2021. ISSN 0888-3270. doi: https://doi.org/10.1016/j.ymssp.2021.107973.[15] G. Shafer. A Mathematical Theory of Evidence. Lim-ited paperback editions. Princeton University Press,1976. ISBN 9780691100425.[16] M. Troffaes and S. Destercke. Probability boxes ontotally preordered spaces for multivariate modelling.International Journal of Approximate Reasoning, 52(6):767–791, 2011. ISSN 0888-613X. doi: https://doi.org/10.1016/j.ijar.2011.02.001.365

https://strathprints.strath.ac.uk/83162/1/De_Angelis_etal_PMLR_2021_Constructing_consonant_predictive_beliefs_from_data.pdf

Constructing Consonant Predictive Beliefs from Data with Scenario Theory

Abstract

Similar works

Full text

Available Versions

University of Liverpool Repository

University of Strathclyde Institutional Repository

University of Liverpool Repository