In this paper we further develop the idea that the PAC-Bayes prior can be defined based on the data-generating distribution. In particular, following Catoni [1], we refine some recent generalisation bounds on the risk of the Gibbs Classifier, when the prior is defined in terms of the data generating distribution, and the posterior is defined in terms of the observed one. Moreover we show that the prior and the posterior distributions can be tuned based on the observed samples without worsening the convergence rate of the bounds and with a marginal impact on their constants

ANGUITA, DAVIDE

ONETO, LUCA

RIDELLA, SANDRO

English

Oneto Luca

Ridella Sandro

Anguita Davide

Archivio della Ricerca - Università di Pisa

Tuning the distribution dependent prior in the PAC-Bayes framework based on empirical data

Archivio istituzionale della ricerca - Università di Genova

Tuning the Distribution Dependent Prior in thePAC-Bayes Framework based on Empirical DataLuca Oneto1, Sandro Ridella2, and Davide Anguita11 - DIBRIS - University of GenovaVia Opera Pia 13, I-16145 Genova - Italy2 - DITEN - University of GenovaVia Opera Pia 11A, I-16145 Genova - ItalyAbstract. In this paper we further develop the idea that the PAC-Bayesprior can be defined based on the data-generating distribution. In partic-ular, following Catoni [1], we refine some recent generalisation bounds onthe risk of the Gibbs Classifier, when the prior is defined in terms of thedata generating distribution, and the posterior is defined in terms of theobserved one. Moreover we show that the prior and the posterior distri-butions can be tuned based on the observed samples without worseningthe convergence rate of the bounds and with a marginal impact on theirconstants.1 IntroductionIt is well known that combining the output of several classifiers results in muchbetter performance than using any one of them alone. In fact many state-of-the-art algorithms search for a weighted combination of simpler classifiers [2]:Bagging [3], Boosting [4] and Bayesian approaches [5] and, in some sense, Kernelmethods [6] and Neural Networks [7]. The major open problem in this scenariois how to weight the different classifiers in order to obtain good performance [8,9, 1], and how this performance can be assessed [10, 6, 11, 12, 13, 14]. The PAC-Bayes approach [15, 11, 1, 16, 17, 2] is one of the sharpest analysis frameworksin this context, since it can provide a tight bound on the risk of the GibbsClassifier (GC), and the Bayes Classifier (BC) [2]. The GC choses a classifiersin a set according to a posterior distribution each time a new sample has to beclassified [17]. In particular, in the PAC-Bayes framework, a prior distributionover the classifiers must be defined before seeing the data, then, based on theavailable data, a posterior distribution is chosen, and the risk of the associateGC is estimated, based on the empirical risk and the divergence between theprior and posterior distributions [11]. The PAC-Bayes analysis bounds the riskof the GC [11, 16], while the C-bound bounds the error of the BC, also calledweighted majority vote classifier, based on the properties of the GC [2].The major weakness in the conventional PAC-Bayes approach is that a pos-terior distribution that minimises the divergence between prior and posteriordistributions must be chosen, since this divergence is part of the bound [17, 18].In order to address this issue, Catoni [1] proposed a localised PAC-Bayes analysis,which exploys a Boltzmann prior distribution defined in terms of the unknowndata distribution. Note that, since the prior depends on the data generating dis-tribution, the PAC-Bayes analysis is still valid because the prior is defined beforeobserving the data [1]. By tuning the prior to the distribution, Catoni was ableto remove the divergence term from the bound, hence significantly reducing thecomplexity penalty [1]. More recently, this approach has been further developedin [17] but the prior still has a free parameter that needs to be fixed beforeobserving the data and can affect the divergence penalty and consequently thetightness of the bound.In this paper, in Section 3, we further refine the bound proposed in [17]and recalled in Section 2. Moreover, in Section 4 we show that the distributiondependent Boltzmann prior distribution developed by Catoni [1] can be tunedbased on the observed samples in order to optimise the bounds over the risk ofthe GC. In particular we will show that this optimisation does not change theconvergence rate of the bounds and has a marginal impact over the constantsinvolved in the bounds.2 State-of-the-art ResultsLet us consider a set of n labeled samples Dn = {(X1, Y1), · · · , (Xn, Yn)} ={Z1, · · · , Zn} drawn i.i.d. according to an unknown probability distribution µover the cartesian product between the input space X and the output spaceY={−1, 1}. Let us consider a function f∈F , where f :X→Y=[−1, 1]. The errorof f in approximating µ is measured with reference to some [0, 1]-bounded lossfunction `:F×(X×Y)→[0, 1]. Then the risk of f , and its empirical estimator, canbe defined as L`(f)=EZ {`(f, Z)} and L̂`(f)=1/n∑ni=1 `(f, Zi). The GC drawsa function f∈F , according to a probability distribution Q over F , each time alabel for an input X∈X is required. For the GC, referred as GQ, we can de-fine its risk together with its empirical counterpart [17]: L`(GQ)=Ef∼Q{L`(f)},and L̂`(GQ)=Ef∼Q{L̂`emp(f)}. Given two probability distributions Q and Pover F , let us denote with KL[Q||P ] the Kullback-Leibler Divergence (KLD)between P and Q, while kl[q||p] is the KLD for the Binomial distributionkl[q||p]=q ln [q/p] +[1−q] ln [1−q/1−p] where, thanks to the Pinsker’s Inequalitywe can state that |q−p|≤√1/2kl[q||p]. Finally, let us recall the definition of alast fundamental quantity in the PAC-Bayes framework, which is a weightedsum of binomial coefficients ξn=∑nk=0(nk)(kn)k(1− kn)n−kwhere√n≤ξn≤2√n[16]. Based on these preliminary definitions we can recall the state of the artbound on the risk of the BC.Theorem 1 ([16]). For any probability distribution P over F , chosen beforeseeing Dn, ∀Q we have P{kl[L̂`(GQ)||L`(GQ)]≥ [KL[Q||P ]+ ln [ξn/δ]]/n}≤δ.The main problem of the PAC-Bayes Theory regards the choice of P and Q.Q should fit our observations, but, at the same time, Q should be close to P ,in order the minimise the KLD. The milestone result of [1], later extended by[17], proposes to use a Boltzmann prior distribution P which depends on thedata generating distribution µ. In particular, let us suppose that the densityfunction associated to P is p(f)=cpe−γL`(f), where γ∈[0,∞) and cp is a nor-malisation term. Basically, this distribution gives more importance to functionsthat possess small risk. If we choose as posterior Q a distribution which givesmore importance to functions with small empirical risk with the following den-sity function q(f)=cqe−γL̂`(f), where cq is a normalisation term, it can be provedthat this theorem, built on the result of Theorem 1, holds.Theorem 2 ([17]). Given the prior P and the posterior Q defined above, wecan state that P{KL[Q||P ]≥KL1(γ, δ, n).=γ2/n+γ√2 ln [ξn/δ]/n}≤2δ. Consequently,we have that P{kl[L̂`(GQ)||L`(GQ)]≥ [KL1(γ, δ, n)+ ln [ξn/δ]]/n}≤3δ.3 Sharpening the Risk BoundIn this part of the paper we prove that the result of Theorem 2 can be furtherimproved (Theorem 3). Furthermore, we show that, if the loss function exploitedfor assessing the risk of the BC is the same used to define the prior and posteriordefined by Catoni [1] (generally they may be different), then Theorem 3 can befurther improved (Theorem 4).Theorem 3. Under the same hypothesis of Theorem 2, we can state the follow-ing inequality P{KL[Q||P ]≥KL2(γ, δ, n)}≤2δ where KL2(γ, δ, n)≤KL1(γ, δ, n) andKL2(γ, δ, n).=γ√γ√2 ln [1/δ]/n/4n+ ln [ξn/δ]/2n+γ2/16n2+γ√ln [1/δ]/2n+γ2/4n. Moreover,we also have that P{kl[L̂`(GQ)||L`(GQ)]≥ [KL2(γ, δ, n) + ln [ξn/δ]]/n}≤3δ.Proof. Let us consider KL[Q||P ], since p(f)=cpe−γL`(f), then cp=p(f)eγL`(f) andconsequently:KL[Q||P ] = Ef∼Q {ln [q(f)/p(f)]}=Ef∼Q{ln[cqe−γL̂`(f)/cpe−γL`(f)]}= ln[cq/cp]+Ef∼Q{γ[L`(f)−L̂`(f)]}=γ[L`(GQ)−L̂`(GQ)]− ln∫F p(f)eγ[L`(f)−L̂`(f)]df≤γ[L`(GQ)−L̂`(GQ)]−γ[L`(GP )−L̂`(GP )]. (1)By bounding the last two terms in square brackets through the Pinsker’s in-equality and Theorem 1 and by solving with respect to KL[Q||P ], Theorem 2 canbe derived. This is indeed sub-optimal. Since P is defined before seeing Dn wecan exploit the Hoeffding’s inequality [19] in order to bound the second termand, consequently, with probability (1− 2δ), we have that:γ[L`(GQ)−L̂`(GQ)]−γ[L`(GP )−L̂`(GP )] ≤ γ√KL[Q||P ]+ln[ξn/δ]2n +γ√ln[1/δ]2n . (2)By bounding Eq. (1) through Eq. (2), solving with respect to KL[Q||P ], andplugging the result in Theorem 1, the statement in this Theorem is proved.Theorem 4. Under the same hypothesis of Theorem 2, if the losses used to de-fine P , Q, L̂`(GQ) and L`(GQ) are the same, we can state the following inequal-ity P{kl[L̂`(GQ)||L`(GQ)]≥[γ|L`(GQ)−L̂`(GQ)|+γ√ln [1/δ]/2n+ ln [ξn/δ]]/n}≤2δ.Proof. The proof can be derived from Eq. (1). Since the losses used to define P ,Q, L̂`(GQ) and L`(GQ) are the same we simply have to bound the last one ofthe two terms in square brackets of Eq. (1) through the Hoeffding’s inequality[19], as in Theorem 3, and plug the result in Theorem 1.The results of Theorems 3 and 4 improve over the state-of the-art bound ofTheorem 2.4 Tuning the Prior on the Available DataUnfortunately, even if an optimal choice of γ that minimises the above boundsexists, this parameter must be chosen before seeing the data in order to maintaintheir validity. In this section we deal with this problem by tuning γ based on theavailable data. The first step consists in proving the following theorem whichbounds the risk of the GC when, given P γ and Qγ defined in Section 2 fordifferent values of γ ∈ {γ1, · · · , γm}, one choses the γi with i ∈ {1, · · · ,m} thatminimises the bound on the GC risk.Theorem 5. Given the prior P γ and Qγ defined in Section 2 for different valuesof γ ∈ {γ1, · · · , γm}, we can state that, ∀γ ∈ {γ1, · · · , γm}:1. P{kl[L̂`(GQ)||L`(GQ)]≥ [KL1(γ, δ, n)+ ln [ξn/δ]]/n}≤3mδ,2. P{kl[L̂`(GQ)||L`(GQ)]≥ [KL2(γ, δ, n) + ln [ξn/δ]]/n}≤3mδ.Moreover, if the losses used to define P γ , Qγ , L̂`(GQγ ), and L`(GQγ ) are thesame, we can state that, ∀γ ∈ {γ1, · · · , γm}:3. P{kl[L̂`(GQ)||L`(GQ)]≥[γ|L`(GQ)−L̂`(GQ)|+γ√ln [1/δ]/2n+ ln [ξn/δ]]/n}≤2mδ.Proof. In order to prove the statement the union bound must by applied overthe different γ ∈ {γ1, · · · , γm} to Theorems 2, 3 and 4.Given the result of Theorem 5, and since KL2(γ, δ, n)≤KL1(γ, δ, n) (Theorem3), if γ is chosen among m=(ξn)η with η≥0 points equally spaced in logarithmicscale in [γmin, γmax], we can state that with probability (1−δ):kl[L̂`(GQ)||L`(GQ)]≤ KL2(γ,δ/3ξηn,n)+(1+η) ln[ξn]+ ln[3/δ]n (3)=γ√γ√2η ln[ξn]+2 ln[3/δ]4n +(1+η) ln[ξn]+ ln[3/δ]2n +γ216n2+γ√η ln[ξn]+ ln[3/δ]2n +γ24n+(1+η) ln[ξn]+ ln[3/δ]n≤ KL1(γ,δ/3ξηn,n)+(1+η) ln[ξn]+ ln[3/δ]n =γ2n +γ√2(1+η) ln[ξn]+2 ln[3/δ]n +(1+η) ln[ξn]+ ln[3/δ]nMoreover, if the losses used to define P γ , Qγ , L̂`(GQγ ), and L`(GQγ ) are thesame, we can state that with probability (1−δ):kl[L̂`(GQ)||L`(GQ)]≤γ|L`(GQ)−L̂`(GQ)|+γ√η ln[ξn]+ ln[2/δ]2n +(1+η) ln[ξn]+ ln[2/δ]n (4)Note that when m=1 (which means that η=0 and consequently Theorems 2, 3,and 4 can be applied) the bound has the same convergence rate as when γ ischosen among m=(ξn)η∈[nη/2, 2nη/2] possible values at the expenses of a slightlyworse constant η≥0.5 DiscussionIn order to get more insights on the proposed bounds, we test them on anartificial problem. In particular, a dataset is created, consisting of n samples ina bidimensional input space: n/2 are equally spaced on a circle of radius 1 andcenter [−c,−c]T while the others n/2 are equally spaced on a circle of radius 1and center [c, c]T . We choose c∈{1/2, 1}, γmin=10−2, and γmax=103. The HardLoss `(f, Z)=1−Y sign[f(X)]/2 is exploited for the P γ , Qγ , L̂`(GQγ ), and L`(GQγ ).We choose, as hypothesis space F , all the possible linear separators in the inputspace. In Figure 1 we have reported the upper bound of L`(GQ) obtained byapplying Theorems 2, 3, 4, and 5 (results 1, 2, and 3) in different situations.In Figures 1(a) and 1(d) we report the comparison of Theorems 2, 3 and 4 fordifferent values of γ: note that Theorems 3 and 4 improve over the state of theart bound of Theorems 2, in particular Theorem 4 is the sharpest one (notethat in this case we cannot choose the γ for which the bound is minimum).In Figures 1(b) and 1(e) we report the comparison between Theorem 4 and thebound obtained by adopting the strategy of Theorem 5 when we look at m=(ξn)ηvalues of γ: note that the bound is worse as soon as we increase η but in thiscase we can choose the γ which minimises the bound and the loss, in terms oftightness of the risk bound, is smaller with respect to the loss of choosing awrong γ. Finally in Figures 1(c) and 1(f) we report the upper bound of L`(GQ),for different value of n, for η = 1 and γ=γ∗, where γ∗ is the one that gives theminimum of Theorem 5 (results 1, 2, and 3) as described in Section 4 againstthe ideal case when γ∗ is known a priori and Theorem 2, 3, and 4 can be useddirectly: obviously the bounds are looser but the loss is negligible if we take intoaccount that γ has been tuned based on the observed samples.The result that we have just presented, even if preliminary, gives interestinginsights on the Theorems reported in this paper. In particular, in this paper wehave shown that our results improve over the state-of-the-art PAC-Bayes GCrisk bounds and that is possible to tune, in a fully empirical fashion, both theprior and posterior PAC-Bayes distributions without impacting on the rates ofthe bounds and with a marginal impact on their constants.References[1] O. Catoni. PAC-Bayesian Supervised Classification. Institute of Mathematical Statistics,2007.[2] P. Germain, A. Lacasse, A. Laviolette, M. Marchand, and Roy J. F. Risk bounds for themajority vote: From a pac-bayesian analysis to a learning algorithm. JMLR, 16(4):787–860, 2015.[3] L. Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.[4] R. E. Schapire, Y. Freund, P. Bartlett, and W. S. Lee. Boosting the margin: A newexplanation for the effectiveness of voting methods. The Annals of Statistics, 26(5):1651–1686, 1998.[5] A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin. Bayesian Data Analysis, volume 2.Taylor & Francis, 2014.[6] V. N. Vapnik. Statistical Learning Theory. Wiley New York, 1998.-2 -1 0 1 2 3log10 γ00.20.40.60.81Lℓ(GQ)Theorem 2Theorem 3Theorem 4(a) c=1, n=100-2 -1 0 1 2 3log10 γ00.20.40.60.81Lℓ(GQ)Theorem 4Theorem 5, Result 3, η = 1Theorem 5, Result 3, η = 2Theorem 5, Result 3, η = 3(b) c=1, n=10050 100 150 200n00.10.20.30.40.5Lℓ(GQ)Theorem 2Theorem 3Theorem 4Theorem 5, Result 1Theorem 5, Result 2Theorem 5, Result 3(c) c=1, γ=γ∗, η = 1-2 -1 0 1 2 3log10 γ0.40.50.60.70.80.91Lℓ(GQ)Theorem 2Theorem 3Theorem 4(d) c=1/2, n=100-2 -1 0 1 2 3log10 γ0.40.50.60.70.80.91Lℓ(GQ)Theorem 4Theorem 5, Result 3, η = 1Theorem 5, Result 3, η = 2Theorem 5, Result 3, η = 3(e) c=1/2, n=10050 100 150 200n0.40.50.60.70.8Lℓ(GQ)Theorem 2Theorem 3Theorem 4Theorem 5, Result 1Theorem 5, Result 2Theorem 5, Result 3(f) c=1/2, γ=γ∗, η = 1Fig. 1: Performance of the different bounds over the artificial problem.[7] B. M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995.[8] S. Nitzan and J. Paroush. Optimal decision rules in uncertain dichotomous choice situa-tions. International Economic Review, 23(2):289–97, 1982.[9] D. Berend and A. Kontorovitch. Consistency of weighted majority votes. In NIPS, 2014.[10] S. Floyd and M. Warmuth. Sample compression, learnability, and the vapnik-chervonenkisdimension. Machine Learning, 21(3):269–304, 1995.[11] D. A. McAllester. Some pac-bayesian theorems. In Computational Learning Theory,1998.[12] O. Bousquet and A. Elisseeff. Stability and generalization. JMLR, 2:499–526, 2002.[13] P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk boundsand structural results. JMLR, 3:463–482, 2003.[14] P. L. Bartlett, O. Bousquet, and S. Mendelson. Local rademacher complexities. Annalsof Statistics, pages 1497–1537, 2005.[15] M. D. Donsker and S. R. S. Varadhan. Asymptotic evaluation of certain markov pro-cess expectations for large time, i. Communications on Pure and Applied Mathematics,28(1):1–47, 1975.[16] P. Germain, A. Lacasse, F. Laviolette, and M. Marchand. Pac-bayesian learning of linearclassifiers. In ICML, 2009.[17] G. Lever, F. Laviolette, and J. Shawe-Taylor. Tighter pac-bayes bounds throughdistribution-dependent priors. Theoretical Computer Science, 473:4–28, 2013.[18] E. Parrado-Hernández, A. Ambroladze, J. Shawe-Taylor, and S. Sun. Pac-bayes boundswith data dependent priors. JMLR, 13(1):3507–3531, 2012.[19] W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal ofthe American Statistical Association, 58(301):13–30, 1963.

https://iris.unige.it/bitstream/11567/850987/2/C042.pdf

Tuning the distribution dependent prior in the PAC-Bayes framework based on empirical data

Abstract

Similar works

Full text

Available Versions

Archivio della Ricerca - Università di Pisa

Archivio istituzionale della ricerca - Università di Genova