Ambergen, A.W.

Schaafsma, W.

English

CWI's Institutional Repository

stichting mathematisch centrum AFDELING MATHEMATISCHE STATISTIEK (DEPARTMENT OF MATHEMATICAL STATISTICS) A.W. AMBERGEN & W. SCHAAFSMA SW 96/83 INTERVAL ESTIMATES FOR POSTERIOR PROBABILITIES IN A MULTIVARIATE NORMAL CLASSIFICATION MODEL ~ MC AUGUSTUS kruislaan 413 1098 SJ amsterdam M!J011itEK MATHEMATISCH CENTRUI\# AA1STERDAM Printed at the Mathematical Centre, Kruislaan 413, Amsterdam, The Netherlands. The Mathematical Centre, founded 11 February 1946, is a non-profit institution for the promotion of pure and applied mathematics and computer science. It is sponsored by the Netherlands Government through the Netherlands Organization for the Advancement of Pure Research (Z.W.O.). 1980 Mathematics subject classification: 62H30 Copyright© 1983, Mathematisch Centrum, Amsterdam Interval estimates for posterior probabilities in a multivariate normal classification model by A.W. Ambergen & W. Schaafsma*) ABSTRACT This paper is devoted to the asymptotic distribution of estimators for the posterior probability that a p dimensional observation vector originates from one of k normal distributions with identical covariance matrices. The estimators are based on training samples from the k distributions involved. Observation vector and prior probabilities are regarded as given constants. The validity of various estimators and approximate confidence intervals is investigated by simulation experiments, KEY WORDS & PRHASES: Estimating posterior probabilities, classification, discriminant analysis, multivariate normal distribu-tions *)Department of Mathematics, University of Groningen 1. INTRODUCTION Suppose that an observation x comes from one of k populations rrh, h = I, ... ,k, which are characterized by p-dimensional multivariate normal distributions with equal covariance matrices. Accordingly let fh denote the p.d.f. of Np(µh,I), h = I, ••• ,k. The parameters µ 1 , ••• ,µk, I are unknown. We assume that past experience is available in the form of outcomes of inde-pendent random vectors ¾ 1, ••• ,¾nh, h = I, ••• ,k, ¾i having density fh. Let ph denote the prior probability that the observation comes from Ilh, h = I, ... ,k I~=l Ph= I. For p 1, ••. ,pk and x given the posterior probabilities k ( I. I) P . I = p f (x) / l phfh (x) , t = I , ••• , k t X t t h=l are considered as unknown parameters which are to be estimated from the training samples. T Let R = (R11 , ••• ,R 1 ) denote any of the estimators for •Ix xT kx P = (p 1 , ••• ,pkl ) to be defined in section 2. We shall prove that • Ix Ix x I ni(R -p 1 ) is asymptotically normal with expectation zero and a singular • 1X • X dispersion matrix. Application to practice requires that the unknown parameters in the asymptotic covariance matrix are replaced by suitable estimates. The diagonal elements of the thus obtained estimated asymptotic covariance matrix provide the means of constructing asymptotic confidence intervals for the posterior probabilities separarely. The whole matrix is needed if one wants to apply a Scheffe-type method for judging linear combinations. Pairwise comparisons might be treated by applying theory for the case k = 2 where certain exact moments can be exploited (see e.g. SCHAAFSMA-VAN VARI< (1979)). The main purpose of this paper is to present the asymptotic variances and covariances of the R 's as means of expressing tlx the involved uncertainties. Most of the literature about estimating posterior probabilities deals with the case k = 2. The case p = I, k = 2 1s considered in SCHAAFSMA-VAN VARK (1977). The case p ~ I, k = 2 can be found in SCHAAFSMA-VAN VARI< (1979). In AMBERGEN-SCHAAFSMA (1983) the extension top~ I, k ~ 2, with no assumption about the equality of covariance matrices, is considered. The corresponding theory is easier than that presented in this paper (p ~ I, 2 k ~ 2, E1= •.. =Ek) because here we have dependence of the density estimators. Apart from the "estimative" methods used in this paper the "predictive" method of GEISSER (1964) has been discussed in the literature. AITCHISON, HABBEMA and KAY (1977) isacomparisonof the two methods. McLACHLAN (1977) studies the bias of sample based posterior probabilities. McLACHLAN (1979) compares the bias of classical plug-in estimators with that of predictive estimators. A recent reference is RIGBY (1982) who constructs credibility intervals for the posterior probabilities in order to compare the estimative and predictive estimators. 2. DEFINITION OF THE ESTIMATORS The densities of the populations are given by (2. 1) h = 1, .•. ,k where (2.2) Hence (2.3) /J.2 x;h 2 k 2 pt]x = pt exp(-½!J. )/ \ Ph exp(--21/J. ) x;t l x;h h=l Fork= 2 is it useful to rewrite (2.3) as (2.4) because then an approximate confidence interval for pt!x can be obtained by transforming the approximate confidence interval for tJ.2 - tJ. 2 3 based on x;t x; -t exact moments (see SCHAAFSMA-VAN VARK (1979) and RIGBY (1982), who used a similar approach in a Bayesian context). Let¾ denote the mean of the h-th sample and S the pooled matrix of cross-products: ~ (2.5) I ¾i, i=l k s = I h=l ~ I i=l It sometimes happens that extra samples are available for estimating S. Therefore, instead of S ~ W (n-k,L) where n = L.n., we shall work with p i i 3 Sf~ WP (f,L). The maximum likelihood estimator R~~~ for Ptl( is obtained by plugging in the estimators~ for µh, (f+k)- 1s for Land~;;~)= (f+k)v!;h for "';;h where (2.6) 2 T -1 V = (x-X.) Sf (x-X. ). x;h -11 -n Other estimators for p I are obtained by plugging in unbiased estimators t X for various parameters in (2.2) or (2.3). This gives us the following estimators for l',2 h x; ~2(1) = fV 2 x;h x;h (2. 7) z2 c2) 2 = (f-p-1 )V x;h x;h ~2(3) 2 -I (f-p-1 )V -p~ x;h x;h -I -I -1 2 -I 2 -1 -I based on ESf = fi, ESf = (f-p-1) L and EVx;h = (f-p-1) "'x;h +¾ (f-p-1) p respectively. By plugging into (2.3) we obtain the estimators (2.8) (") ~2 (j) k ~2 (j) RtJlx = p exp(-½L'I )/ l Ph exp(-½L', ·h ) t x;t h=l x, (t=l , ... ,k; j=O, ... ,3). 3. THE ASYMPTOTIC DISTRIBUTION OF THE ESTIMATORS 2 All estimators for "'x;h' suggested in section 2, are asymptotically equivalent. In this section it is therefore sufficient to focus on fV 2 h x; The corresponding estimator Rtlx = R(l) for Ptlx has the same asymptotic tix (") ._ T distribution as each of the other estimators Rtlx (J-0,2,3). (R1 lx'''''~lx) is asymptotically efficient and the asymptotic covariance matrix follows from Fisher's information matrix. Elaborating on this and related principles we have to consider the inverse Wishart distribution. LEMMA 3.1. If wf ~ Wp(f,E) then (3. l) and (3.2) with l· -1 Lf2(f vec(Wf)-vec(E)) + N 2 (0,A) p l -1 -I Lf2(f vec(Wf ) -vec(E )) + N 2(0,B) Aijki = CJikCJjl + CJilCJjk ik . l .k if. B CJ CJ J + CJJ CJ ijki = p 4 where we use the notations M .. 1_ 0 = M ~or Ma p 2 x p 2 matr~x, l\.,{.. (J·-1 )p+i, (l-J )p+k J' (/ _ ij _ - I 1.J CJ .. - L . and E - (E ) ..• l.J l.J l.J PROOF. (3.1) is an immediate consequence of the multivariate central limit theorem. (3.2) follows from the o-method: Bijkl = I rstu a l.J a kl (_CJ_) A (-CJ-) clCJ rstu clCJ . rs tu The proof is completed by using clCJ1.J ia. Bj --=-CJ CJ D cl CJ a S LEMMA 3.2. If f + 00 , n. /f • bh > 0 (h=I , ... ,k), v2 2 2 2 T h x A = (A 1, ••• ,A k) then x x; x; 2 2 = (V 1 , ••• ,v k) and x; x; (3.3) where r is determined by 2 + 2A4 rh h = 4Ax;h/bh ' x;h (3 .4) T -I 2 h I- t. rh,t = 2{(x-µh) E (x-µt)} PROOF. The independent random variables x1, •.• ,~,sf satisfy -1 1 -I -I L Np(O,bh E) and f 2 (fSf -E ) • N 2(0,B) where Bis defined in Sf in section 2. Consider the p~rtial derivatives and cJ6. 2 . x;1. aµ. J -] = -2 r (x-µ. ) 0 .. 1. 1.J i,j = I, •.• ,k i L f2(~ -µh) • lemma 3.1 and = (x-µ.) (x-µ.) 1.a, Jf3 1. = l, ... ,k; a. 'f3 = I' ... ,p where a .. = l if i = J and= 0 if if j. 1.J The a-method gives cJ6.2 • T r .. = ( x;J) b:t i.1. \ aµ. i J and r .. = 1.J I rstu 2 2 ( a~ ·\ (cJ6. .) x;1. B x;J \ aars } rs tu\ aa tu (3.4) follows by simple computation. i f J THEOREM 3.3. If f + 00 , 1\/f • bh > 0 (h=l, ..• ,k) R .Ix and P.lx = (pl lx'··•,Pklx)T then ! (3. 7) Lf 2 (R f -p I ) • Nk(O,fff) , X • X where r is determined by (3.5) and f by (3. 8) t f h. PROOF. With the a-method where f is the matrix of partial derivatives. 4. FOUR METHODS TO CONSTRUCT CONFIDENCE INTERVALS An approximate 100(1-a.)% confidence interval for ptlx is given by 5 ( 4. 1) where R(~) has been defined in (2.8) and t ix (4.2) 6 with u 1 defined by P(U > u!,..,) = ½a if U has a standard normal distribution. 2a i~ The estimators f(j) and ;CJ) for the corresponding parameters in (3.4) and (3.8) are obtained by plugging in the estimators R(jl) for p l , ~2 ~hj) for 2 T -1 t x t x x, t:, ·h and the parameter (x-µh) I (x-µ ) in (3.4) is estimated by Xi -1 (') t ·(O) (1) Q(J)(x-~)S (x-Xt) + c J , j = 0, .•• ,3, where b = f+k, b = f, b( 2) = f-p-1, b(3) = f-p-1, c(O) = c(l) = c(2) = 0 and c(3) = -p~1oh t with ' o = 1(0) if h = t (hit). h, t 5. SIMULATION EXPERIMENT An overall comparison of small sample performance of the estimators R(~) and the approximate confidence tix . 1 R ( ,· ) + 1 ( ,· ) ( • -O 3 ) . 1nterva s - 2L J- , ... , 1s t X t X rather complicated because the performance depends on the very large number of parameters (5. I) where t indicates the number of the density from which the score vector has been drawn. We selected 500 parameter points for the simulation experiments to be performed. The results for the chosen parameter points are rather accurate because we did the following for each point: compute pt Ix' generate 1000 times a set of training samples and compute each time R~i ~, L~i ~ (j =O, ••. , 3) • Count the number of times the interval contains the true value pt Ix. This number, divided by IO, should be compared with the value 100 (1-a). The 500 points were grouped into 25 clusters of each 20 points. Within a cluster only the x vectors differ because they were drawn independently. For the points within a cluster the same training set was used. We made the restrictions t = I, -I a= 0.05, µ 1 = Op, I= Ip, ph = k (h=l, •.• ,k) and considered only pllx 7 which is the most important, because largest, posterior probability. For each cluster we averaged the results of the 20 points. These averaged results with their standard deviations are presented in table l; a cluster corresponds with a row in the table. In order to get a nice layout of the table we intro-duce the following notations: n = (nl ' .•• ,~)' µ = (µ 1 ; ... ; µk) T T T T a = (O,O,O,O) ; b = (2,0,0,0) ; C = (0,2,0,0) ; d = (l,1,1,1) T T T T e = (1,1,0,0); f = (0,0,2,0) ; g = (0,0,0,2) ; h = (0,0,1,1) 14 = (1,1,1,l); 18 = (14;l4); m4 = (0,1,0,1); ms = (m4 ;m4) . d d f h . . R(j) c·-o 3· -1) Bi.as, m.s.e. an m.a .. o t e point estimators I J- , ... , , t- were t X also studied. CONCLUSIONS For the chosen parameter points we conclude that the m.l. estimator R~I~ has smaller bias, smaller m.a.d. and smaller m.s.e. than its competitors, at least, on the average. Table 1 shows that the confidence intervals for j = 1,2 and 3 are slightly more reliable than those based on the m.l. estimator (j=O). Sample sizes should certainly not be smaller than 50 (25) if one requires that the true confidence coefficient of the interval based on the m. L estimator and I-a = • 9 5, should not be smaller than . 90 (. 85) . ACKNOWLEDGEMENTS The motivation for this paper arose from discussions with the anthropol-ogist G.N. van Vark. Several computations were verified by A.G.M. Steerneman, 8 Input parameter val- Averaged confidence coefficients with ues for the clusters. standard deviations for the four procedures. p,k,µ n j = 0 j = 1 j = 2 j = 3 p = 4 50. 1 4 92.0 2.0 9 2. 8 I . 8 93.3 1 . 6 93.0 I • 6 k = 4 50.1 4-25m4 90.8 2.6 9 I . 8 2. 3 92.8 1 . 9 92.3 2.0 µ = (abed) 25.1 4+25m4 9 0. 1 2.9 9 I . 2 3.0 9 2 . 1 2. 7 9 1 . 6 2.4 2 5 . I 4 88.9 3.2 90.6 2 . 7 92.0 2 . 1 9 I . 4 1 . 9 1 5 • 1 4 84.3 4.5 87.2 4 . 1 89.7 3.4 88.4 3.2 p = 4 50. 1 8 92.4 2. 2 9 3. 1 I . 7 93.3 1 . 4 92.9 I • 2 k = 8 50. 18-25m8 92.3 2.2 9 3 . I I . 4 93.4 I . I 92.4 1 . 2 µ = 25.J 8 +25m8 90.0 3.0 9 1 . 8 2. 3 9 2. 2 I . 8 9 I. 7 I . 5 (ab ... gh) 2 5. 1 8 90.5 3.3 9 I . 6 2. 3 9 2. 2 1 . 6 9 1 . 3 1 . 3 1 5 . 1 8 8 7. 7 4.8 89.8 3. 3 90.0 2. 2 89.4 1 • 6 p = 8 50. 1 4 88.7 1 . 6 89.3 1 • 5 90.9 I . 5 90.4 1 . 6 k = 4 50.1 4-25m4 87.0 2.4 8 7. 9 2. 2 90.0 2 . 1 89.4 1 . 9 µ = (abed\ 25.1 4+25m 4 86.3 2.4 87.5 2. 2 89.6 2.0 8 9 . 1 2.0 aaaa) 25. 14 83.4 2.4 85.0 2 . 2 88.2 2. 3 87.2 2. 3 1 5 . 1 4 76.2 4.0 78.7 3.8 84.8 3. 3 8 3. 1 3.3 p = 8 5 0. 1 4 86.4 2. 3 88.0 2 • 1 9 I . 4 2 . 7 9 I. 1 2. 7 k = 4 50.1 4-25m4 83.8 2.5 86.0 2.4 90.4 2. 8 90.5 3 • 1 µ = (abed) 25.1 4+25m 4 84.4 2.6 86.2 2 . 7 90.3 3.5 89.2 3. 2 abed 25. 14 80.2 2 . 7 83.3 2.8 89.0 4. 4 88.6 4.6 I 5 . I 4 73.3 3. 8 7 7. 5 4. 3 86.5 6. 3 85,6 6.7 p = 8 5 0. 1 8 89.4 2. 2 90.0 1 • 8 90.8 I . 4 90.3 I . 4 k = 8 50.1 8-25m8 8 9 . I 2 • 1 90.0 2 . I 90.9 I • 6 89.4 1 • 7 µ = 25.I 8 +25m8 85.9 3 . 1 86.7 2.6 8 7. 9 I . 9 88.2 1 • 7 (ab • •• gh) 2 5. 1 8 85.0 4. 2 86.6 3.2 88.0 2. 3 87.3 2.2 \aa ... aa I 5 . 1 8 79.2 5. 2 82.0 3.6 84.8 2. 8 83.5 2.6 Table I. The reliability of the confidence intervals. [1] AITCHISON, J., J.D.F. HABBEMA & J.W. KAY (1977), A critical comparison of two methods of statistical discrimination. Appl. Statist., 26, no. 1, pp. 15-25. [2] AMBERGEN, A.W. & W. SCHAAFSMA (1983), Interval estimates for posterior probabilities, applications to Border Cave. In Multivariate Statistical Methods, (G.N. van Vark and W.W. Howells, Ed.). Reidel, Dordrecht, The Netherlands. 9 [3] GEISSER, S. (1964), Posterior odds for Multivariate Normal Classification. J. Roy. Statist. Soc. Ser. B 26, pp. 69-76. [4] McLACHLAN, G.J. (1977), The bias of sample based posterior probabilities. Biom. J. vol. 19, no. 6, pp. 421-426. [SJ McLACHLAN, G.J. (1979), A comparison of the estimative and predictive methods of estimating posterior probabilities. Connn. Statist. A-Theory Methods, 8(9), pp. 919-929. [6] RIGBY, R.A. (1982), A credibility interval for the probability that a new observation belongs to one of two multivariate normal populations. J.R. Statist. Soc. B, 44, no. 2, pp. 212-220. [7] SCHAAFSMA, W. & G.N. VAN VARK (1977), Classification and discrimination problems with applications, part I, Statist. Neerlandica, 31, pp. 25-45. [8] SCHAAFSMA, W. & G.N. VAN VARK (1979), Classification and discrimination problems with applications, part Ila. Statist. Neerlandica, 33, pp. 91-126. [9] VAN VARK, G.N. & W.W. HOWELLS (1983), Multivariate Statistical Methods in Physical Anthropology. Reidel, Dordrecht, The Netherlands. ONTVAN0£N 1 5 SEP. 1983 

Interval estimates for posterior probabilities in a multivariate normal classification model

Abstract

Similar works

Full text

Available Versions

CWI's Institutional Repository