The kernel multivariate density estimation is an important technique to estimate the multivariate density function. In this investigation we will use Hellinger Distance as a measure of error to evaluate the estimator, we will derive the mean weighted Hellinger distance for the estimator, and we obtain the optimal bandwidth based on Hellinger distance. Also, we propose and study a new technique to select the matrix of bandwidths based on Hellinger distance, and compare the new technique with the plug-in and the least squares techniques

Anver, Haneef

Mugdadi, A. R.

English

Stellenbosch University: SUNJournals

South African Statist. J. (2016) 50, 221 – 236 221THE WEIGHTED HELLINGER DISTANCE INTHE MULTIVARIATE KERNEL DENSITYESTIMATIONA. R. Mugdadi 1Jordan University of Science and Technology, Jordane-mail: aamugdadi@just.edu.joHaneef AnverSouthern Illinois University, Carbondale, USAKey words: Bandwidth Selection, Hellinger distance, Least squares, Multivariate kernel estima-tion, Plug-inAbstract: The kernel multivariate density estimation is an important technique to estimate the mul-tivariate density function. In this investigation we will use Hellinger Distance as a measure of errorto evaluate the estimator, we will derive the mean weighted Hellinger distance for the estimator,and we obtain the optimal bandwidth based on Hellinger distance. Also, we propose and study anew technique to select the matrix of bandwidths based on Hellinger distance, and compare the newtechnique with the plug-in and the least squares techniques.1. IntroductionThe univariate kernel density estimators for a random sample X1, X2, . . . , Xn, drawn from a proba-bility density function f , isf̂ (x,h) = n−1n∑i=1kh(x−Xi),where k is chosen to be a unimodal probability density function that is symmetric about zero.The general form of the d-variate kernel density estimator, for a random sample X1, X2, . . . , Xndrawn from a density f , isf̂ (x,H) = n−1n∑i=1kH(x−Xi),where x = (x1,x1, . . . ,xd)T and Xi = (Xi1,Xi2, . . . ,Xin)T ,i = 1,2, . . . ,n. Here k is the unscaled kernel, kH is the scaled kernel and H is thed × d (fixed) bandwidth matrix, which is non-random, symmetric and positive definite. The scaledand unscaled kernels are related by kH(x) = |H|−1/2k(H−1/2x). This formulation is a little differentfrom the univariate case since the 1 × 1 bandwidth matrix is H = h2 so we are dealing with squaredbandwidths here.1 Corresponding author.AMS: 62G05, 62G07, 62H12222 MUGDADI & ANVERWe restrict our attention to kernel functions k that are spherically symmetric probability densityfunctions. Moreover, we mostly use normal kernels throughout this paper for two reasons: they leadto smooth density estimates and they simplify the mathematical analysis. The bandwidth selectorplays a central role in determining the performance of kernel density estimators. Thus we wishto select bandwidths which give the optimal performance which is measured by the closeness ofa kernel density estimate to its target density. There are many possible error criteria from whichto choose. In this investigation we will concentrate on the mean weighted Hellinger distance as ameasure of error. Also, we propose a new technique to select the matrix of bandwidths. A detaileddiscussion about the multivariate density estimation and the bandwidth selection techniques can befound in Anver (2010), Scott (1992) and Wand and Jones (1995).2. The asymptotic mean weighted Hellinger distanceThe common global error criterion is the mean integrated squared error (MISE), which is definedby:MISE{f̂ (x;H)}= E{ISE f̂ (.;H)}= E∫ [f̂ (x;H)− f (x)]2dx.Another technique to measure the error is called the mean weighted Hellinger distance, which isdefined for the univariate case by:MWHD{( f̂ (x;H)}= E∫ [f̂ 1/2(x)− f 1/2(x)]2f (x)dx. (1)Kanzawa (1995) discussed the relationship between the asymptotic mean Hellinger Distance(AMHD) and the asymptotic mean integrated square error (AMISE) when f (x) defined on a com-pact set. Ahmad and Mugdadi (2006) discussed the relation between the asymptotic mean weightedHellinger distance (AMWHD) and the AMISE for both f̂ (x) and the estimated kernel distributionfunction F̂(x). Mugdadi (2004) used Hellinger distance to derive the bandwidth for the kernel den-sity estimation of function of observations. Thus, the MWHD for the multivariate kernel densityestimation can be defined by:MWHD( f̂ (x;H)) = E∫( f̂ 1/2(X)− f 1/2(X))2 f (X)dX.Theorem 1 Using the assumptions:1. Each entry of H f (.) is piecewise continuous and square integrable.2. H = Hn is a sequence of bandwidth matrices such that n−1|H|−1/2 and all entries of H approachzero as n→ ∞. Also, we assume that the ratio of the largest and smallest eigenvalues of H isbounded for all n.3. k is a bounded and compactly supported d-variate kernel.4. f (4)(x) exists.THE WEIGHTED HELLINGER DISTANCE 223The AMWHD{ f̂ (.;H)} is given by:AMWHD{ f̂ (.;H)}= 14n−1|H|−1/2R(k)+ 116µ2(k)2∫tr2{HH f (x)}dx.Proof.Note thatMWHD( f̂ (.;H)) = E∫( f̂ 1/2− f 1/2)2 f (x)dx=∫E f̂ (x) f (x)dx−2∫E f̂ 1/2 f 3/2dx+∫E f 2(x)dx= I−2II +R( f ).However by the multivariate version of Taylor’s theoremE f̂ (x;H) =∫kH(x−y) f (y)dy=∫k(z) f (x−H1/2z)dz=∫{ f (x)− (H1/2z)T D f (x)+12(H1/2z)T H f (x)(H1/2z)}dz+o{tr(H)}= f (x)−∫zT H1/2D f (x)k(z)dz+12∫zT H1/2H f (x)H1/2zk(z)dz+o{tr(H)}= f (x)+12tr{H1/2H f (x)H1/2∫zzT k(z)dz}+o{tr(H)}= f (x)+12µ2(k)tr{HH f (x)}+o{tr(H)}.ThusI =∫f (x)2dx+12µ2(k)∫tr{HH f (x)} f (x)dx= R( f )+12µ2(k)∫tr{HH f (x)} f (x)dx.Next, from the standard analysis of f̂ (x), we have thatf̂ (x) = f (x)+12µ2(k)tr{HH f (x)}+o{tr(H)}+Z[n−1|H|−1/2R(k) f (x)]1/2f̂ (x)1/2 = f (x)1/2{1+12 f (x)µ2(k)tr{HH f (x)}+o{tr(H)}+Z[n−1|H|−1/2R(k)f (x)]1/2}1/2.Thus,f̂ 1/2(x)∼= f (x)1/2{1+14 f (x)µ2(k)tr{HH f (x)}+o{tr(H)}2+Z2[n−1|H|−1/2R(k)f (x)]1/2}− w28,224 MUGDADI & ANVERwherew2 =[12f (x)µ2(k)tr{HH f (x)}]2+ z2[n−1|H|−1/2R(k)f (x)]+ z1f (x)µ2(k)tr{HH f (x)}[n−1|H|−1/2R(k)f (x)]1/2.ThusE f̂ 1/2(x;H) = f (x)1/2{1+14 f (x)µ2(k)tr{HH f (x)}+12[n−1|H|−1/2R(k)f (x)]1/2− 132 f (x)2µ2(k)2tr2{HH f (x)}−18[n−1|H|−1/2R(k)f (x)]}.2II = 2∫E f̂ 1/2(x) f32 (x)dx+2∫f 2(x)dx+µ2(k)2∫f (x)tr{HH f (x)}dx+∫ [n−1|H|−1/2R(k)f (x)1/2 f 2(x)]dx−∫ 116µ2(k)2tr2{HH f (x)}dx−14∫ [n−1|H|−1/2R(k)f (x)]f 2(x)dx= 2∫f 2(x)dx+µ2(k)2∫f (x)tr{HH f (x)}dx+[n−1|H|−1/2R(k)]1/2 ∫f (x)dx− 116µ2(k)2∫tr2{HH f (x)}dx−14[n−1|H|−1/2R(k)]∫f (x)dx.Therefore I - 2II + R( f )= R( f )+12µ2(k)∫tr{HH f (x)} f (x)dx−2R( f )−µ2(k)2∫tr{HH f (x)} f (x)dx− [n−1|H|−1/2R(k)]−1/2 + 116µ2(k)2∫tr2{HH f (x)}dx+14[n−1|H|−1/2R(k)]+R( f )=18µ2(k)2∫tr2{HH f (x)}dx− [n−1|H|−1/2R(k)]−1/2 +14[n−1|H|−1/2R(k)]=14[n−1|H|−1/2R(k)]+ 116µ2(k)2∫tr2{HH f (x)}dx− [n−1|H|−1/2R(k)]1/2.The theorem is now proved. Under the above conditions and in the case where H = h2I we obtain the following corollary:Corollary 1AMWHD{ f̂ (.;H)}= n−1h−dR(k)4+116h4µ2(k)2∫{∇2 f (x)}2dx,where∇2 f (x) =d∑i=1(∂ 2/∂x2i ) f (x).THE WEIGHTED HELLINGER DISTANCE 225In this case the optimal bandwidth has an explicit formula given by the following corollary.Corollary 2 Under the conditions of Theorem 1 and in the case of H = h2I, the optimal bandwidthis given ashAMWHD =[dR(k)µ2(k)2∫{∇2 f (x)}2dx n]1/(d+4).This can be easily proved by differentiating (1) with respect to h and setting the derivative equalto zero and solving for h.3. Data-dependent bandwidth choicesThe problem of selecting the scalar bandwidth in univariate kernel density estimation is quite wellunderstood. A number of methods exist that combine good theoretical properties with strong prac-tical performance. Many of these techniques can be extended to the multivariate case in a relativelystraight forward fashion if H is constrained to be a diagonal matrix. However, imposing such aconstraint on the bandwidth matrix can result in markedly suboptimal density estimates.The preceding discussion indicates a need for data-dependent methods for choosing full (i.e un-constrained) bandwidth matrices. The development of selectors for full H is rather more challengingthan that for diagonal H. In particular, the need to consider the orientation of kernel functions tothe coordinate axes in the former case introduces a problem without a univariate analogue. The ad-ditional difficulties in selecting full (as opposed to diagonal) bandwidth matrices partly explain therelatively slow progress in this area of the two major approaches to bandwidth selection, namely theplug-in method and cross validation (CV) techniques. Only the former has received attention to datein the context of full bandwidth matrices. Stone (1984) looks at the multivariate least squares crossvalidation (LSCV) criterion for the multivariate kernel density estimator. Wand and Jones (1994)outlined a plug-in selector that can be applied to full bandwidth matrices, but they concentratedon a diagonal H when presenting methodological particulars. A more detailed account of plug-inselectors for full H was provided by Duong and Hazelton (2003).All CV bandwidth matrix selectors aim to estimate MISE, or AMISE, and some combine thetwo (modulo a constant) and then minimize the resulting function. The unbiased cross-validation(UCV) targets MISE and employs the objective functionUCV (H) =∫f̂ (x;H)2dx−2n−1n∑i=1f̂−i(Xi;H),wheref̂−i(xi;H) = (n−1)−1n∑j 6=ikH(x−X j),is a leave-one-out estimator of f . The function UCV (H) is unbiased in the sense that E[UCV (H)]=MISE[ f̂ (.;H)] - R( f ). It can be expanded to giveUCV (H) = n−2n∑i=1n∑j=1(kH ∗kH)(Xi−X j)−n−1(n−1)−1n∑i=1n∑j=1kH(Xi−X j),where * denotes the convolution. The UCV bandwidth matrix selector ĤUCV is the minimiser ofUCV (H). It is usual to judge the performance of a bandwidth matrix H according to a global error226 MUGDADI & ANVERcriterion for f̂ (x). Next we derive the estimate for the bandwidth matrix H using MWHD as theerror criteria and call it WHCV(H).WHCV(H) =E∫ {f̂ 1/2(x;H)− f 1/2(x)}2f (x)dx=E{∫f̂ (x;H) f (x)dx−2∫f̂ 1/2(x;H) f 3/2(x)dx+R( f )}=E{∫f̂ (x;H) f (x)dx−2n∑i=1f̂ 1/2(xi;H) f̂1/2i (xi)∫f (x)dx}+R( f ).Thus the WHC(H) can be estimated byˆWHCV (H) =n∑i=1f̂ (xi)n− 2nn∑i=1f̂ 1/2i (xi;H) f̂1/2(xi;H)+R( f ).The WHCV bandwidth matrix selector ĤWHCV is the minimiser of ˆWHCV (H). Minimising ofsuch function can be done using the Powell Optimization method — an efficient method for findingthe minimum of a function of several variables without calculating derivatives (Powell, 1964).3.1. Comparing the performance of the WHCV matrix selector with otherbandwidth selectors for normal mixture densitiesIn order to asses the performance of the new technique to select the matrix of bandwidths we willsimulate from the densities that are displayed in Table 1. These densities represent a wide range ofmultivariate densities as shown in Figure 1.Table 1: Formulas for target densities A – E.Target Density FormulaA N([00],[0.25 00 1])B 12 N([10],[4/9 00 4/9])+1/2N([−10],[4/9 00 4/9])C 1/2N([1−0.9],[1 0.90.9 1])+1/2N([−10.9],[1 0.90.9 1])D 1/2N([1−1],[4/9 14/4514/45 4/9])+1/2N([−11],[4/9 00 4/9])E N([00],[1 9/109/10 1])THE WEIGHTED HELLINGER DISTANCE 227Figure 1: Contour plots for target densities A – E.228 MUGDADI & ANVERBandwidth matrices were selected for 100 random sample generated from the densities in Table1. The integrated square error (ISE) for each resulting density estimate was recorded. They aredisplayed in Figure 2 using box plots with a log scale. The performance of a selector dependslargely on the target density shape. We see that the plug-in bandwidth selector performs well fordensities A, B, D and E. Both CV selectors perform adequately for densities B, D and E, but cannotcompete with the plug-in bandwidth selector on density B and E and to a lesser extent on densityD. For density C one noteworthy aspect is that the performance of WHCV is better than the plug-inbandwidth selector. LSCV suffers in comparison to WHCV for densities A, B and C, and givingsimilar results for D and E.Performance of bandwidth selectors are more visible and provide a variety of interpretations ofthe structure of the data when we plot contour plots. To illustrate this we simulated random samplesfor densities A – E in two cases. The first case considers small sample simulations of size 10 foreach density for two instances and the results are given in Figures 3, 5, 7, 9 and 11. For the secondcase we looked at comparatively large samples of sizes 200 for each density and results are given inFigures 4, 6, 8, 10 and 12. Various bandwidth selectors provide a variety of interpretations of thestructure of the data.Figure 2: Box plots of log(ISE) for bivariate bandwidth selectors.THE WEIGHTED HELLINGER DISTANCE 229Figure 3: Density A: small sample.Figure 4: Density A: large sample.230 MUGDADI & ANVERFigure 5: Density B: small sample.Figure 6: Density B: large sample.THE WEIGHTED HELLINGER DISTANCE 231Figure 7: Density C: small sample.Figure 8: Density C: large sample.232 MUGDADI & ANVERFigure 9: Density D: small sample.Figure 10: Density D: large sample.THE WEIGHTED HELLINGER DISTANCE 233Figure 11: Density E: small sample.Figure 12: Density E: large sample.234 MUGDADI & ANVERFigures 3 to 12 give the kernel density estimates using the bandwidth selectors HPI, LSCV andWHCV. As might be expected, no method is uniformly best for all the densities. Both LSCV andWHCV cannot compete with the performance of HPI for increasing sample sizes. HPI gives a betterestimate for all the densities for large samples. We see that both CV selectors give very poor estimateof density B. It is known that CV techniques do not perform well for duplicated data values, whichcannot be the case for poor density B estimate. For small sample size we can see that for densities A,B and D signs of oversmoothing is displayed with all three bandwidth selectors while for densities Eand C the bimodality is preserved for all three selectors. In the case of density E all three bandwidthselectors give a better estimate for small sample whereas for density A, the performance is poor.Computation times were not taken in to consideration because of computing resources. HoweverHPI running time was much lesser than the other two CV selectors.3.2. Bivariate real-life data analysisWe now turn to the analysis of data on under-5 mortality (per 1000 live births) and average lifeexpectancy (in years) for 73 countries with Gross National Income < $1000 per person per year.The data were obtained from Unicef (United Nations Children’s Fund). A scatter plot of the Unicefdata is given in Figure 13. This dataset has probability mass oriented to the axes. Since the datasetcontains duplicates, LSCV’s estimate was very poor. Therefore we used the smooth cross validationbandwidth selectors (SCV) as our second choice and the density estimates obtained using Plug-in,SCV and WHCV are displayed in Figure 14. The plot for SCV has spurious features and the mostnoisy whereas the plots for HPI and WHCV are smoother, hence giving better estimates.Figure 13: Unicef-data Scatterplot.THE WEIGHTED HELLINGER DISTANCE 235Figure 14: Unicef Plots for Bandwidth Selectors.236 MUGDADI & ANVER4. ConclusionThe selection of full bandwidth matrices for multivariate density estimation raises issues that haveno univariate counterpart. In particular, the orientation of the kernel functions to the coordinate axesmust be determined. Furthermore, intuition drawn from the univariate setting cannot necessarilybe transformed to the multivariate case. The additional difficulties involved in the multivariate casegenerate significant scope for further research in full bandwidth matrix selection. Looking furtherafield, the use of adaptive kernel density estimators has great potential for multivariate data. Whilesome progress has been made, this remains a challenging research direction.ReferencesAHMAD, I. A. AND MUGDADI, A. R. (2006). Weighted Hellinger distance as an error criterion forbandwidth selection in kernel estimation. Journal of Nonparametric Statistics, 18 (2), 215–226.ANVER, H. (2010). Mean Hellinger Distance as an Error Criterion in Univariate and Multivari-ate Kernel Density Estimation. Ph.D. dissertation, Southern Illinois University: Carbondale,U.S.A.DUONG, T. AND HAZELTON, M. (2003). Plug-in bandwidth matrices for bivariate kernel densityestimation. Journal of Nonparametric Statistics, 15 (1), 17–30.KANZAWA, Y. (1995). Hellinger distance and Kullback-Leibler loss for the kernel density estimator.Statistics and Probability Letters, 18, 315–321.MUGDADI, A. R. (2004). Bandwidth selection for the function of observations using Hellingerdistance. Journal of Applied Statistical Science, 13 (3), 231–240.POWELL, M. J. D. (1964). An efficient method for finding the minimum of a function of severalvariables without calculating derivatives. The Computer Journal, 7 (2), 155–162.SCOTT, D. W. (1992). Multivariate Kernel Density Estimation: Theory, Practice and Visualization.John Wiley & Sons Inc.: New York.STONE, C. J. (1984). An asymptotically optimal window selection rule for kernel density estimates.The Annals of Statistics, 12, 1285–1297.WAND, M. P. AND JONES, M. C. (1994). Multivariate plug-in bandwidth selection. ComputationalStatistics, 9, 97–116.WAND, M. P. AND JONES, M. C. (1995). Kernel Smoothing. Chapman and Hall: New York.Manuscript received, 2015-11-23, revised, 2016-01-29, accepted, 2016-05-31.

The weighted Hellinger distance in the multivariate kernel density estimation

https://www.journals.ac.za/index.php/sasj/article/download/5261/3269

The weighted Hellinger distance in the multivariate kernel density estimation

Abstract

Similar works

Full text

Available Versions

Stellenbosch University: SUNJournals