2002 Mathematics Subject Classification: 62F35, 62F15.The concept for trimming and weighing the terms in the Method of Maximum Likelihood gives us a very flexible and useful way to improve the robustness of MLE. Till now the studies were focused mainly on the trimming factor. The theory of d-fullness gives us a powerful method to determine this property in the cases of WLTE, MLE and LTE.
The aim of this study is to consider the weights of the WLTE estimators and to compare the results obtained by using different algorithms for calculating weights.Partially supported by Mathind IST-2000-26015 and by Pro-enbis GTC1-2001-43031

Atanasov, Dimitar

Bulgarian Digital Mathematics Library at IMI-BAS

Pliska Stud. Math. Bulgar. 14 (2003), 5–13STUDIA MATHEMATICABULGARICAABOUT THE CONCEPT OF WEIGHTS OF WLTE(K)ESTIMATORSDimitar AtanasovThe concept for trimming and weighing the terms in the Method of Max-imum Likelihood gives us a very flexible and useful way to improve the ro-bustness of MLE. Till now the studies were focused mainly on the trimmingfactor. The theory of d-fullness gives us a powerful method to determinethis property in the cases of WLTE, MLE and LTE.The aim of this study is to consider the weights of the WLTE estima-tors and to compare the results obtained by using different algorithms forcalculating weights.1. IntroductionAfter the introduction to WLTE(k) (Vandev & Neykov, 1993) a lot of workswere focused on the properties of these estimators. The aim of these workswas to study the breakdown properties and the trimming factor. The obtainedresults are similar to the results obtained for LTE(k) (Neykov and Neytchev,1990). Here we consider the properties of the second factor included in WeightedLeast Trimmed Estimators - the weights.Let x1, · · · , xn be a sample of n iid observations with a probability densityfunction ϕ (xi, θ), where θ is an unknown vector parameter. The WLTE(k)estimators of θ are defined asWLTE(k) = argminθ∈Θpk∑i=1wifν(i) (θ) ,(1)2002 Mathematics Subject Classification: 62F35, 62F15Key words: Robust statistics, WLT Estimators, Bayes EstimatorPartially supported by Mathind IST-2000-26015 and by Pro-enbis GTC1-2001-43031.6 Dimitar Atanasovwhere fν(1) (θ) ≤ fν(2) (θ) ≤ · · · ≤ fν(n) (θ) are the ordered values of fi (θ) =− log ϕ (xi, θ) at θ, and ν = (ν(1), · · · , ν(n)) is the corresponding permutation ofthe indexes, which may depend on θ. The weights wi ≥ 0, i = 1, · · · , k are suchthat there exists an index k = max {i : wi > 0}.These estimators were introduced independently by Hadi and Lucen˜o (1997),and Vandev and Neykov (1998), as an extension of the Maximum LikelihoodEstimators (MLE).Vandev and Neykov (1998) proved that the finite sample breakdown point ofthe WLTE(k) estimators is not less than (n− k)/n if the set F = {fi(θ), i =1, · · · , n} is d-full, n ≥ 3d and (n + d)/2 ≤ k ≤ n−d. We remind the reader that,according to Vandev (1993), a finite set F of n functions is called d-full if foreach subset of cardinality d of F , the supremum of this subset is a subcompactfunction. A real valued function g (θ) is called subcompact, if its Lesbegue setsLg(θ) (C) = {θ : g (θ) ≤ C} are compact for any constant C (see Vandev andNeykov, 1993).We remind the reader that the finite sample BP of an estimator T , at thefinite sample X = {xi; i = 1, · · · , n}, is defined as the largest fraction m/n forwhich the supX˜∥∥∥T (X)− T (X˜)∥∥∥ is finite, where X˜ is a sample obtained from Xwith replacing any m of the points in X by arbitrary values (see Hampel et al.1986, Rousseeuw and Leroy 1987).Vandev and Neykov (1993) determined the value d for the set of log likelihoodsfor the multivariate normal case while Vandev and Neykov (1998) did the samefor different regression models, including the grouped binary logistic regression,and showed that the corresponding sets of log likelihoods are (p + 1)-full.Another very powerful property of the estimators are weights. Using themone can obtain different types of estimators. If all the weights are equal to 1we have LTE(k). The weights can also be proportional to the distribution inthe estimated point, or in general proportional to something which depends onthe distribution as errors. We can obtain different weights according to the usedalgorithm. For example, here the case when the weights are proportional to thenumber of times in which a given observation enters into the model is considered.There is another interesting point of view to the concept of weights. They canbe used not only to find the outliers in the data, but also to make a distinctionbetween two sets of observations. For example, if there is some kind of discreteheterogeneity in the data we can differentiate the cases according to the valuesof the weights.About the Concept of Weights of WLTE(k) Estimators 72. Analysis of different types of weightsNow we will consider the behaviour of WLTE(k). To do this we will use anumerical example. The set X consists of 200 observations in two subsets X1and X2, which are generated in different ways. So we can study the conduct ofthe estimator changing the proportion of these sets. We will suppose that theobservations have a normal distribution and the distribution of the observations inthe set X2 is N(0, 1). We will change the variance of the observations in set X1 butkeep the mean equal to 3. All the estimations were done using one minimizationprocedure with appropriate parameters for different models. The optimizationalgorithm is based on Golden Section search and parabolic interpolation.First we will consider the case when all the weights are equal to 1. As it wasmentioned above, in this case the WLTE(k) estimator is equivalent to LTE(k).The used algorithm can be summarized as follows:1. Setting the initial value for the unknown parameter2. Sorting the observations according to the log-density function at the cur-rent value of the unknown parameter3. The weights are equal to 14. Finding the value which satisfies (1)5. If the exit conditions are not satisfied than go back to 2In Table 1 some of the obtained results are presented. They are compared tothe results from the Maximum Likelihood Estimation for the set X1.m k DX1 MLE St. err MLE Result St. err Iter10 100 1 2.81 0.13 2.73 0.15 610 150 1 2.87 0.10 2.88 0.11 410 190 1 3.1 0.087 3.13 0.098 410 100 2 2.93 0.20 2.90 0.25 510 150 2 2.90 0.17 2.29 0.19 410 190 2 3.02 0.16 3.01 0.19 390 100 1 2.93 0.21 2.45 0.32 390 110 1 2.87 0.19 2.43 0.32 390 100 2 2.97 0.19 2.31 0.37 350 100 1 3.11 0.21 3.02 0.27 450 100 2 3.09 0.23 3.01 0.35 350 150 1 2.97 0.18 3.10 0.21 450 150 2 3.01 0.23 2.96 0.29 5Table 18 Dimitar AtanasovWith m we denote the number of observations in the set X2. The first sixrows of the table show the behaviour of the estimator if there are some outliersin the data (10 observations). It is seen that with the increasing of the trimmingfactor k the error of the result decreases as in the case of MLE (Figure 1.).Also the error increases whit the increasing of the variance of the observations.The standard error of the Maximum Likelihood Estimator is less than the sameof LTE due to the existence of the trimming factor and using the smaller numberof observations.This type of estimator can be used to distinct between two sets of observa-tions. This is shown in the next three rows of the table. Just about 15% of theobservations are misplaced in their groups.In the last three rows is shown that when the cardinality of set X2 is smaller,the estimations are better than in the previous case.Next we will consider the case when the weights are proportional to thenumber of iterations in which given observation enters into the model. Theoptimization algorithm differs from the previous one only in step 3, when theweights are calculated. At each iteration step the algorithm remembers which ofthe observations are in the model and at the next step they have bigger weights.We keep the sum of the weights equal to 1. We thought that this will allow us touse the weights in a very flexible way, because they are not directly dependenton the probability distribution. But in fact the weights are dependent on thedistribution, because they are calculated using the probability density functionto order the observations and to choose which of them to enter into the model.The results, obtained using such a model, are in Table 2.m k DX1 MLE St. err MLE Result St. err Iter50 100 2 2.93 0.27 2.34 0.96 1550 100 1 2.96 0.24 2.57 0.88 450 150 1 2.93 0.21 2.96 0.73 1650 150 2 3.01 0.26 2.51 0.93 610 100 2 2.94 0.19 2.56 0.82 810 150 2 2.90 0.16 2.65 0.76 610 190 2 3.02 0.14 2.84 0.63 710 100 1 3.01 0.17 2.57 0.79 1510 150 1 3.05 0.15 2.97 0.78 2110 190 1 3.12 0.13 3.12 0.71 1090 100 2 3.13 0.31 2.65 1.02 2190 100 1 3.07 0.25 2.71 0.93 7Table 2About the Concept of Weights of WLTE(k) Estimators 9Here it is seen that the standard error of the estimator is bigger than inprevious case. Also the number of iterations is bigger than in the case of LTEand the value of the unknown parameter is not as close to the true value as thevalues estimated with MLE and LTE.The first four rows in Table 2 show that the standard error of the estimatorincreases with the increasing of the variance of the set X1 and decreases with theincreasing of the trimming factor k.When there are just a few outliers in the set (rows 5 - 10) the obtained resultsare not better. The standard error is big and the estimation is not good enough.When the number of outliers is near the 50% of the observations the estima-tion is very bad.We suppose that these results are due to the smooth likelihood function inthis case. Adding the weights we smooth out the likelihood, so the number ofiterations and the standard error are larger.The last case studied here is the case when the weights are proportional tothe value of the probability density function for the observation at the currentvalue of the estimated parameter. The obtained results are similar to the resultsin the case when the weights are proportional to the number of times in whichthey are in the model. The results are presented in Table 3. If there are just afew outliers in the data the results are close to the results from MLE.m k DX1 MLE St. err MLE Result St. err Iter50 100 2 2.95 0.26 2.22 0.91 250 150 2 2.93 0.22 2.15 0.88 250 100 1 2.98 0.21 2.95 0.77 650 150 1 2.99 0.19 2.98 0.59 510 100 1 3.10 0.18 2.92 0.33 810 150 1 2.97 0.16 2.92 0.24 610 190 1 3.05 0.14 3.05 0.22 790 100 1 3.10 0.41 2.28 0.98 2190 100 2 2.91 0.55 2.04 1.13 7Table 3We do believe that we will have similar results if the weights depended insome way on the probability density function. So we can assume that addingthe weights to the log-likelihood function makes the function more smooth andit is more difficult to have a fast and good optimization procedure. As a resultof that the standard error will be bigger than the standard error of the estimator10 Dimitar Atanasovobtained using MLE or LTE.3. Bayesian Modification of the LTEIn fact, the values of the likelihood function for a given observation weigh this ob-servation in the likelihood. So it is not necessary to add other weights, especiallyif the weights depend on the distribution of the observations.To improve the performance of the LTE estimator we can use the concept ofBayesian estimation of the unknown parameter.Let us suppose that the density of a priori distribution of the unknown pa-rameter is h(θ) and the probability density function of the observations is φ(x, θ).Then the a posteriori distribution of the parameter ish(θ | x) =φ(x, θ)h(θ)φ(x),whereφ(x) =∞∫−∞φ(x, θ)h(θ)dθ.Using this notation, we can consider the estimatorθˆ = argminθ∈Θpk∑i=1− log h(θ | xν(i)),(2)where xν(i), i = 1, · · · , n are the observations, ordered according to the value of− log h(θ | x), − log h(θ | xν(1)) ≤ − log h(θ | xν(2)) ≤ · · · ≤ − log h(θ | xν(n)).Now let us consider the breakdown properties of the proposed estimator.Using the definition (2) we haveθˆ = argminθ∈Θpk∑i=1− logφ(xν(i), θ)h(θ)φ(xν(i))== argminθ∈Θpk∑i=1(− log φ(xν(1), θ)− log h(θ) + log φ(xν(i))).Let us note the functiongi(θ) = − log φ(xν(1), θ)− log h(θ) + log φ(xν(i)).About the Concept of Weights of WLTE(k) Estimators 11According to Vandev (1993) we need to study the index of fullness for the set offunctions F = {gi(θ)}. As log φ(xν(i)) is a constant, using the unequality− log φ(xν(1), θ)− log h(θ) ≥ − log h(θ)we have that gi(θ) is subcompact function if and only if − log h(θ) is a subcompactone. We can use the following theorem:Theorem 1. (Atanasov & Neykov, 2001) Let D be an open subset of Rn,θ0 belong to the boundary of D and g (θ) be a real valued continuous functiondefined on D. Then g (θ) is subcompact if and only if for any sequence θi → θ0g (θi) →∞ when i →∞.Corollary 1. If h(θ) is a continuous probability density function then ac-cording to the Theorem the function − log h(θ) will be a subcompact one. So theindex of fullness of the set F is d = 1.Therefore, the breakdown point of the WLTE(k) for these models is equal to(n−k)nwhen (n−1)2 ≤ k ≤ n−1, according to Vandev and Neykov (1998) and someadditional arguments given there. This gives us a very flexible way to managesuch an estimator, because we can choose the minimal possible value for k.For example let us consider the case when the observations have a normaldistribution N(θ, 1) and the distribution of the unknown parameter is also normalN(0, τ2), for an appropriate value for τ . Under these conditions we haveh(θ | x) =φ(x, θ)h(θ)φ(x)==e−(x−θ)22− θ22τ2∞∫∞e−(x−θ)22− θ22τ2 dθ=e−(x−θ)22n− θ22τ2√2piτ2e−x22(τ2+1)(τ2−1)==1√2pi(1 + 1τ2)e−(θ− x1+ 1τ2)2(1+ 1τ2)−1.So the a posteriori distribution of the unknown parameter is N(x1+ 1τ2,(1 + 1τ2)−1)and we can use the ordinary LTE estimator with a calculated a posteriori prob-ability density function.The obtained results are given in Table 4.12 Dimitar Atanasovm k DX1 MLE St. err MLE Result St. err Iter90 100 2 2.88 0.36 2.71 0.38 290 110 2 2.83 0.32 2.65 0.33 290 100 1 3.05 0.28 2.95 0.29 390 110 1 2.99 0.26 2.98 0.26 550 100 2 2.91 0.19 2.92 0.21 550 150 2 2.97 0.15 2.94 0.19 350 100 1 3.02 0.14 3.05 0.18 550 150 1 2.95 0.12 3.01 0.16 310 150 1 3.04 0.21 2.99 0.26 310 190 1 2.96 0.18 2.98 0.23 310 190 2 2.87 0.22 2.91 0.24 2Table 4The first four rows show the case when the observations consist of two subsetswith almost equal sizes. The estimation of the parameter is close to the maximumlikelihood estimation. When the outliers are not so much the estimation is betterand the standard error is smaller.It is seen that using this estimator we have a very small number of iterationsneeded to obtain a result. Also we can say that the procedure is strongly de-pendent on the starting point, so we need some kind of a priori information forit.This estimator is more flexible than LTE and gives better results than WLTE(k)models, considered above.REFERE NCES[1] D. V. Atanasov, N. M. Neykov. About the Finite Sample BreakdownPoint of the WLTE(k) Estimators. In: Proceedings of the XXV SummerSchool Sozopol’99, (eds. Cheshankov, B.I., Todorov, M.D.) (2000), 105–106.[2] D. V. Atanasov, N. M. Neykov. On the Finite Sample BreakdownPoint of the WLTE(k) and d-fullness of a Set of Continuous Functions. In:Proceedings of the VI International Conference ”Computer Data AnalysisAnd Modeling”, Minsk, Belarus. (2001), 1–2.About the Concept of Weights of WLTE(k) Estimators 13[3] A. Hadi, A. Lucen˜o. Maximum Trimmed Likelihood Estimators: AUnified Approach, Examples and Algorithms. Comput. Statist. and DataAnalysis 25 (1997), 251–272.[4] F. R. Hampel, E. M. Ronchetti, P. J. Rousseeuw, W. A. Stahel.Robust Statistics: The Approach Based on Influence Functions. JohnWiley and Sons, New York, 1986.[5] N. M. Neykov, P. N. Neytchev. A Robust Alternative of the Maxi-mum Likelihood Estimators. COMPSTAT 1990 (1990), 99–100.[6] P. J. Rousseeuw, A. Leroy. Robust Regression and Outlier Detection.John Wiley and Sons, New York, 1987.[7] D. L. Vandev. A Note on Breakdown Point of the Least Median ofSquares and Least Trimmed Estimators. Statistics and Probability Letters16 (1993), 117–119.[8] D. L. Vandev, N. M. Neykov. Robust Maximum Likelihood in theGaussian Case. In: New Directions in Statistical Data Analysis and Ro-bustness. (eds. S. Morgenthaler, E. Ronchetti, W.A. Stahel), BirkhauserVerlag, Basel, 1993.[9] D. L. Vandev, N. M. Neykov. About Regression Estimators with HighBreakdown Point. Statistics 32 (1998), 111–129.Faculty of Mathematics and InformaticsSofia University5 J. Boucher Str.14071113 Sofia, Bulgariae-mail: datanasov@fmi.uni-sofia.bg

About the Concept of Weights of WLTE(k) Estimators

http://sci-gems.math.bas.bg/jspui/bitstream/10525/2175/1/Pliska-14-2003-005-013.pdf

About the Concept of Weights of WLTE(k) Estimators

Abstract

Similar works

Full text

Available Versions

Bulgarian Digital Mathematics Library at IMI-BAS