Robust estimation in structural equation models using Bregman divergences

Structural equation models seek to find causal relationships between latent variables by analysing the mean and the covariance matrix of some observable indicators of the latent variables. Under a multivariate normality assumption on the distribution of the latent variables and of the errors, maximum likelihood estimators are asymptotically efficient. The estimators are significantly influenced by violation of the normality assumption and hence there is a need to robustify the inference procedures. We propose to minimise the Bregman divergence or its variant, the total Bregman divergence, between a robust estimator of the covariance matrix and the model covariance matrix, with respect to the parameters of interest. Our approach to robustification is different from the standard approaches in that we propose to achieve the robustification on two levels: firstly, choosing a robust estimator of the covariance matrix; and secondly, using a robust divergence measure between the model covariance matrix and its robust estimator. We focus on the (total) von Neumann divergence, a particular Bregman divergence, to estimate the parameters of the structural equation model. Our approach is tested in a simulation study and shows significant advantages in estimating the model parameters in contaminated data sets and seems to perform better than other well known robust inference approaches in structural equation models. References E. G. Baranoff, S. Papadopoulos and T. W. Seger. Capital and Risk Revisited: A Structural Equation Model Approach for Life Insurers. Journal of Risk and Insurance, 74:653–681, 2007. doi:10.1111/j.1539-6975.2007.00229.x K. Bollen. Structural Equations with Latent Variables. Wiley, New York, 1989. I. S. Dhillon and J. A. Tropp. Matrix Nearness Problems with Bregman Divergences. SIAM Journal on Matrix Analysis and Applications, 29:1120–1146, 2008. doi:10.1137/060649021 F. Nielsen and S. Boltz. The Burbea–Rao and Bhattacharyya Centroids. IEEE transactions in information theory, 57:5455–5466, 2011. doi:10.1109/TIT.2011.2159046 B. C. Vemuri, M. Liu, S.-I. Amari and F. Nielsen. Total Bregman Divergence and Its Applications to DTI Analysis. IEEE Transactions on medical imaging, 30:475–483, 2011. doi:10.1109/TMI.2010.2086464 S. Verboven and M. Hubert. LIBRA: a MATLAB library for robust analysis, Chemometrics and intelligent laboratory systems, 75:127–136, 2005. doi:10.1016/j.chemolab.2004.06.003 K.-H. Yuan and P. M. Bentler. Structural equation modeling with robust covariances. Sociological Methodology, 28:363–396, 1998. doi:10.1111/0081-1750.00052 K.-H. Yuan and P. M. Bentler. Robust mean and covariance structure analysis, British Journal of Mathematical and Statistical Psychology, 51:63–88, 1998. doi:10.1111/j.2044-8317.1998.tb00667.x K.-H. Yuan, P. M. Bentler and W. Chan. Structural Equation Modeling with Heavy Tailed Distributions, Psychometrika, 69:421–436, 2004. doi:10.1007/BF02295644 X. Zhong and K.-H. Yuan. Bias and Efficiency in Structural Equation Modeling: Maximum Likelihood Versus Robust Methods. Multivariate Behavioral Research, 46:229–265, 2011. doi:10.1080/00273171.2011.558736 LIBRA: a Matlab Library for Robust Analysis. http://wis.kuleuven.be/stat/robust/LIBRA/LIBRA-hom

Total Jensen divergences: Definition, Properties and k-Means++ Clustering

We present a novel class of divergences induced by a smooth convex function called total Jensen divergences. Those total Jensen divergences are invariant by construction to rotations, a feature yielding regularization of ordinary Jensen divergences by a conformal factor. We analyze the relationships between this novel class of total Jensen divergences and the recently introduced total Bregman divergences. We then proceed by defining the total Jensen centroids as average distortion minimizers, and study their robustness performance to outliers. Finally, we prove that the k-means++ initialization that bypasses explicit centroid computations is good enough in practice to guarantee probabilistically a constant approximation factor to the optimal k-means clustering.Comment: 27 page

ACQUIRE: an inexact iteratively reweighted norm approach for TV-based Poisson image restoration

We propose a method, called ACQUIRE, for the solution of constrained optimization problems modeling the restoration of images corrupted by Poisson noise. The objective function is the sum of a generalized Kullback-Leibler divergence term and a TV regularizer, subject to nonnegativity and possibly other constraints, such as flux conservation. ACQUIRE is a line-search method that considers a smoothed version of TV, based on a Huber-like function, and computes the search directions by minimizing quadratic approximations of the problem, built by exploiting some second-order information. A classical second-order Taylor approximation is used for the Kullback-Leibler term and an iteratively reweighted norm approach for the smoothed TV term. We prove that the sequence generated by the method has a subsequence converging to a minimizer of the smoothed problem and any limit point is a minimizer. Furthermore, if the problem is strictly convex, the whole sequence is convergent. We note that convergence is achieved without requiring the exact minimization of the quadratic subproblems; low accuracy in this minimization can be used in practice, as shown by numerical results. Experiments on reference test problems show that our method is competitive with well-established methods for TV-based Poisson image restoration, in terms of both computational efficiency and image quality.Comment: 37 pages, 13 figure

Proximity Operators of Discrete Information Divergences

Information divergences allow one to assess how close two distributions are from each other. Among the large panel of available measures, a special attention has been paid to convex $\varphi$-divergences, such as Kullback-Leibler, Jeffreys-Kullback, Hellinger, Chi-Square, Renyi, and I$_{\alpha}$ divergences. While $\varphi$-divergences have been extensively studied in convex analysis, their use in optimization problems often remains challenging. In this regard, one of the main shortcomings of existing methods is that the minimization of $\varphi$-divergences is usually performed with respect to one of their arguments, possibly within alternating optimization techniques. In this paper, we overcome this limitation by deriving new closed-form expressions for the proximity operator of such two-variable functions. This makes it possible to employ standard proximal methods for efficiently solving a wide range of convex optimization problems involving $\varphi$-divergences. In addition, we show that these proximity operators are useful to compute the epigraphical projection of several functions of practical interest. The proposed proximal tools are numerically validated in the context of optimal query execution within database management systems, where the problem of selectivity estimation plays a central role. Experiments are carried out on small to large scale scenarios