An attempt to analyse Iterative Data Snooping and L1-norm based on Monte Carlo simulation in the context of leveling networks

[EN] The goal of this paper is to evaluate the outlier identification performance of iterative Data Snooping (IDS) and L-1-norm in levelling networks by considering the redundancy of the network, number and size of the outliers. For this purpose, several Monte-Carlo experiments were conducted into three different levelling networks configurations. In addition, a new way to compare the results of IDS based on Least Squares (LS) residuals and robust estimators such as the L-1-norm has also been developed and presented. From the perspective of analysis only according to the success rate, it is shown that L-1-norm performs better than IDS for the case of networks with low redundancy ((r) over bar < 0.5), especially for cases where more than one outlier is present in the dataset. In the relationship between false positive rate and outlier identification success rate, however, IDS performs better than L-1-norm, independently of the levelling network configuration, number and size of outliers.Klein, I.; Suraci, SS.; De Oliveira, LC.; Rofatto, VF.; Matsuoka, MT.; Baselga Moreno, S. (2022). An attempt to analyse Iterative Data Snooping and L1-norm based on Monte Carlo simulation in the context of leveling networks. Survey Review. 54(382):70-78. https://doi.org/10.1080/00396265.2021.187833870785438

Performance comparison of least squares, iterative and global L1 Norm minimization and exhaustive search methods for outlier detection in leveling networks

[EN] Different approaches have been proposed to determine the possible outliers existing in a dataset. The most widely used consists in the application of the data snooping test over the least squares adjustment results. This strategy is very likely to succeed for the case of zero or one outliers but, contrary to what is often assumed, the same is not valid for the multiple outlier case, even in its iterative application scheme. Robust estimation, computed by iteratively reweighted least squares or a global optimization method, is other alternative approach which often produces good results in the presence of outliers, as is the case of exhaustive search methods that explore elimination of every possible set of observations. General statements, having universal validity, about the best way to compute a geodetic network with multiple outliers are impossible to be given due to the many different factors involved (type of network, number and size of possible errors, available computational force, etc.). However, we see in this paper that some conclusions can be drawn for the case of a leveling network, which has a certain geometrical simplicity compared with planimetric or three-dimensional networks though a usually high number of unknowns and relatively low redundancy. Among other results, we experience the occasional failure in the iterative application of the data snooping test, the relatively successful results obtained by both methods computing the robust estimator, which perform equivalently in this case, and the successful application of the exhaustive search method, for different cases that become increasingly intractable as the number of outliers approaches half the number of degrees of freedom of the network.Baselga Moreno, S.; Klein, I.; Suraci, SS.; Castro De Oliveira, L.; Matsuoka, MT.; Rofatto, VF. (2020). Performance comparison of least squares, iterative and global L1 Norm minimization and exhaustive search methods for outlier detection in leveling networks. Acta Geodynamica et Geomaterialia. 17(4):425-438. https://doi.org/10.13168/AGG.2020.003142543817

Global Optimization of Redescending Robust Estimators

[EN] Robust estimation has proved to be a valuable alternative to the least squares estimator for the cases where the dataset is contaminated with outliers. Many robust estimators have been designed to be minimally affected by the outlying observations and produce a good fit for the majority of the data. Among them, the redescending estimators have demonstrated the best estimation capabilities. It is little known, however, that the success of a robust estimation method depends not only on the robust estimator used but also on the way the estimator is computed. In the present paper, we show that for complicated cases, the predominant method of computing the robust estimator by means of an iteratively reweighted least squares scheme may result in a local optimum of significantly lower quality than the global optimum attainable by means of a global optimization method. Further, the sequential use of the proposed global robust estimation proves to successfully solve the problem of M-split estimation, that is, the determination of parameters of different functional models implicit in the data.Baselga Moreno, S.; Klein, I.; Sampaio Suraci, S.; Castro De Oliveira, L.; Tomio Matsuoka, M.; Francisco Rofatto, V. (2021). Global Optimization of Redescending Robust Estimators. Mathematical Problems in Engineering. 2021:1-13. https://doi.org/10.1155/2021/9929892S113202

Monte Carlo-Based Covariance Matrix of Residuals and Critical Values in Minimum L1-Norm

[EN] Robust estimators are often lacking a closed-form expression for the computation of their residual covariance matrix. In fact, it is also a prerequisite to obtain critical values for normalized residuals. We present an approach based on Monte Carlo simulation to compute the residual covariance matrix and critical values for robust estimators. Although initially designed for robust estimators, the new approach can be extended for other adjustment procedures. In this sense, the proposal was applied to both well-known minimum L1-norm and least squares into three different leveling network geometries. The results show that (1) the covariance matrix of residuals changes along with the estimator; (2) critical values for minimum L1-norm based on a false positive rate cannot be derived from well-known test distributions; (3) in contrast to critical values for extreme normalized residuals in least squares, critical values for minimum L1-norm do not necessarily tend to be higher as network redundancy increases.This work was supported by the Department of Science and Technology of the Brazilian Army. The authors would like to thank the research group "Controle de Qualidade e Inteligencia Computacional em Geodesia" (dgp.cnpq.br/dgp/espelhogrupo/0178611310347329).Suraci, SS.; Castro De Oliveira, L.; Klein, I.; Rofatto, VF.; Matsuoka, MT.; Baselga Moreno, S. (2021). Monte Carlo-Based Covariance Matrix of Residuals and Critical Values in Minimum L1-Norm. Mathematical Problems in Engineering. 2021:1-9. https://doi.org/10.1155/2021/812349319202