Kondrashov and Ghil (2006) (KG hereafter) describe a method for imputing missing values in incomplete datasets that can exploit both spatial and temporal covariability to estimate missing values from available values. Temporal covariability has not been exploited as widely as spatial covariability in imputing missing values in geophysical datasets, but, as KG show, doing so can improve estimates of missing values. However, there are several inaccuracies in KG’s paper. Since similar inaccuracies have surfaced in other recent papers, for example, in the literature on paleo-climate reconstructions, I would like to point them out here

Schneider, T.

English

International audienceNo abstract available

HAL-INSU

Comment on ”Spatio-temporal filling of missing pointsin geophysical data sets” by D. Kondrashov and M.Ghil, Nonlin. Processes Geophys., 13, 151?159, 2006T. SchneiderTo cite this version:T. Schneider. Comment on ”Spatio-temporal filling of missing points in geophysical data sets”by D. Kondrashov and M. Ghil, Nonlin. Processes Geophys., 13, 151?159, 2006. NonlinearProcesses in Geophysics, European Geosciences Union (EGU), 2007, 14 (1), pp.1-2. <hal-00300066>HAL Id: hal-00300066https://hal.archives-ouvertes.fr/hal-00300066Submitted on 15 Jan 2007HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.Nonlin. Processes Geophys., 14, 1–2, 2007www.nonlin-processes-geophys.net/14/1/2007/© Author(s) 2007. This work is licensedunder a Creative Commons License.Nonlinear Processesin GeophysicsComment on “Spatio-temporal filling of missing points ingeophysical data sets” by D. Kondrashov and M. Ghil, Nonlin.Processes Geophys., 13, 151–159, 2006T. SchneiderCalifornia Institute of Technology, Pasadena, CA, USAReceived: 10 July 2006 – Revised: 1 October 2006 – Accepted: 22 December 2006 – Published: 15 January 2007Kondrashov and Ghil (2006) (KG hereafter) describe amethod for imputing missing values in incomplete datasetsthat can exploit both spatial and temporal covariability to es-timate missing values from available values. Temporal co-variability has not been exploited as widely as spatial covari-ability in imputing missing values in geophysical datasets,but, as KG show, doing so can improve estimates of miss-ing values. However, there are several inaccuracies in KG’spaper. Since similar inaccuracies have surfaced in other re-cent papers, for example, in the literature on paleo-climatereconstructions, I would like to point them out here.(i) In estimating covariance matrices, KG treat an incom-plete dataset with imputed values filled in as if it were acomplete dataset. Possible variations of the missing valuesaround the imputed values are ignored, leading to biasedestimates of covariance matrices (Little and Rubin, 2002).The expectation-maximization (EM) algorithm takes varia-tions of the missing values around the imputed values intoaccount, which is essential to obtain maximum likelihood es-timates of parameters such as covariance matrices with theirattendant optimality properties (Dempster et al., 1977). Reg-ularized variants of the EM algorithm (Schneider, 2001) like-wise take variations of the missing values around the imputedvalues into account in the estimation of variances and covari-ances, such that they reduce, in the limit of no regularization,to the EM algorithm for Gaussian data. The same could bedone in KG’s method by adding estimated covariance matri-ces of the imputation error to the sample covariance matrixof the completed dataset. This would improve the accuracyof KG’s method, for example, in estimating variability. Ac-curate variability estimates are particularly important for es-timating higher-order statistics, such as extreme value statis-tics, which can be strongly biased if variations of missingvalues around imputed values coming from the center of adistribution of possible values are not taken into account.Correspondence to: T. Schneider(tapio@caltech.edu)(ii) Except for neglecting the variations of the missingvalues around the imputed values and for an unusual orderof iterations – iteratively re-estimating individual principalcomponents – KG’s method is similar to the regularized EMalgorithm exploiting spatial and stationary temporal covari-ability described in Schneider (2001). (A principal compo-nent technique similar to that of KG and Beckers and Rixen(2003) but with the more usual order of iterations – itera-tively re-estimating covariance matrices and all relevant prin-cipal components – was presented by Everson and Sirovich(1995).) KG’s principal component technique for imputingmissing values corresponds to an orthogonal or truncated to-tal least squares (auto-)regression (Fierro et al., 1997) andcan be used in a regularized EM algorithm as discussed inSchneider (2001). As KG’s method, a regularized EM algo-rithm with truncated total least squares regression uses lead-ing principal components based on the entire dataset, includ-ing all records and variables with missing and available val-ues. An innovation in KG’s method is to make the time lagup to which temporal covariability is exploited an adaptiveparameter.(iii) As a result of the similarity of KG’s method and aregularized EM algorithm exploiting spatio-temporal covari-ability with truncated total least squares regressions, severalof KG’s claims of how their method differs from regularizedEM algorithms are incorrect. For example, KG’s contrast-ing of their method as being “non-parametric” as opposed tothe “parametric” regularized EM algorithm is incorrect. TheEM algorithm for Gaussian data yields maximum likelihoodestimates of mean values, covariance matrices, and missingvalues, with their attendant optimality properties, but it andits regularized variants can also be justified under weaker as-sumptions (as least squares methods or regularized variants).KG’s method is just as parametric as the regularized EM al-gorithm.Published by Copernicus GmbH on behalf of the European Geosciences Union and the American Geophysical Union.2 T. Schneider: Comment on “Spatio-temporal filling of missing points . . . ”(iv) There are other inaccuracies, particularly where KGcontrast their method with other methods. For example, it isnot correct that an “EM-based method . . . [relies] on the ran-domness in time of the missing values.” The EM algorithmand regularized variants rely on the assumption that miss-ing values are missing at random, which does not mean thatvalues are missing randomly in time or in space but that theprobability that a value is missing is independent of the miss-ing value – the central necessary condition for the mecha-nisms responsible for missingness to be ignorable (Little andRubin, 2002). KG’s method relies on the same assumption.While using different terms and concepts may createthe impression that methods used to estimate statisticsfrom incomplete data differ more strongly than they do,actual methodological differences, even if small, may beimportant in determining the performance of the methods. Asystematic exploration of the advantages and disadvantagesof different methods is desirable, including methods such asthat of KG that exploit spatio-temporal covariability.Edited by: B. D. MalamudReviewed by: B. D. MalamudReferencesBeckers, J. M. and Rixen, M.: EOF calculations and data fill-ing from incomplete oceanographic datasets, J. Atmos. OceanicTechnol., 20, 1839–1856, 2003.Dempster, A. P., Laird, N. M., and Rubin, D. B.: Maximum like-lihood estimation from incomplete data via the EM algorithm(with discussion), J. Roy. Stat. Soc. B, 39, 1–38, 1977.Everson, R. and Sirovich, L.: Karhunen-Loe`ve procedure for gappydata, J. Opt. Soc. Am. A, 12, 1657–1664, 1995.Fierro, R. D., Golub, G. H., Hansen, P. C., and O’Leary, D. P.: Reg-ularization by truncated total least squares, SIAM J. Sci. Com-put., 18, 1223–1241, 1997.Kondrashov, D. and Ghil, M.: Spatio-temporal filling of missingpoints in geophysical data sets, Nonlin. Processes Geophys., 13,151–159, 2006,http://www.nonlin-processes-geophys.net/13/151/2006/.Little, R. J. A. and Rubin, D. B.: Statistical Analysis with MissingData, Series in Probability and Mathematical Statistics, Wiley,New York, 2nd edn., 2002.Schneider, T.: Analysis of incomplete climate data: Estimation ofmean values and covariance matrices and imputation of missingvalues, J. Climate, 14, 853–871, 2001.Nonlin. Processes Geophys., 14, 1–2, 2007 www.nonlin-processes-geophys.net/14/1/2007/

Comment on "Spatio-temporal filling of missing points in geophysical data sets" by D. Kondrashov and M. Ghil, Nonlin. Processes Geophys., 13, 151?159, 2006

Analysis of incomplete climate data: Estimation of mean values and covariance matrices and imputation of missing values,

EOF calculations and data filling from incomplete oceanographic datasets,

Karhunen-Loe`ve procedure for gappy data,

Maximum likelihood estimation from incomplete data via the EM algorithm (with discussion),

Regularization by truncated total least squares,

Spatio-temporal filling of missing points in geophysical data sets,

Statistical Analysis with Missing Data,

http://authors.library.caltech.edu/7323/1/SCHNnpg07.pdf

Comment on "Spatio-temporal filling of missing points in geophysical data sets" by D. Kondrashov and M. Ghil, Nonlin. Processes Geophys., 13, 151–159, 2006

Comment on "Spatio-temporal filling of missing points in geophysical data sets" by D. Kondrashov and M. Ghil, Nonlin. Processes Geophys., 13, 151–159, 2006

Abstract

Similar works

Full text

Available Versions

HAL-INSU