International audienceNo abstract available

Ghil, M.

Kondrashov, D.

HAL-INSU

Reply to T. Schneider’s comment on ”Spatio-temporalfilling of missing points in geophysical data sets”D. Kondrashov, M. GhilTo cite this version:D. Kondrashov, M. Ghil. Reply to T. Schneider’s comment on ”Spatio-temporal filling of miss-ing points in geophysical data sets”. Nonlinear Processes in Geophysics, European GeosciencesUnion (EGU), 2007, 14 (1), pp.3-4. <hal-00331096>HAL Id: hal-00331096https://hal.archives-ouvertes.fr/hal-00331096Submitted on 15 Jan 2007HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.Nonlin. Processes Geophys., 14, 3–4, 2007www.nonlin-processes-geophys.net/14/3/2007/© Author(s) 2007. This work is licensedunder a Creative Commons License.Nonlinear Processesin GeophysicsReply to T. Schneider’s comment on “Spatio-temporal filling ofmissing points in geophysical data sets”D. Kondrashov1 and M. Ghil1,*1University of California, Los Angeles, CA, USA*Ecole Normale Supe´rieure, Paris, FranceReceived: 9 August 2006 – Revised: 21 December 2006 – Accepted: 22 December 2006 – Published: 15 January 2007First, we thank T. Schneider (TS hereafter) for his posi-tive and constructive comments about Kondrashov and Ghil(2006) (KG hereafter). KG focused on exploiting temporalcovariability in geophysical data sets, an idea that Schneider(2001; S01 hereafter) had suggested, but not applied to anydata, synthetic or geophysical. Two unfortunate inaccuracies– corrected in comments (iii) and (iv) of TS – did crop upwhen KG described the expectation-maximization (EM) al-gorithm and its regularized version used by S01 for filling inmissing data. We regret this slip, being thoroughly familiarwith the general EM framework, which we used for prob-ability density estimation when studying multiple weatherregimes (Smyth et al., 1999; Kondrashov et al., 2004, 2006).We thus agree with TS that the regularized EM algorithmand KG’s method are both based on estimating mean and co-variance components of the gappy data set under study (hiscomment (iii)), and that several gap-filling methods, includ-ing regularized EM and our own (multi-channel) singular-spectrum analysis (M-)SSA, rely – among many other as-sumptions – also on the probability of a data point’s absencebeing independent of the missing value itself (his comment(iv)).Singular-value decomposition (SVD; Golub and VanLoan, 1989) underlies both regression and principal compo-nents analysis, and thus represents a common basis for KG’sM-SSA as well as S01’s regularized EM method. In thisreply, we concentrate on discussing several differences be-tween our methods, which might look minor to TS, but leadto differences in computational performance and numericalresults in practical applications. We have tried out, beforesubmitting KG, the free gap-filling software kindly providedon TS’s personal website and plan to add KG’s gap-fillingfeature as soon as feasible to the SSA-MTM Toolkit, avail-able for free at http://www.atmos.ucla.edu/tcd/ssa; see alsoGhil et al. (2002).Correspondence to: D. Kondrashov(dkondras@atmos.ucla.edu)KG aim to fill the gaps with smooth information from aniteratively inferred “signal” that represents coherent spatio-temporal structures, and discard the “noise” variance. Thisidea is best illustrated by our synthetic example of a noise-contaminated oscillatory signal with a gap, see Fig. 2 in KG.Doing so can be quite valuable in a wide variety of applica-tions, ranging from climate predictability and paleoclimatereconstruction to oceanographic and space physics data. Toillustrate as simply as possible the difference between regu-larized EM (S01) and M-SSA (KG), consider a multivariatedata set with just one missing value xij in one record at timetj ; the number of points i in each record (channels) is M ,while the number of records (i.e., of sampling times tj ) is N .S01 (see Eq. 1 there) stresses regularization of the EM algo-rithm for rank-deficient cases, while in M-SSA regularizationcomes in the form of discarding “noise” EOFs, regardless ofwhether the data set is rank-deficient or not; see also Fig. A1in Ghil et al. (2002).For simplicity, we consider in this reply reconstructionbased on spatial correlations only; our method is identical,in this case, to that of Beckers and Rixen (2003), while theemphasis in KG was on the use of purely temporal or mixed,spatio-temporal correlations. In our approach, we start aninner-loop iteration by computing the leading empirical or-thogonal function (EOF) of the centered, zero-padded record.Then we perform the algorithm again on the new time se-ries in which the principal component corresponding to thatEOF alone was used to obtain nonzero values in place of themissing point and correct the channel’s mean, the covariancematrix and the EOF. When this inner iteration has converged,we perform an outer-loop iteration by adding a second EOFfor reconstruction and repeat the inner iteration. The outeriteration is performed only for a few significant, or “signal”EOFs, whose number is found by cross-validation. Beckersand Rixen (2003) discuss, in their Appendix A, how the biasintroduced into the EOFs by missing data disappears as theiteration progresses.Published by Copernicus GmbH on behalf of the European Geosciences Union and the American Geophysical Union.4 D. Kondrashov and M. Ghil: Reply to T. Schneider’s commentIn S01, iterations are also used in order to obtain an esti-mate of the missing value xij , along with the temporal mean<xi> in the spatial channel i, and the spatial covariance ma-trix. Estimating the regression coefficients in the EM algo-rithm with ridge regression has to be done record by record(see Eq. 2 in S01), including both “signal” and “noise” EOFsin the covariance matrix. As the number of records withmissing data increases, KG offers potential computationalsavings since KG need (and want) to compute only a fewleading EOFs in the outer iteration.KG’s double iteration differs, furthermore, from theKarhunen-Loe`ve procedure for image processing of Eversonand Sirovich (1995), where a previously fixed number ofEOFs are estimated simultaneously in a single estimationloop. Everson and Sirovich’s version of gap filling is closestto using the total truncated least-squares (TTLS) option incombination with the EM algorithm in S01. In this case,the relative computational performance of both KG andEM methods will largely depend on the number of EOFsinvolved, but ideas from both approaches could be useful indevising even better gap-filling methods in the future.Edited by: B. D. MalamudReviewed by: T. Schneider and another refereeReferencesBeckers, J. and Rixen, M.: EOF calculations and data filling fromincomplete oceanographic data sets, J. Atmos. Ocean. Technol.,20, 1839–1856, 2003.Everson, R. M. and Sirovich, L.: The Karhuren-Loe`ve transformfor incomplete data, J. Opt. Soc. Am., 12, 1657–1664, 1995.Ghil, M., Allen, R. M., Dettinger, M. D., Ide, K., Kondrashov, D.,et al.: : Advanced spectral methods for climatic time series, Rev.Geophys., 40(1), 3.1–3.41, doi:10.1029/2000RG000092, 2002.Golub, G. H. and Van Loan, C. F.: Matrix Computations (3rd ed.),The Johns Hopkins University Press, Baltimore and London, 728pp, 1996.Kondrashov, D., Ide, K., and Ghil, M.: Weather regimes and pre-ferred transition paths in a three-level quasigeostrophic model, J.Atmos. Sci., 61, 568–587, 2004.Kondrashov, D., Kravtsov, S., and Ghil, M.: Empirical mode re-duction in a model of extratropical low-frequency variability, J.Atmos. Sci., 63, 1859–1877, 2006.Smyth, P., Ide, K., and Ghil, M.: Multiple regimes in NorthernHemisphere height fields via mixture model clustering, J. Atmos.Sci., 56, 3704–3723, 1999.Schneider, T.: Analysis of incomplete climate data: Estimation ofmean values and covariance matrices and imputation of missingvalues, J. Climate, 14, 853–871, 2001.Nonlin. Processes Geophys., 14, 3–4, 2007 www.nonlin-processes-geophys.net/14/3/2007/

Reply to T. Schneider's comment on "Spatio-temporal filling of missing points in geophysical data sets"

https://hal.archives-ouvertes.fr/hal-00331096

Reply to T. Schneider's comment on "Spatio-temporal filling of missing points in geophysical data sets"

Abstract

Similar works

Full text

Available Versions

HAL-INSU