Trend extraction from time series is often performed by using the filter proposed by Leser (1961), also known as the Hodrick-Prescott filter. A practical problem arises, however, when some data points are missing. This note proposes a method for coping with this problem

Schlicht, Ekkehart

English

Universität München: Elektronischen Publikationen

Trend Extraction From Time Series With Missing Observations

EconStor (ZBW Kiel)

Open Access LMU ( Ludwig-Maximilians-Univ. München)

Ekkehart Schlicht:Trend Extraction From Time Series With MissingObservationsMunich Discussion Paper No. 2007-18Department of EconomicsUniversity of MunichVolkswirtschaftliche FakultätLudwig-Maximilians-Universität MünchenOnline at http://epub.ub.uni-muenchen.de/1927/Trend Extraction From Time Series WithMissing ObservationsEkkehart SchlichtAbstract: Trend extraction from time series is often performed by using the filterproposed by L (), also known as the Hodrick-Prescott filter. A practicalproblem arises, however, when some data points are missing. This note proposes amethod for coping with this problem.Keywords: Trend extraction, missing observations, gaps, Hodrick-Prescott filter, Leserfilter, spline, time-series, smoothing, interpolation.Journal of Economic Literature Classification: C, C, C, C IntroductionTrend extraction from time series is often performed by using the filter proposed by L(), also known as the Hodrick-Prescott filter, or HP-Filter. A practical problem arises,however, when some data points are missing. This note proposed a method for coping withthis problem. The Leser FilterThe idea proposed by L () for the case where all data are available is to look for atrend y ∈ RT such that deviationu = x− y ()is “small” and the trend is “smooth.” The size of the deviation is measured by the sum ofsquared residuals u′u, and the smoothness of the trend is measured by the sum of squares ofchanges in the direction of the trend v′v where the trend disturbances v ∈ RT−2 are definedasvt = ((yt − yt−1)− (yt−1 − yt−2)) t = 3, 4, ... , Torv = P y ()withP :=1 −2 1 01 −2 1. . .0 1 −2 1of order (T − 2)× T .The decomposition of the original series x into trend y and and residual u is obtained byminimizing the weighted sum of squaresV = u′u+ α · v′v = (x− y)′ (x− y) + α · y′P ′P y ()with respect to y. This gives the first-order condition(IT + α · P ′P ) y = x. () The formailization below follows S ().As (I + αP ′P ) is positive definite, the second order condition is satisfied in any case.Equation () has the unique solutiony = (IT + αP′P )−1 x ()which defines the Leser-Filter. It associates a trend y with the time series x, depending onthe smoothing parameter α.From () and () we obtainV = x′(IT − (IT + α · P ′P )−1)x ()as the value of the criterion function (). Missing ObservationsA practical problem arises when some data are missing. Equation () cannot be appliedsuch cases. The obvious way to generalize the filter in order to cope with this problem is tolook for a smooth trend by substituting all missing observations by numbers that minimizethe criterion function (). This can be done as follows.Consider a time series x ∈ RT withm ≤ T − 2 data points missing. The missing data areindicated by a vectorm ∈ RT with componentsmt = 0 for all t ∈ {1, 2, ...T} where xt isdefined, andmt = 1 for all t ∈ {1, 2, ...T} where xt is not defined. Now define a new timeseries z ∈ RT such that zt = xt for all t ∈ {1, 2, ...T} where xt is defined, and zt = 0 forall t ∈ {1, 2, ...T} where xt is not defined. Define the (T ×m)-matrix with componentsbi,1 = 1 if the first missing variable is xi, bj,2 = 1 if the second missing variable is xj , etc.,and all other components of B being zero. For T = 5 and variables  and  missing wewould have, for exampleB =0 01 00 00 10 0 .Define further the vector of substitute values for the missing variables s ∈ Rm, where sigives the substitute value for the i-th missing observation which is to be determined.The replenished vector of observations isx∗ = z +Bs. ()It is a function of the values assumed for the missing values s. These values can now easilybe determined by replacing x by x∗ in (), and minimizing this expression. Thus we obtainthe quadratic formV = (z′ + s′B′)(IT − (IT + α · P ′P )−1)(z +Bs) ()that is to be minimized with respect to s. The necessary condition for a minimum is∂V∂s= 2B′(IT − (IT + α · P ′P )−1)(z +Bs) = 0. ()and the second-order condition is that B′(IT − (IT + α · P ′P )−1)B is positive definite.As(IT + αP′P )−1 = IT − αP ′P + (αP ′P )2 − (αP ′P )3 + ...we can write (IT − (IT + α · P ′P )−1)= αP ′ (IT−2 + α · PP ′)−1 P.Hence(IT − (IT + α · P ′P )−1)is non-negative definite of rank T − 2 andB′(IT − (IT + α · P ′P )−1)B has full rank and is positive definite. Therefore equa-tion () defines the unique maximizing choice of the substitute terms ass∗ = −(B′(IT − (IT + α · P ′P )−1)B)−1B′(IT − (IT + α · P ′P )−1)z. ()The replenished time series is obtained now by inserting () into ()x∗ = z +Bs∗and the trend is obtained by using the replenished series x∗ instead of the original series xin ():y∗ = (IT + αP ′P )−1(z +Bs∗) .10 20 30 40 50246810Figure : US Unemployment -, original series and smoothed series, usinga smoothing constant α = 100. The arrow indicate values that have been omittedin order to produce a time series with missing data. (Data Source: Bureau ofLabor Statistics.)This gives the trend of the time series with missing observations x. An ExampleAs an example, consider the time series of US unemployment (Figure ). I have deleted twodata points, numbers 3 and 27, with values of 2.9% and 7.0%, respectively. The gap at 27 isuncritical because the point sits in the middle of the data range, and assumes also a middleposition between adjacent data points. The gap at  is critical, as it is close to the boundaryof the time series, and is also extreme in its deviation from the trend.Dropping these values and estimating replacements according to the method outlinedabove yields estimated values 4.2% and 6.9%. This is illustrated in Figure . The twotrend series are depicted in Figure . It can be seen that the omission at data point 3 has anoticeable effect, while the omission at data point 27 does not change the trend estimate inany significant way. All computations done with the package by L ().2 3 4 5 6123456deletedreplaced(a)26 27 28 29 305.566.577.58deletedreplaced(b)Figure : Deleted values and their computed replacements at data points 3 (a)and 27 (b).10 20 30 40 50567(a)10 20 30 40 50-0.25-0.2-0.15-0.1-0.050.050.1(b)Figure : (a) Original trend estimation and trend estimated from incomplete data(b) the difference between these two trend estimates. The arrows indicate theposition of the gaps in the data. Concluding CommentsLooking at the substitutions illustrated in Figure , it may be asked whether a simplelinear interpolation would not do as well. In a way, this seems a reasonable positionto take. However, and practically speaking, with contemporary computing power, gapdetection and the substitution would be done automatically in both cases. If adjacent gapsoccur, linear interpolation would require case distinctions that are not necessary in themethod proposed here. In this sense, the proposed method is computationally simpler.The substitution proposed here, as compared to a simple linear interpolation, offers twofurther advantages. First, the substitute values are points on the estimated trend. In thissense, the estimation proposed here does not distort the trend; and second, the estimatedsubstitutes can be interpreted as Maximum-Likelihood estimates of the missing values, ifthe stochastic interpretation of the Leser method proposed by S () is adopted.In short, the method is more systematically linked to the smoothing method at hand thanother interpolation methods are.ReferencesL, C. E. V. , “A SimpleMethod of Trend Construction,” Journal of the Royal StatisticalSociety. Series B (Methodological), , –.L, J. , “HPFilter,” Mathematica package, Wolfram Research, online at http://library.wolfram.com/infocenter/MathSource/Mathematics/ProbabilityStatistics/.S, E. , “A Seasonal Adjustment Principle and a Seasonal Adjustment MethodDerived from This Principle,” Journal of the American Statistical Association, (),–, online at http://www.semverteilung.vwl.uni-muenchen.de/mitarbeiter/es/paper/schlicht-seasonal_adjustment.pdf .— , “Estimating the Smoothing Constant in the So-Called Hodrick-Prescott Filter,”Journal of the Japan Statistical Society, , –, online at http://www.scipress.org/journals/jjss/pdf//.pdf .

http://epub.ub.uni-muenchen.de/1927/1/schlicht_missing_data-DP18.pdf

Trend Extraction From Time Series With Missing Observations

Abstract

Similar works

Full text

Available Versions

Universität München: Elektronischen Publikationen

EconStor (ZBW Kiel)

Open Access LMU ( Ludwig-Maximilians-Univ. München)