Trend extraction from time series is often performed by using the filter proposed by Leser (1961), also known as the Hodrick-Prescott filter. Practical problems arise, however, if the time series contains structural breaks (as produced by German unification for German time series, for instance), or if some data are missing. This note proposes a method for coping with these problems

Schlicht, Ekkehart

Journal of the Japan Statistical Society

English

Ekkehart Schlicht

Crossref

JOURNAL OF THE JAPAN STATISTICAL SOCIETY

Trend Extraction from Time Series with Structural Breaks and Missing Observations

Open Access LMU

Ekkehart Schlicht:Trend Extraction From Time Series With StructuralBreaks and Missing ObservationsMunich Discussion Paper No. 2008-3Department of EconomicsUniversity of MunichVolkswirtschaftliche FakultätLudwig-Maximilians-Universität MünchenOnline at http://epub.ub.uni-muenchen.de/2127/Trend Extraction From Time Series WithStructural Breaks and Missing ObservationsEkkehart Schlicht∗Abstract: Trend extraction from time series is often performed by using the filterproposed by L (), also known as the Hodrick-Prescott filter. Practical problemsarise, however, if the time series contains structural breaks (as produced by Germanunification for German time series, for instance), or if some data are missing. Thisnote proposes a method for coping with these problems.Keywords: dummies, gaps, Hodrick-Prescott filter, interpolation, Leser filter, missingobservations, smoothing, spline, time-series, trend, structural breaks, break point,break point locationJournal of Economic Literature Classification: C, C, C, C∗Department of Economics, University of Munich, Schackstr. ,  Munich, Germany, schlicht@lmu.de.The final version of the paper will appear in the Journal of the Japan Statistical Society. I thank an anonymousreferee of that journal for very helpful comments and for suggesting the issue of break point location. IntroductionTrend extraction from time series is often performed by using the filter proposed by L(), also known as the Hodrick-Prescott filter, or HP-Filter. Practical problems arise,however, if the time series contains structural breaks (as produced by German unificationfor German time series, for instance), or if some data are missing. This note proposes amethod for coping with these problems. The Leser FilterThe idea proposed by L () for the case where all data are available is to look for atrend y ∈ RT such that deviationu = x− y ()is “small” and the trend is “smooth.” The size of the deviation is measured by the sum ofsquared residuals u′u, and the smoothness of the trend is measured by the sum of squares ofchanges in the direction of the trend v′v where the trend disturbances v ∈ RT−2 are definedasvt = ((yt − yt−1)− (yt−1 − yt−2)) t = 3, 4, ... , Torv = P y ()withP :=1 −2 1 01 −2 1. . .0 1 −2 1of order (T − 2)× T .The decomposition of the original series x into trend y and and residual u is obtained byminimizing the weighted sum of squaresV = u′u+ α · v′v = (x− y)′ (x− y) + α · y′P ′P y ()with respect to y. The smoothing constant α denotes the weight given for the trenddeviations. It is typically selected in an arbitrary way but may also be estimated from the The formalization below follows S ().time series (S, ). Minimization of () entails the first-order condition(IT + α · P ′P ) y = x. ()As (I + αP ′P ) is positive definite, the second order condition is satisfied in any case.Equation () has the unique solutiony = (IT + αP′P )−1 x ()which defines the Leser-Filter. It associates a trend y with the time series x, depending onthe smoothing parameter α.From () and () we obtainV = x′(IT − (IT + α · P ′P )−1)x ()as the value of the criterion function (). Formalizing Structural Breaks and Missing ObservationsPractical problem arise with time series containing structural breaks and missing observa-tions. Equation () would interpret structural breaks as changes in the trend, and cannoteven be applied if some data points are missing. The obvious way to generalize the filter inorder to cope with this problem is to introduce dummies for the structural breaks, and tosubstitute missing values of the time series by numbers that minimize the criterion function() and generate a trend as smooth as possible. This can be done as follows.Consider a raw time series x of length T with m structural breaks and n missing datapoints. We require m+ n ≤ T − 2 and m ≥ 0 and n ≥ 0. The structural breaks occur atpoints in time b = (b1, b2, ...bm) ∈ Zm+ , where Z+denotes the set on non-negative integers.If the first break point is at t = 3, we would have b1 = 3, for instance. The missing dataare missing at points c = (c1, c2, ...cn) ∈ Zn+. If the first missing data point is at t = 5, wewould have c1 = 5, for instance.Given this information, we define the filled time series x˜ ∈ RT by taking the raw timeseries x and replacing all undefined elements by zero, viz. x˜ci = 0 for all i = 1, 2, ..., n.Further, we define a T ×m matrix D with elements di,j = 0 for i < bj and di,j = 1 fori ≥ bj . For T = 5 and break points at b1 = 2 and b2 = 4 we would have, for exampleD =0 01 01 01 11 1 .In a similar way, we define a T × n matrix E with elements ei,1 = 1 if the first missingvariable is xi, ej,2 = 1 if the second missing variable is xj , etc., and all other components ofE being zero. For T = 5 and variables  and  missing we would have, for exampleE =0 01 00 00 10 0 .The block matrixF = (D,E)combines these two matrices and will be used for further computations. It is of orderT × (m+ n), and we require that it is of full rank m + n. This assumption assures thatmissing observations do not mask structural breaks fully. With the above example we wouldhave rank (D,E) = 4, and the requirement would be satisfied, but if we had D as aboveandE =0 00 00 01 00 1we would have rank (D,E) = 3 and the requirement would not be met; a structural breakwould entirely be masked by missing data.Define further the vector of dummies d ∈ Rm, where di is the dummy for the i-thstructural break and define the vector e ∈ Rn of replacements for the missing (or empty)observations. We combine these two vectors in the vector f ′ = (d′, e′) ∈ Rm+n. Giventhese definitions, the stage is set for dealing with the estimation problem. EstimationDefine the amended vector of observations asx∗ = x˜+ Ff. ()It is a function of the filled time series x˜, the values assumed for the dummies d and thereplacements e comprised in the vector f . The vector f can now easily be determined byreplacing x by x∗ in (), and minimizing this expression. Thus we obtain the quadraticformV = (x˜′ + f ′F ′)(IT − (IT + α · P ′P )−1)(x˜+ Ff) ()that is to be minimized with respect to f . The necessary condition for a minimum is∂V∂s= 2F ′(IT − (IT + α · P ′P )−1)(x˜+ Ff) = 0. ()and the second-order condition is that F ′(IT − (IT + α · P ′P )−1)F be positive definite.As(IT + αP′P )−1 = IT − αP ′P + (αP ′P )2 − (αP ′P )3 + ...we can write (IT − (IT + α · P ′P )−1)= αP ′ (IT−2 + α · PP ′)−1 P.Hence(IT − (IT + α · P ′P )−1)is non-negative definite of rank T − 2 andF ′(IT − (IT + α · P ′P )−1)F = αF ′P ′ (IT−2 + α · PP ′)−1 PF has full rank and ispositive definite. Therefore equation () defines the unique minimizing choice of thedummies and missing terms asf ∗ = −(F ′(IT − (IT + α · P ′P )−1)F)−1F ′(IT − (IT + α · P ′P )−1)x˜. ()The refurbished time series is obtained now by inserting () into ()x∗ = x˜+ Ff ∗and the trend is obtained by using the refurbished series x∗ instead of the original series xin ():y∗ = (IT + αP ′P )−1(x˜+ Ff ∗) .This gives the trend of the time series with structural breaks and missing observations x. Example: Structural BreaksAs an example for treating structural breaks, consider the time series of US unemployment(Figure ). Beginning with period , a structural break has been introduced by adding percentage points to the original time series. The correction obtained by the methodsketched above overcorrects this break by subtracting . percentage points. The correctedtrend estimation is overcorrected as well. As can be seen, the correction produces a smoothertrend than the original one, as is implied by the logic of the method. A manual correctionwould look not very much different, or would look even worse if the adjustment is madesuch that the adjacent data points  and  are made to have identical values. (Thecorrection would have been -. rather than -. in this case.)The example illustrates the functioning, as well as the problematic, of introducing dum-mies, as these will not only correct for structural breaks, but will also mask changes in theunderlying trend.Another illustration is provided in Figure (a). It depicts the time series with the structuralbreak, as given in Figure (a) together with the estimated trend of the corrected series plusthe estimated structural break. Digression: Locating Structural BreaksSometimes the analyst may be in doubt about the exact positioning of the structural break.A simple way to deal with such uncertainty would be to estimate corrected time paths x∗ foralternative break points and evaluate the criterion () for these alternative time-paths. Thepreferred break point would be the one giving the smallest value for the criterion. Figure All computations done with the package by L ().10 20 30 40 50246810121416with breakoriginalcorrected(a)10 20 30 40 50246810121416with breakoriginalcorrected(b)Figure : (a) The original US unemployment rate - has been augmentedby adding, beginning with period 25, five percentage points to the original se-ries. The corrected series overcorrects the structural break in this case. (b) Thesmoothed series reproduce this pattern. (Smoothing constant is α =100, datafrom the Bureau of Labor Statistics.)10 20 30 40 50246810121416(a) Time series and estimated trend with esti-mated break0 10 20 30 40 5025(b) The criterion () for alternative break pointsFigure : (a) The time series with breaks (the top time series from Fig. (a))together with the the estimated trend (the bottom series from Fig. (b)), increasedby the estimated jump of 7.2 from period 25 upwards. (b) The criterion () forthe time series depicted in Figure (a) corrected for alternative assumed breakpoints. The minimum occurs at point 25, which is the correct break point.10 20 30 40 50246810Figure :US Unemployment -, original series and smoothed series, usinga smoothing constant α = 100. The arrow indicate values that have been omittedin order to produce a time series with missing data. (Data Source: Bureau ofLabor Statistics.)(b) depicts the value of the criterion for alternative break points and the correspondingcorrected time paths. The minimum is attained at the correct break point 25. Example: Missing ObservationsConsider again the time series of US unemployment (Figure ). I have deleted two datapoints, numbers 3 and 27, with values of 2.9% and 7.0%, respectively, to obtain a timeseries with missing observations. The gap at 27 is uncritical because the point sits in themiddle of the data range, and assumes also a middle position between adjacent data points.The gap at  is critical, as it is close to the boundary of the time series, and is also extreme inits deviation from the trend.Dropping these values and estimating replacements according to the method outlinedabove yields estimated values 4.2% and 6.9%. This is illustrated in Figure . The twotrend series are depicted in Figure . It can be seen that the omission at data point 3 has anoticeable effect, while the omission at data point 27 does not change the trend estimate inany significant way.Looking at the substitutions illustrated in Figure , it may be asked whether a simplelinear interpolation would not do as well. In a way, this seems a reasonable position to take.2 3 4 5 6123456deletedreplaced(a)26 27 28 29 305.566.577.58deletedreplaced(b)Figure : Deleted values and their computed replacements at data points 3 (a)and 27 (b).10 20 30 40 50567(a)10 20 30 40 50-0.25-0.2-0.15-0.1-0.050.050.1(b)Figure : (a) Original trend estimation and trend estimated from incomplete data(b) the difference between these two trend estimates. The arrows indicate theposition of the gaps in the data.However, and practically speaking, with contemporary computing power, gap detectionand the substitution would be done automatically in both cases. If adjacent gaps occur,linear interpolation would require case distinctions that are not necessary with the methodproposed here. In this sense, the proposed method is computationally simpler. Concluding CommentsThe unified treatment of structural breaks and missing observations proposed here maynot, practically speaking, be very much different from doing similar adjustments “byhand,” as is common practice. The treatment proposed here can easily be automated,though, and appears less arbitrary. Further, the dummies selected here, and the substitutesfor missing values can be interpreted as maximum-likelihood estimates, if the stochasticinterpretation of the Leser method proposed by S () is adopted, and thesmoothing constant may be estimated by the method given there. In short, the methodis more systematically linked to the smoothing method at hand than other methods ofadjustment and interpolation are.ReferencesL, C. E. V. , “A Simple Method of Trend Construction,” Journal of the Royal StatisticalSociety. Series B (Methodological), , –.L, J. , “HPFilter,” Mathematica package, Wolfram Research, online at http://library.wolfram.com/infocenter/MathSource//.S, E. , “A Seasonal Adjustment Principle and a Seasonal Adjustment MethodDerived from This Principle,” Journal of the American Statistical Association, (),–, online at http://www.semverteilung.vwl.uni-muenchen.de/mitarbeiter/es/paper/schlicht-seasonal_adjustment.pdf .— , “Estimating the Smoothing Constant in the So-Called Hodrick-Prescott Filter,”Journal of the Japan Statistical Society, , –, online at http://www.scipress.org/journals/jjss/pdf//.pdf .

A Seasonal Adjustment Principle and a Seasonal Adjustment Method Derived from This Principle,”

Estimating the Smoothing Constant in the So-Called Hodrick-Prescott Filter,”

HPFilter,” Mathematica package, Wolfram Research, online at http:

Trend Extraction From Time Series With Structural Breaks and Missing Observations

Universität München: Elektronischen Publikationen

EconStor (ZBW Kiel)

http://epub.ub.uni-muenchen.de/2127/1/JJSS-Missing_Date_and_Breaks_DP.pdf

Trend Extraction From Time Series With Structural Breaks and Missing Observations

Abstract

Similar works

Full text

Available Versions

Crossref

Open Access LMU

Universität München: Elektronischen Publikationen

EconStor (ZBW Kiel)