In this paper, we portray the essential features of the finite-sample signal extraction problem in both the stationary and the nonstationary cases. The computational procedures can be simplified in the light of our analysis. An important outcome of the analysis is a demonstration that the start-up problem can be handled far more easily that one might expect from a passing acquaintance with the usual practices.Signal extraction, Linear filtering, Trend estimation

Stephen Pollock

English

Research Papers in Economics

Department of Economics
Signal Extraction, Maximum Likelihood Estimation
Working Paper No. 433           February 2001           ISSN 1473-0278
Stephen Pollock
and the Start-up ProblemSignal Extraction, Maximum Likelihood
Estimation and the Start-up Problem
by D.S.G. POLLOCK
Queen Mary, University of London
and
GREQAM: Groupement de Recherche en
Economie Quantitative d’Aix–Marseille
Email: stephen pollock@sigmapi.u-net.com
In this paper, we portray the essential features of the ﬁnite-sample signal
extraction problem in both the stationary and the nonstationary cases. The
computational procedures can be simpliﬁed in the light of our analysis.
An important outcome of the analysis is a demonstration that the start-
up problem can be handled far more easily that one might expect from a
passing acquaintance with the usual practices.
Date: January, 2001
Key words: Signalextraction, Linear ﬁl tering, Trend estimation
JEL Classiﬁcation: C22
1. Introduction
The treatment of signal-extraction problems in econometrics rests heavily upon
the Wiener–Kolmogorov theory, which is a mainstay of communications en-
gineering. The theory envisages that the data are generated by stationary
stochastic processes and that they form a lengthy sequence. Therefore, the
classical results of the Wiener–Kolmogorov theory have been developed in the
context of the theoretically tractable assumption that the data form a doubly
inﬁnite sequence stretching into the indeﬁnite past and the indeﬁnite future.
(For the originalexpositions, see Kol mogorov [5] and Wiener [12]. For an ac-
cessible modern treatment of the theory see Whittle [13].)
In practice, econometric time series are of a strictly limited duration and
they are often highly nonstationary; and some radical adaptations of the
Wiener–Kolmogorov theory are necessary if it is to be applied successfully
in such circumstances. In econometric analysis, the adaptations have been
accomplished, typically, by adopting the device of the Kalman ﬁlter and its as-
sociated smoothing algorithms, (see Harvey [2] and Harvey and Todd [3]). The
non-stationarity of the data necessitates a carefulhandl ing of the start-up prob-
lem; and this has been accomplished via a development known as the diﬀuse
Kalman ﬁlter, (see De Jong [1], whose algorithm has been used by Koopman
et al. [6]).
1D.S.G. POLLOCK: Signal Extraction
The Kalman ﬁlter is a very ﬂexible device which is also a complex one, as
it must be if it is to achieve a high degree of generality. (For an account of
the Kalman ﬁlter and of the associated smoothing algorithms, see Pollock [9]).
Part of the complexity of its associated algorithms is due to fact that they are
intended for online processing, which is rarely a requirement in econometric
analysis. Therefore, there is scope for simplifying the treatment of the econo-
metric problem, the essential features of which are masked by the complexity
of the Kalman ﬁlter. In fact, the econometric problem is a relatively simple
one which can be approached in a direct manner.
In this paper, we portray the essentialfeatures of the ﬁnite-sampl e signal
extraction problem in both the stationary and the nonstationary cases. In the
light of our analysis, the computational procedures can be simpliﬁed somewhat.
An important outcome of the analysis is a demonstration that the start-up
problem can be handled far more easily that one might expect from a passing
acquaintance with the usualpractices.
2. The Statistical Assumptions
Consider a vector
(1) y = ξ + η,
of T observations, which consists of a signalcomponent ξ and a noise component
η. It is assumed that
(2) ξ ∼ N(0,Ωξ) and η ∼ N(0,Ωη)
are independent, normally distributed, random vectors generated by two con-
temporaneous stationary stochastic processes. The assumption of independence
implies that
(3)
D(y)=Ω ξ +Ω η,
C(ξ,y)=Ω ξ and
C(η,y)=Ω η.
The joint density function of ξ and η is
(4) N(ξ,η)=( 2 π)−T|Ωξ|−1/2|Ωη|−1/2 exp{−
1
2
(ξ Ω
−1
ξ ξ + η Ω−1
η η)},
whereas the density function of their sum y = ξ + η is
(5) N(y)=( 2 π)−T/2|Ωξ +Ω η|−1/2 exp{−
1
2
y (Ωξ +Ω η)−1y}.
2D.S.G. POLLOCK: Signal Extraction
An alternative assumption regarding ξ is that it represents a trend com-
ponent generated by a nonstationary ARIMA process incorporating d roots of
unity. Then ξ will be reduced to stationarity by a compound matrix diﬀerence
operator Q  of order (T − d) × T, with the result that
(6) Q ξ = ζ ∼ N(0,Ωζ).
In that case, we must consider, in place of (4), the density function
(7) N(ζ,η)=( 2 π)−(2T−d)/2|Ωζ|−1/2|Ωη|−1/2 exp{−
1
2
(ξ QΩ
−1
ζ Q ξ+η Ω−1
η η)},
whereas (5) must be replaced by
(8) N(g)=( 2 π)−(T−d)/2|Ωζ + Q ΩηQ|−1/2 exp{−
1
2
g (Ωζ + Q ΩηQ)−1g},
where g = Q y is the diﬀerenced version of the data vector.
In statistical signal extraction, we face two related problems. The ﬁrst
problem is to estimate the parameters of the distributions of ξ and η. The
second problem is to ﬁnd estimates of ξ and η given the data vector y and
given the values of the statistical parameters, which amounts to knowing Ωξ or
Ωζ and knowing Ωη. We shall concentrate on the second of these two problems,
and we shall pass a few comments on the ﬁrst problem at the end of the paper.
3. Extracting the Latent Components
The problem of estimating ξ and η, in the case where ξ is stationary, can
be construed as a matter of maximising the likelihood function N(ξ,η)o f( 4 )
subject to the condition that ξ + η = y. This entails minimising a chi-square
criterion function:
(9) S(ξ)=( y − ξ) Ω−1
η (y − ξ)+ξ Ω
−1
ξ ξ.
It makes no odds whether we consider this criterion function or an analogous
function with η = y−ξ as its argument. The eventual results will be the same.
Diﬀerentiating S with respect to ξ and setting the result to zero gives the
ﬁrst-order condition from which the minimising value can be obtained:
(10) x =( Ω
−1
ξ +Ω −1
η )−1Ω−1
η y.
However, a familiar matrix identity concerning the inverse of a sum of matrices
(see Pollock [9, p. 228], for example) enables us to write
(11)
(Ω
−1
ξ +Ω −1
η )−1 =Ω ξ − Ωξ(Ωξ +Ω η)−1Ωξ
=Ω η − Ωη(Ωξ +Ω η)−1Ωη.
3D.S.G. POLLOCK: Signal Extraction
Using the second of these identities, we get
(12)
x = {I − Ωη(Ωξ +Ω η)−1}y
= y − h,
where h =Ω η(Ωξ +Ω η)−1y is the estimate of η. Since Ωη(Ωξ +Ω η)−1 +
Ωξ(Ωξ +Ω η)−1 = I, we can also write x =Ω ξ(Ωξ +Ω η)−1y.
It is easy to see, in reference to (3), that
(13)
x = E(ξ|y)=E(ξ)+C(ξ,y)D−1(y){y − E(y)} and
h = E(η|y)=E(η)+C(η,y)D−1(y){y − E(y)}
are equally the conditional expectations and the minimum-mean-square-error
estimates of ξ and η. Here, we have E(ξ)=E(η) = 0 and, therefore, E(y)=0 .
Now consider the case where ξ is generated by a nonstationary ARIMA
process. Then the chi-square criterion function of (9), by which the likelihood
is maximised, is replaced by
(14) S(ξ)=( y − ξ) Ω−1
η (y − ξ)+ξ QΩ
−1
ζ Q ξ,
of which the minimising value is
(15) x =( QΩ
−1
ζ Q  +Ω −1
η )−1Ω−1
η y.
The matrix inversion lemma indicates that
(16) (QΩ
−1
ζ Q  +Ω −1
η )−1 =Ω η − ΩηQ(Q ΩηQ +Ω ζ)−1Q Ωη,
and it follows that
(17)
x = {I − ΩηQ(Q ΩηQ +Ω ζ)−1Q }y
= y − h,
Equation (17) seems markedly more complicated that equation (12). How-
ever, the elaboration is due entirely to the matrix diﬀerence operator Q  which
will have zeros everywhere except on d + 1 diagonal bands. It is unlikely that
d will exceed 2.
The task of computing h can be accomplished by a handful of direct multi-
plications and recursions. The matter may be illustrated by the case where Ωζ
and Ωη are variance–covariance matrices of moving-average processes. There-
fore consider
(18)
h =Ω ηQ(Q ΩηQ +Ω ζ)−1g
=Ω ηQb.
4D.S.G. POLLOCK: Signal Extraction
The ﬁrst task is to calculate b by solving the equation
(19) (Q ΩηQ +Ω ζ)b = g.
The solution is found via a Cholesky decomposition which sets Q ΩηQ +Ω ζ =
GG , where G is a lower-triangular matrix. The system GG b = g can be cast
in the form of Gp = q and solved for p. Then G b = p can be solved for b.
These are the recursive operations. Finding h =Ω ηQb thereafter entails only
direct multiplications.
If Ωξ and Ωη are variance–covariance matrices of autoregressive processes,
then, knowing that their inverses are matrices with a limited number of diagonal
bands, we might choose to calculate the signal estimate x, in the stationary case,
via the formula of (10).
In general, the matrix on the LHS of (19) has the structure of the variance–
covariance matrix of an ARMA process. At its most complicated, it takes
the form of Q ΩηQ +Ω ζ = A−1VA  −1, where A is a banded lower-triangular
Toeplitz matrix, due to the autoregressive part, and V is a symmetric matrix
with a limited number of nonzero bands around the diagonal, (see, for example,
Pollock [9, Ch. 22]). The system A−1VA  −1b = g can be cast in the form of
Vp= q, where p = A −1b and q = Ag. The vector q can be calculated by a direct
multiplication and, when this is available, the solution for p can be obtained
via the Cholesky decomposition of V . Then b = A p can be calculated.
4. Handling the Start-UpProblem
Problems often arise from not knowing how to supply the initial conditions
with which to start the recursive ﬁltering procedures that are entailed in the
extraction of latent signals and other components of the data. By choosing
inappropriate starting values, one can generate so-called transient eﬀects which
are liable, in fact, to aﬀect all of the processed values.
The start-up problem can sometimes be overlooked without detriment
when the data components are generated by zero-mean stationary stochastic
processes. In that case, the zero-valued expectations of the process can serve
as the initialconditions for the recursive ﬁl tering. However, what may seem
remarkable is the absence of an explicit treatment of the problem in our forego-
ing analysis, which has been devoted to the extraction of nonstationary trends;
and an explanation is called for.
For a simple explanation of how the problem has been handled, we can
concentrate on equation (17) which provides the estimate x of the tend. This
estimate is obtained by subtracting the estimate h of a stationary noise or
residue component from the data vector y. Since the noise process has a zero-
valued expectation, it transpires that the recursive procedures that generate
the vector h do not require explicit start-up values. However, there are some
more intriguing explanations of the treatment of the start-up problem.
5D.S.G. POLLOCK: Signal Extraction
Let us consider reducing the data vector to stationarity using the matrix
Q . The result is
(20)
Q y = g = ζ + κ
= z + k,
where ζ = Q ξ and κ = Q η are the diﬀerenced versions of the trend and the
residue, respectively, and where z = Q x and k = Q h are their estimates. It
can be show, via the foregoing analysis of the signal-extraction problem in the
stationary case, that
(21)
z =Ω ζ(Q ΩηQ +Ω ζ)−1g and
k = Q ΩηQ(Q ΩηQ +Ω ζ)−1g.
Now it might seem enough to remark that the estimate of h of η, given by
equation (18), is obtained from the estimate k of κ = Q η, given above, simply
by stripping away the matrix factor Q . However, since Q  is a noninvertible
matrix, such an argument is insuﬃcient.
There are two legitimate ways of recovering x and h from z and k which
are mathematically equivalent. The ﬁrst way is to solve the following problem:
(22) Minimise (y − x) Ω−1
η (y − x) subject to Q x = z.
This is a matter of ensuring that the trend estimate x follows the data sequence
y as closely as possible, subject to the condition that the diﬀerenced trend
estimate Q x equals the estimate z of (21). This prescription leads to the value
of x given in equation (17).
The second way of approaching the problem is to seek to generate the value
of h by accumulating the values of its diﬀerenced version k. For this purpose, it
is necessary to estimate some starting values with which to begin the process of
cumulation. Let M =[ S∗,S] be the inverse of the (invertible) matrix diﬀerence
operator ∇ =( I−LT)d, where LT =[ e1,...,e T−1,0] is the matrix lag operator
which is obtained from the identity matrix IT =[ e0,e 1,...,e T−1] of order T
by deleting the leading column and by appending a vector of zeros to the last
column. Then the estimate of h will be given by the equation
(23) h = S∗k∗ + Sk,
where k∗ is a vector of d initialconditions or start-up val ues. The start-up
values are obtained by evaluating the criterion
(24) Minimise (S∗k∗ + Sk) Ω−1
η (S∗k∗ + Sk) with respect to k∗.
As one might expect, this leads to the value of h given in equation (18). A
formal demonstration of this result had been provided by Pollock [11].
6D.S.G. POLLOCK: Signal Extraction
5. Determining the Statistical Parameters
The estimation of the statisticalparameters is a common accompaniment of the
business of extracting the latent components, but, as we shall suggest, there
are circumstances in which the requisite parameters can be derived by other
means. If the parameters are to be estimated, then it is appropriate to pursue
the method of maximum likelihood.
If the components of the data are generated by stationary ARMA pro-
cesses, then their sum will also be an ARMA process. In the special case where
both the components are generated by moving-average processes, their sum will
also be a moving-average process.
The likelihood function, in the case of stationarity, is provided by (5). In
the case of a nonstationary data process, the data vector y is generated by an
ARIMA process, and it is the likelihood function pertaining to the diﬀerenced
data g = Q y which must be considered. This is provided by equation (8).
The likelihood function can be written either in terms of the parameters of
the component processes or in terms of the parameters of a composite univariate
process which results from adding the components. Harvey and Todd [3] have
described the former as the structural likelihood function and the latter as the
reduced-form function. The parameters of the reduced form may be subject to
certain restrictions arising out of the speciﬁcation of the structuralform.
If the reduced form is subject to restrictions, then it is easiest to estimate
the structuralparameters via the structuralform of the l ikel ihood function.
This is the approach that has been pursued by Harvey [2].
If there are no restrictions on the reduced form, then the option exists of
of inferring the estimates of the structuralparameters from estimates of the
parameters of the unrestricted reduced form. This approach has been taken by
Hillmer and Tiao [4], by Pierce [8] and by Maravall and Pierce [7].
In an alternative approach to econometric signal extraction, the statistical
parameters are determined by manipulation with a view to ensuring that the
resulting signal-extraction ﬁlter has certain preconceived properties. A common
objective is to derive a lowpass ﬁlter with a designated cut-oﬀ frequency for
which the transition from the pass band to the stop band is a rapid as possible,
given the constraints of the ﬁlter order and the need to maintain numerical
stability.
The leading example in econometric analysis of a ﬁlter whose parameters
are determined by manipulation is the so-called Hodrick–Prescott ﬁlter. This
has only a single adjustable parameter. Another example is provided by the
digitalversion of the Butterworth ﬁl ter, the use of which has been advocated
by the present author in a recent paper, (see Pollock [10]). The lowpass Butter-
worth ﬁlter, which is used in extracting the trend, can be speciﬁed by choosing
a cut-oﬀ frequency, which marks the midpoint of the transition from the pass-
band to the stopband, and by choosing the ﬁlter order, which governs the rate
of transition—with the rate increasing as the ﬁlter order is increased.
7D.S.G. POLLOCK: Signal Extraction
The analogue version of the Butterworth ﬁlter is a time-honoured device
in the ﬁeld of electrical engineering; and, in the experience of the author, it
has proved diﬃcult to devise a discrete-time Wiener–Kolmogorov ﬁlter with a
performance superior to that of the digitalButterworth ﬁl ter.
References
[1] De Jong, P., (1991), The Diﬀuse Kalman Filter, The Annals of Statistics,
19, 1073–1083.
[2] Harvey, A.C., (1989), Forecasting, Structural Time Series Models and the
Kalman Filter, Cambridge University Press, Cambridge.
[3] Harvey, A.C., and P.H. Todd, (1983), Forecasting Economic Time Series
with Structuraland Box–Jenkins Model s: A Case Study, Journal of Busi-
ness and Economic Forecasting, 1, 299–307.
[4] Hillmer, S.C., and G.C. Tiao, (1982), An ARIMA-Model-Based Approach
to SeasonalAdjustment, Journal of the American Statistical Association,
77, 63–70.
[5] Kolmogorov, A.N., (1941), Interpolation and Extrapolation, Bulletin de
l’academie des sciences de U.S.S.R., Ser. Math., 5, 3–14.
[6] Koopman, S.J., A.C. Harvey, J.A. Doornick and N. Shephard, (1995),
STAMP 5.0: Structural Time Series Analyser Modeller and Predictor—
The Manual, Chapman and Hall, London.
[7] Maravall, A., and D.A. Pierce, (1987), A Prototypical Seasonal Adjust-
ment Model, Journal of Time Series Analysis, 8, 177–193.
[8] Pierce, D.A., (1979), SignalExtraction in Nonstationary Time Series, The
Annals of Statistics, 6, 1303–1320.
[9] Pollock, D.S.G., (1999), A Handbook of Time-Series Analysis, Signal Pro-
cessingand Dynamics, The Academic Press, London.
[10] Pollock, D.S.G., (2000), Trend Estimation and Detrending via Rational
Square Wave Filters, Journal of Econometrics, 99, 317–334.
[11] Pollock, D.S.G., (2000), Filters for Short Nonstationary Sequences, Dis-
cussion Paper 00A030 of GREQAM, University of Aix–Marseille.
[12] Wiener, N., (1950), Extrapolation, Interpolation and Smoothingof Station-
ary Time Series, MIT Technology Press and John Wiley and Sons, New
York.
[13] Whittle, P., (1983), Prediction and Regulation by Linear Least-Square
Methods, Second Revised Edition, Basil Blackwell, Oxford.
8This working paper has been produced by
the Department of Economics at
Queen Mary, University of London
Copyright © 2001 Stephen Pollock
All rights reserved. 
Department of Economics 
Queen Mary, University of London
Mile End Road
London E1 4NS
Tel: +44 (0)20 7882 5096 or Fax: +44 (0)20 8983 3580
Email: j.conner@qmw.ac.uk
Website: www.econ.qmw.ac.uk/papers/wp.htm

A Handbook of Time-Series Analysis, Signal Processingand Dynamics,

A Prototypical Seasonal Adjustment Model,

An ARIMA-Model-Based Approach to SeasonalAdjustment,

Extrapolation, Interpolation and Smoothingof Stationary Time Series,

Filters for Short Nonstationary Sequences, Discussion Paper 00A030 of GREQAM,

Forecasting Economic Time Series with Structuraland Box–Jenkins Model s: A Case Study,

Forecasting, Structural Time Series Models and the Kalman Filter,

Interpolation and Extrapolation, Bulletin de l’academie des sciences de U.S.S.R.,

Prediction and Regulation by Linear Least-Square Methods, Second Revised Edition, Basil Blackwell, Oxford. 8This working paper has been produced by the

Signal Extraction, Maximum Likelihood Estimation and the Start-up Problem

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics