Monte Carlo evidence is provided as to the efficiency of the impulse saturation estimator in a location-scale model with heavy-tailed distributions. Comparisons show that the IS estimator is always more efficient than the OLS and can even outperform the Method of Moments estimator in some instances.nonnormality; impulse saturation; robust estimation

Carlos Santos

English

Research Papers in Economics

 
 
 
 
 
 
 
 
 
 
DOCUMENTOS DE TRABALHO 
 
WORKING PAPERS 
ECONOMIA 
 
ECONOMICS 
 
 
 
 
 
 
 
 
 
 
Nº 05/2008 
 
 
 
A NOTE ON THE MONTE CARLO ASSESSMENT OF 
IMPULSE SATURATION WITH FAT TAILED 
DISTRIBUTIONS 
 
Carlos Santos 
Universidade Católica Portuguesa (Porto) 
 
 
 
 
 A note on the Monte Carlo assessment of
Impulse Saturation with fat tailed distributions
Carlos Santos, Faculdade de Economia e Gestão,
Universidade Católica Portuguesa, and CEGE
August 8, 2008
1 Introduction
Impulse saturation (Santos, Hendry and Johansen, 2008) has become a major
development in model selection in linear regression. The authors have estab-
lished that a general-to-speciﬁc strategy is feasible to select from a set of T
candidate indicator variables, one for each observation. Such an initial model
cannot be estimated from the outset, so subset selection is used (where the
subsets are sample partitions either in halves, thirds, etc), followed by searches
across the union of the terminal models. For a split of T/2, this entails saturat-
ing half the sample and storing the signiﬁcant indicators, and then examining
the other half. Under the null hypothesis that no indicator matters, the im-
pulse saturation procedure is shown to have the correct null rejection frequencies
(NRFs) precluding overﬁtting, independently of the number of splits used for
the subsets. For individual tests conducted on each indicator at a signiﬁcance
level α, the average retention rate is αT, matching exactly the binomial resul
and showing low costs of search for low α. The asymptotic distribution of the
post-selection estimators of the mean and variance, in a location-scale model
with IID errors is derived, and extensive Monte Carlo evidence conﬁrms the
theoretical results.
A plethora of recent papers extends the earlier work of Santos et al. (2008):
Santos (2008) discusses break tests based on the impulse saturation principle;
Santos and Hendry (2008) extend the principle to develop a new test for su-
per exogeneity. Santos and Hendry (2006) and Nielsen and Johansen (2007)
extend impulse saturation to a class of dynamic models, namely those of the
autoregressive type; Doornik and Sprudsz (2007) extend the analysis to study
the possibility of having more variables than observations in wider settings than
linear regression models.
In this paper, we explore some of the connections of impulse saturation with
robust statistics (see, inter alia, Huber, 1980), namely with respect to eﬃciency
results. A location-scale model is used as a baseline to assess how the impulse
saturation estimator of the location parameter compares with the OLS, the
1Maximum Likelihood (ML) and the Mehod of Moments (MM) estimator, for
some error distributions with fat tails. Results in Santos et. al (2008) as well
as in Hendry and Santos (2005) had already suggested that impulse saturation
could work with nonnormal errors. Indeed, Monte Carlo evidence had shown low
spurious retention rates for irrelevant indicators even when the error distribution
had fat tails. Notwithstanding, in this note we are not concerned with model
selection and variables retention rates, but rather with estimation properties,
namely relative eﬃciency. In the next section, we discuss results for the Laplace
distribution, whilst in section 3 we compare results with those arising from a
t(4) error distribution. In both sections, the ultimate purpose is to confront
the variance of the impulse saturation estimator for the mean with the OLS
estimator and with the median (which is the ML estimator in the ﬁrst case,
and the MM estimator in the second). Discussion is based on Monte Carlo
evidence1. Section 4 concludes.
2 Estimation with the Laplace (Double Expo-
nential) distribution
The impulse saturation estimator of the mean, in a simple location-scale model,
of the form:
yt = µ + εt
where εt ∼IID, E[εt] = 0, and V[εt] ∈ R+,∀t, is given by,
￿ µ =
￿T1
t=1 yt1{|t1,ˆ δt|<cα} +
￿T
t=T1+1 yt1{|t2,ˆ δt|<cα}
￿T1
t=1 1{|t1,ˆ δt|<cα} +
￿T
t=T1+1 1{|t2,ˆ δt|<cα}
(1)
as derived in Santos, Hendry and Johansen (2008). This is similar to an (α/2)
− trimmed mean (see, inter alia, Stuart and Ord (1994)), where a fraction α/2
of observations is annihilated in each tail. Nonetheless, there is a fundamental
diﬀerence between the usual α/2 trimmed mean and ˜ µ in (1): the trimming
values r and s in the impulse saturation estimator are themselves parameters to
be estimated. In that sense, for a given sample of size T, ˆ r = g(y) and ˆ s = h(y),
where y is a (T × 1) column vector containing the yts. Hence, there is a selection
of the order statistics in (1), and the selection algorithm is non-linear.
The OLS estimator of µ would be the sample mean:
ˆ µ = ¯ y =
￿T
t=1 yt
T
(2)
where T is the sample size. According to the Gauss-Markov theorem, ˆ µ would
be the best linear unbiased estimator of µ (BLUE).
1All simulations were conducted using Ox (Doornik, 2001).
2Let us now suppose that
f (εt) =
1
2
e−|εt| (3)
that is, the errors are drawn independently from a Laplace (or double expo-
nential) distribution, with mean zero and variance 2. More generally, we could
write:
f (εt) =
1
2φ
e−|εt−θ|/φ (4)
as the density for a Laplace random variable, where φ > 0. In fact, given the
location-scale model above,
f (yt) =
1
2φ
e−|yt−µ|/φ (5)
The maximum likelihood estimator of µ in this case is the value that minimizes
with respect to µ
T ￿
t=1
|yt − µ|
and is given by ˇ µML = median(y1,...,yT), a result due to Keynes (1911).
For an even T, the ML estimator is the arithmetic mean of observations 1
2T
and
￿1
2T + 1
￿
. For odd T, ˇ µML = y T+1
2 . In both cases, we obtain unbiased
estimators of µ. As discussed in Johnson, Kotz and Balakrishnan (1995), the
median is not necessarily the minimum variance, linear unbiased estimator of µ.
Indeed, only for T ≥ 7 do we have V[¯ y] ≥ V[ˇ µML] (Chu and Hotelling, 1955).
This result in turn shows that the arithmetic mean can’t also be the BLUE
estimator of µ (Johnson et al. 1995).
Govindarajulu (1966) deduces the BLUE estimator as a trimmed mean.
Hence ˜ µ, the impulse saturation estimator of the mean, is closer in spirit to
the BLUE estimator.
2.1 Simulation Results for Bias and Eﬃciency
It is therefore of interest to compare ˜ µ, ˆ µOLS and ˇ µML in terms of unbiasedness
and eﬃciency. For this purpose we consider a Monte Carlo design where the
Data Generating Proccess (DGP) is given by the location-scale model, where
the errors are independent draws from a Laplace distribution with parameters
φ = 0.5 and µ = 2, so that (5) holds.
M = 10000 replications are conducted. We consider signiﬁcance levels of
α = 0.01, α = 0.025 and α = 0.05 to retain impulses, and a sample size of
T = 100.2 Table (1) reports the results. In order to proxy for the bias, say, of
estimator ˘ µ, E[˘ µ]−µ, we use the diﬀerence between the mean value of ˇ µ across
2Results for other sample sizes and DGP parameter values were also obtained. They are
not included here as they added little value. Nonetheless, all results are available from the
author on request.
3T = 100 α = 0.01 α = 0.025 α = 0.05
E[˜ µ] − µ 6.9451e-005 1.9123e-005 0.00016297
V[˜ µ] 0.0045882 0.0045130 0.0043788
E[ˆ µOLS] − µ -0.00030139
V[ˆ µOLS] 0.0050780
E[ˇ µML] − µ -8.2640e-006
V[ˇ µML] 0.0029196
Table 1: Laplace Distribution: MC bias and variance comparison, T = 100
the M = 10000 replications and the true parameter value. In order to proxy
for the estimator’s variance, V[˘ µ], we compute the variance of ˘ µ across Monte
Carlo replications.
In table (1), the Monte Carlo results for the mean bias and for the variance
of both the OLS and the ML estimators do not vary with α. Indeed, neither
the full sample mean (the OLS estimator) nor the median (the ML estimator)
depend on the choice of the signiﬁcance level used for impulse saturation. On
the other hand, the impulse saturation estimate of the location parameter varies
with α. We conclude from table (1) that V[˜ µ] decreases as α increases.
The main conclusion to draw from table (1) is that V[˜ µ] < V[ˆ µOLS]. For all
estimators and signiﬁcance levels considered, the Monte Carlo evidence suggests
the unbiasedness predicted by theory (Santos et. al, 2008). Unless one uses a
deﬁnition of a linear estimator diﬀerent from the one implied in Johnson et
al. (1995), namely precluding predetermining order statistics, one is bound to
conclude the Gauss-Markov theorem does not apply with Laplace errors.
It is also worth noticing that V[ˇ µML] is the lowest of the three presented.
Deﬁning relative eﬃciency as the inverse ratio of an estimator’s variance to the
variance of the ML estimator, and considering T = 100, we obtain 57.5% for
the OLS estimator and 64%, 65% and 67% for the impulse saturation estimator,
referring respectively to α = 0.01, α = 0.025 and α = 0.05.
More interestingly, for α = 0.01, V[˜ µ] is 90% of V[ˆ µOLS] when T = 100.
These values would reveal an even greater gap with α increasing. Then, on
average, our Monte Carlo results reveal that the impulse saturation estimator
of the mean induces an eﬃciency gain of about 10% relative to the OLS estimator
(this value is stable across sample sizes3).
In conclusion, the confrontation of mean bias and variance for the impulse
saturation and the OLS estimators of the mean, reveals that the ﬁrst is relatively
more eﬃcient than the second. They both lose to the ML estimator in terms of
relative eﬃciency, but here the ML estimator requires a considerable degree of
prior information about the DGP, namely with respect to the error distribution.
3Additional evidence is available upon request.
4T = 100 α = 0.01 α = 0.025 α = 0.05
E[˜ µ] − µ -0.0013207 -0.0019079 -0.0016083
V[˜ µ] 0.016288 0.016271 0.016253
E[ˆ µOLS] − µ -0.0012718
V[ˆ µOLS] 0.020058
E[˘ µMM] − µ -0.0010736
V[˘ µMM] 0.017708
Table 2: t(4) Distribution: MC bias and variance comparison, T = 100
3 Estimation with a t(4): simulation results on
bias and eﬃciency
In this section, the DGP is again a location-scale model of the form:
yt = µ + εt
where εt are IID and εt ∼ t(4), ∀t. Hence, E[εt] = 0 and V[εt] = 2. We consider
the impulse saturation estimator for the mean, ˜ µ, given by (1), as well as the
OLS estimator, ˆ µOLS = ¯ y, and the method of moments estimator (which is now
the sample median), ˘ µMM.
For the Monte Carlo experiments, the defaults from the previous section
apply. Table (2) reports the results.
In the same way as before, the ﬁnite-sample distribution of the OLS and
the MM estimators does not depend on the signiﬁcance level used for impulse
saturation. Therefore, their moments do not vary with α. However, the impulse
saturation estimate of µ depends on α. In fact, once again we observe that V[˜ µ]
diminishes as α increases.
Again, the impact on bias of using ˜ µ or ˆ µOLS is irrelevant, as the Monte
Carlo results suggest that both are unbiased.
The relative gain in eﬃciency from using ˜ µ instead of the OLS estimator
is greater for the case of a t(4) error distribution. Notice that, for α = 0.01,
the ratio of V[˜ µ] to V[ˆ µOLS] is 81% when T = 100. That is, on average there
is an eﬃciency gain of 19% when using the impulse saturation mean estimator
instead of the full sample mean. It is once more clear that the assessment of
the gain would be increased for higher α (although this eﬀect would be of 1 or
2 percentage points only4).
Furthermore, for a t(4), the impulse saturation estimator is relatively more ef-
ﬁcient than the sample median itself. Indeed, for α = 0.01 the inverse ratio of the
variance of the sample median to the variance of the impulse saturation estima-
tor for the mean is 92%, for a sample size of T = 100. On average, there is a gain
of about 8% in eﬃciency when using ˜ µ instead of ˘ µMM = median(y1,...,yT).
4Additional evidence not reported here but available from the corresponding author on
request.
5For this error distribution, Monte Carlo evidence suggests that:
V[˜ µ] < V[˘ µMM] < V[ˆ µOLS]
In conclusion, the impulse saturated mean estimator seems more robust than
the OLS estimator: retaining eﬃciency properties under departures from the
assumption of normality (see the diﬁnition of robustness in Peracchi, 2001).
4 Conclusion
The main conclusion to draw from this note is that the impulse saturation
estimator of the location parameter is more eﬃcient the OLS estimator for a
some classes of nonnormal error distributions. Furthermore, although it can be
less eﬃcient than the ML estimator, maximum likelihood requires a high degree
of prior information on the distribution of the unobserved errors. Therefore, it
seems advisable, at least when the residuals distribution suggests the erros might
not be normally distributed, to use impulse saturation. This result enforces,
from the point of view of eﬃciency, the good properties of this procedure: it
is safe to assume it is more eﬃcient under nonnormality, and we had already
established it did not cause over selection (Santos et. al, 2008).
An open point of debate has to do with the deﬁnition of a BLUE estimator.
In the sense that ordering is not a linear operation, OLS would remain BLUE.
This, however, is not the interpretation in Johnson et. al (1995). Notwith-
standing, we believe linearity is not such an interesting property once we have
established that the impulse saturation estimator is unbiased and more eﬃcient
than OLS under some nonnormal error distributions.
REFERENCES
Chu, J. T. and Hotelling, H. (1955) The Moments of the Sample Median,
Annals of Mathematical Statistics, 26, 593-606.
Doornik, J. A. (2001) OX, An Object Oriented Matrix Programming Lan-
guage, 4th edition, London: Timberlake Consultants Press.
Doornik, J. A. and U. Sprudsz (2007) Econometric Modelling when there
are more Variables than Observations, Nuﬃeld College, mimeo
Govindarajulu, Z. (1966) Best Linear Estimates under Symmetric Censoring
of the Parameters of a Double Exponential Population, Journal of the American
Statistical Association, 61, 248-258 (Correction: 71, 255)
Huber, P. J. (1980) Robust Statistics, New York, Wiley Inter-Science
Hendry, D. F. (2000) Econometrics: Alchemy or Science?, Oxford: Oxford
University Press
Hendry, D. F. and Reade, J. (2005) Problems with Model Averaging with
Dummy Variables, Univ. of Oxford. Department of Economics, mimeo
Hendry, D. F. and Santos, C. (2005) Regression Models with Data-Based
Indicator Variables, Oxford Bulletin of Economics and Statistics, 67, 571-595
Hendry, D. F.and Santos, C. (2008) An Automatic Test for Super Exogeneity,
Oxford University, Department of Economics, mimeo
6Johnson, N. L., Kotz, S. and Balakrishnan, N. (1995) Continuous Univariate
Distributions —2, 2nd edition, New York: John Wiley and Sons.
Nielsen, B. and S. Johsanen (2007) Saturation by Indicators in Autoregres-
sive Models, paper presented at the Conference in Honour of David F. Hendry,
Oxford
Peracchi, F. (2001) Econometrics, New York: John Wiley and Sons.
Santos, C. (2008) Impulse Saturation Break Tests, Economic Letters,98 (2),
pp. 136-143
Santos, C. and Hendry, D. F. (2006) Saturating Autoregressive Models, No-
tas Económicas, 24 , pp.8-19
Santos, C., Hendry, D. F. and S. Johansen (2008) Automatic Selection of
Indicators in Fully Saturated Regressions, Computational Statistics, 23(2), 317-
335
Stuart, A. and Ord, K. (1994) Kendall’s Advanced Theory of Statistics, Vol-
ume 1: Distribution Theory, 6th edition, London: Arnold.
7

Econometric Modelling when there are more Variables than Observations, Nuﬃeld College, mimeo

Impulse Saturation Break Tests,

OX, An Object Oriented Matrix Programming Language, 4th edition,

Saturation by Indicators in Autoregressive Models, paper presented at the Conference in Honour of

A note on the Monte Carlo assessment of Impulse Saturation with fat tailed distribution

http://www.porto.ucp.pt/feg/repec/WP/052008%20-%20Santos%20-%20A%20note%20on%20the%20Monte%20Carlo%20assessment.pdf

A note on the Monte Carlo assessment of Impulse Saturation with fat tailed distribution

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics