In this note a Monte Carlo approach is suggested to determine critical values for diagnostic tests of Value-at-Risk models that rely on binary random variables. Monte Carlo testing offers exact significance levels in finite samples. Conditional on exact critical values the dynamic quantile test suggested by Engle and Manganelli (2004) turns out more powerful than a recently proposed Portmanteau type test (Hurlin and Tokpavi 2006). --Value-at-Risk,Monte Carlo test

Herwartz, Helmut

English

Research Papers in Economics

econstor
www.econstor.eu
Der Open-Access-Publikationsserver der ZBW – Leibniz-Informationszentrum Wirtschaft
The Open Access Publication Server of the ZBW – Leibniz Information Centre for Economics
Nutzungsbedingungen:
Die ZBW räumt Ihnen als Nutzerin/Nutzer das unentgeltliche,
räumlich unbeschränkte und zeitlich auf die Dauer des Schutzrechts
beschränkte einfache Recht ein, das ausgewählte Werk im Rahmen
der unter
→  http://www.econstor.eu/dspace/Nutzungsbedingungen
nachzulesenden vollständigen Nutzungsbedingungen zu
vervielfältigen, mit denen die Nutzerin/der Nutzer sich durch die
erste Nutzung einverstanden erklärt.
Terms of use:
The ZBW grants you, the user, the non-exclusive right to use
the selected work free of charge, territorially unrestricted and
within the time limit of the term of the property rights according
to the terms specified at
→  http://www.econstor.eu/dspace/Nutzungsbedingungen
By the first use of the selected work the user agrees and
declares to comply with these terms of use.
zbw
Leibniz-Informationszentrum Wirtschaft
Leibniz Information Centre for Economics
Herwartz, Helmut
Working Paper
Exact inference in diagnosing value-at-
risk estimates: A Monte Carlo device
Economics working paper / Christian-Albrechts-Universität Kiel, Department of Economics,
No. 2008,16
Provided in cooperation with:
Christian-Albrechts-Universität Kiel (CAU)
Suggested citation: Herwartz, Helmut (2008) : Exact inference in diagnosing value-at-risk
estimates: A Monte Carlo device, Economics working paper / Christian-Albrechts-Universität
Kiel, Department of Economics, No. 2008,16, http://hdl.handle.net/10419/27671 
 
 
 
Exact inference in diagnosing  
value-at-risk estimates – A Monte Carlo 
device 
 
by Helmut Herwartz 
No 2008-16 
Economics Working Paper Exact inference in diagnosing value-at-risk estimates - A Monte
Carlo device
Helmut Herwartz¤
September 2, 2008
Abstract
In this note a Monte Carlo approach is suggested to determine critical values for diagnostic
tests of Value-at-Risk models that rely on binary random variables. Monte Carlo testing
o®ers exact signi¯cance levels in ¯nite samples. Conditional on exact critical values the
dynamic quantile test suggested by Engle and Manganelli (2004) turns out more powerful
than a recently proposed Portmanteau type test (Hurlin and Tokpavi 2006).
Keywords: Value-at-Risk, Monte Carlo test.
JEL Classi¯cation: C22, C52, G28.
1 Introduction
Value-at-risk (VaR) is a widely used measure of portfolio risk. Formally, suppose that fytg
T
t=1
is a process of speculative returns. Then, conditional on the information set available in time
t ¡ 1, ­t¡1, the value-at-risk with coverage ®, denoted VaRt(®), is the quantile such that
Prob[yt < ¡VaRt(®)j­t¡1] = ®: (1)
The computation of VaRt(®) is challenging since the `true' conditional distribution of returns
is unknown. In the light of a plentitude of alternative approaches to VaR determination, model
diagnosis is of essential importance in applied ¯nance.
The remainder of this short note is organized as follows. Two particular tools for VaR
diagnosis (Engle and Manganelli 2004, Hurlin and Tokpavi 2006) are sketched in the next Section.
In light of their ¯nite sample size biases a Monte Carlo approach to testing VaR accuracy is
adopted in Section 3. Concluding remarks are provided in Section 4.
¤Institut fÄ ur Statistik und Ä Okonometrie, Christian-Albrechts-UniversitÄ at zu Kiel, Germany,
Herwartz@stat-econ.uni-kiel.de2 VaR diagnosis
Starting from the de¯nition of VaRt(®) in (1) so-called hit processes formalize the event of a
conditional return shortfall. The centered hit process is
±®
t = Hitt(®) ´ I(yt < ¡VaRt(®)j­t¡1) ¡ ®; (2)
where I() is an indicator function. For diagnosing (condtional) VaRt(®) estimates it is important
to verify that no piece of information contained in ­t¡1 carries explanatory content for the mean
zero process ±®
t . In particular, ±®
t¡1;±®
t¡2 ::: and lagged hits observed at other VaR nominal
coverages a 6= ® are part of ­t¡1. For considering 'cross coverage' dynamics a hit vector is
de¯ned as ±t = ( ±
®1
t ;±
®2
t ;:::;±®m
t )0. In the following three competing statistics for VaR diagnosis
are listed that rely on the centered hit process ±®
t or its vector valued counterpart ±t.
1. Binary regression (I): Engle and Manganelli (2004) suggest the dynamic quantile test
building upon a regression design for binary variables. Conditional on presample values
this regression reads as
±®
t = ¯0 +
p X
i=1
¯i±®
t¡i + ut; (3)
= x®0
t¡1¯ + ut; t = 1;:::;T;
where x®
t¡1 = (1;±®
t¡1;±®
t¡2;:::;±®
t¡p)0. For backtesting the conventional Wald statistic,
W® =
^ ¯0(
P
t¡1 x®
t¡1x®0
t¡1)^ ¯
®(1 ¡ ®)
d ! Â2(p + 1); (4)
is used, where p + 1 is the row dimension of x®
t¡1 and ^ ¯ contains OLS estimates from (3).
The asymptotic Â2(p + 1)-distribution holds if the VaR model is well speci¯ed (H0).
2. Binary regression (II): Analogous to (3), diagnosing VaRt(®) may also exploit lagged hits
measured for VaRt(a); a 6= ®. Relying on lag one hits, for instance, obtains
±
(®)
t = ¯0 +
p X
i=2
¯i±
(®)
t¡i +
m X
a=1
°a±
(a)
t¡1 + ut; (5)
= x0
t¡1¯ + ut;
with a rede¯ned vector of explanatory variables xt¡1 and ¯ = (¯0;¯2;:::;¯p;°1;:::;°m).
To test H0 : ¯ = 0 the Wald statistic in (4) applies with a Â2(p+m) asymptotic distribution
under H0.
3. A multivariate portmanteau statistic: Although the regression in (5) takes 'cross depen-
dence' of VaR violations into account, the test is speci¯c for nominal level ®. For the
vector hit process ±t joint accuracy of VaR estimates implies
Cov(±t;±t¡j) =
(
C0 for j = 0
0 else.
(6)
1Covariance and correlation estimates are, respectively,
^ Cj =
1
T
T X
t=j+1
±t±0
t¡j; ^ Rj = D1=2 ^ CjD1=2 and D = Im ¯ ^ C0;
where Im is the m-dimensional identity matrix and '¯' signi¯es element-by-element mul-
tiplication. Hurlin and Tokpavi (2006) propose a portmanteau statistic with asymptotic
distribution applying under the null hypothesis of a well speci¯ed VaR model
Qm(J) = T
J X
j=1
³
vec ^ Rj
´0 ³
^ R¡1
0 ­ ^ R¡1
0
´³
vec ^ Rj
´
d ! Â2(Jm2): (7)
Both approaches the dynamic quantile regression, in (3) or (5) and the portmanteau statistic
in (7) adopt (auto)regression tools that originated in the framework of modelling continous
random variables. Since hit statistics de¯ned in (2) are binary, the postulated asymptotic
distributions may only hold in very large samples. Thus, with access to ¯nite dimensional
sample information, an analyst runs the risk of biased inferential conclusions when founding
test decisions on standard, Â2(²), critical values.
3 Monte Carlo test
The test statistics (4) and (7) build upon processes that can be very easily constructed by
simulation under the null hypothesis of a correct (dynamic) VaR speci¯cation. Thus, Monte
Carlo critical values (Dufour 2006) o®er exact signi¯cance levels for VaR diagnosis. This section
illustrates the merits of Monte Carlo based VaR diagnosis. Size violations invoked by inference
based on asymptotic critical values are illustrated. Moreover, alternative diagnostic tools are
compared in terms of power estimates implied by Monte Carlo critical values.
3.1 Design
To imitate a well speci¯ed VaR model, ±®
t processes are determined by means of iid Gaussian
variates. Let ©(²) denote the Gaussian distribution function. Then,
±®
t = I(»t < ©¡1(®)) with »t » iidN(0;1); t = 1;:::;T: (8)
Vector hit processes are obtained by stacking ±®
t for ® 2 f:005;:01;:015;:02;:025;:03;:035g (m =
7). To investigate the performance of VaR diagnostics in case of misspeci¯cation, »t in (8) is
generated as a ¯rst order autoregressive process, »t = 0:1»t¡1 + ³t; ³t » iidN(0;0:99). The
adjusted variance for drawing ³t ensures that the unconditional variance of »t is one. To specify
the regression models in (3) or (5) the lag order is p = 5. For the portmanteau statistic in
(7) order parameters J = 5;15 are considered alternatively. Thus, the asymptotic distributions
of alternative diagnostic tools are Â2(q); q = 6;12;245;735. Simulated sample sizes vary from
2T W®; Â2(p + 1) W®; Â2(p + m) Qm(J)
.005 .01 .025 .005 .01 .025 J = 5 15
Empirical size (Â2(²) 5% critical values)
2 .223 .101 .062 .201 .149 .102 .265 .250
4 .118 .079 .059 .147 .125 .084 .224 .215
8 .094 .074 .054 .134 .099 .069 .170 .167
20 .083 .063 .049 .094 .077 .059 .113 .116
50 .067 .051 .049 .075 .063 .056 .080 .078
Empirical power (Monte Carlo 5% critical values)
2 .068 .080 .098 .100 .120 .143 .106 .085
4 .076 .082 .122 .121 .150 .190 .134 .090
8 .085 .109 .187 .176 .200 .280 .200 .126
20 .112 .174 .397 .259 .343 .491 .324 .176
50 .165 .327 .766 .434 .606 .823 .580 .320
Table 1: Top panel: Empirical size estimates for alternative asymptotic tests diagnosing VaRt(®)
with 5% nominal signi¯cance level. Bottom panel: Power estimates implied by applying 5%
Monte Carlo critical values. Considered sample sizes are T ¢ 1000.
conventional magnitudes to extreme time dimensions, i.e. T 2 f2000;4000;8000;20000;50000g.
10000 Monte Carlo replications are used for all experiments.
An often raised caveat of Monte Carlo results is their dependence on the data generating
process used for the simulation. For simulating VaR hits under the null hypothesis of a well
speci¯ed risk model, it is worthwhile to point out that the design in (8) matches the null hypoth-
esis in a one-to-one manner. Employing competing, well speci¯ed VaR models for imitation of
the null hypothesis would not deliver systematically di®erent empirical size features. The data
generating mechanism employed to imitate poor VaR speci¯cations lacks parametric rigor. It is
interesting, though, to investigate how alternative tests cope with a 'nonparametric' alternative.
3.2 Results
Table 1 provides inferential results for diagnosing VaR speci¯cations with 5% signi¯cance. The
top panel documents rejection frequencies of alternative tests based on the ¯rst order asymptotic
approximations. Rejection frequencies under misspeci¯cation of the VaR model as implied by
Monte Carlo critical values are displayed below. Performing analogous experiments for nominal
signi¯cance levels 1% or 10% yields qualitatively identical results which are not displayed for
space considerations.
Marked size distortions feature all diagnostic tools for conventional sample sizes (T =
2000;4000) if the nominal coverage level ® is small, 1% or less. Throughout, overrejections
3under the null hypothesis are higher for the Portmanteau statistic Qm(J) as for the Wald statis-
tics W®. For instance, empirical rejection frequencies for diagnosing VaRt(:01) estimates exceed
twice the nominal level for both regression models (3) and (5) conditional on T = 2000. Testing
joint accuracy of the 7-dimensional hit process (T = 2000) obtains empirical size estimates of
26.5% (J = 5) and 25.0% (J = 15), respectively.
Convergence to the asymptotic pivotal distributions is e®ectively very slow. The speed
of convergence of Wald test rejection frequencies depends on the investigated nominal VaR
coverage. For instance, employing test regression (3) for ® = :01 and ® = :025 the empirical
signi¯cance level comes close to the nominal level for T = 50000 (with rejection frequency 5.1%)
and T = 20000 (with rejection frequency 4.9%), respectively. With respect to the augmented test
regression (5) signi¯cant oversizing features VaRt(®) diagnosis for ® = :005;:01 even conditional
on an 'extreme' sample size T = 50000.
Using Monte Carlo 5% critical values to diagnose VaR misspeci¯cation uncovers marked
power di®erentials. Irrespective of the considered sample size Qm(5) turns out more powerful
in comparison with Qm(15). The former is less powerful than a Wald statistic based on the
regression (5) and testing VaRt(®) speci¯cations with ® ¸ 0:1. Diagnosing VaRt(:005) appears
most challenging in terms of power. Obviously, the power of the Wald test in (4) is improved
by conditioning on lagged hit processes measured for nominal VaR coverage a 6= ®.
4 Conclusion
In diagnosing conditional VaR estimates the dynamic quantile test (Engle and Manganelli 2004)
and a Portmanteau approach (Hurlin and Tokpavi 2006) are shown to su®er from marked size
distortions prevailing in even rather large (¯nite) samples. With regard to the dynamic quantile
test oversizing is particularly likely if low VaR coverage levels are subjected to diagnostic testing.
Monte Carlo inference o®ers exact empirical signi¯cance levels. In terms of power the dynamic
quantile regression augmented with lagged hits measured for distinct VaR nominal coverages is
most e®ective.
References
[1] Dufour, J.M. (2006) 'Monte Carlo Tests with Nuisance Parameters: A General Approach to
Finite Sample Inference and Nonstandard Asymptotics in Econometrics, Journal of Econo-
metrics, 133, 443-477.
[2] Engle, R.F., S. Manganelli (2004), 'CAViaR: Conditional Autoregressive Value-at-Risk by
Regression Quantiles, Journal of Business and Economic Statistics, 22, 367-381.
[3] Hurlin, C., S, Tokpavi (2006), 'Backtesting VaR Accuracy: A New Simple Test', Working
Paper, LEO, University of Orl¶ eans.
4

Backtesting VaR Accuracy: A New Simple Test', Working Paper, LEO,

CAViaR: Conditional Autoregressive Value-at-Risk by Regression Quantiles,

Monte Carlo Tests with Nuisance Parameters: A General Approach to Finite Sample Inference and Nonstandard Asymptotics in Econometrics,

Exact inference in diagnosing value-at-risk estimates: A Monte Carlo device

http://econstor.eu/bitstream/10419/27671/1/EWP-2008-16.pdf

Exact inference in diagnosing value-at-risk estimates: A Monte Carlo device

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics