We study in this paper the consequences of using the Mean Absolute Percentage
Error (MAPE) as a measure of quality for regression models. We show that
finding the best model under the MAPE is equivalent to doing weighted Mean
Absolute Error (MAE) regression. We show that universal consistency of
Empirical Risk Minimization remains possible using the MAPE instead of the MAE.Comment: European Symposium on Artificial Neural Networks, Computational
  Intelligence and Machine Learning (ESANN), Apr 2015, Bruges, Belgium. 2015,
  Proceedings of the 23-th European Symposium on Artificial Neural Networks,
  Computational Intelligence and Machine Learning (ESANN 2015

De Myttenaere, Arnaud

Golden, Boris

Grand, Bénédicte Le

Rossi, Fabrice

English

arXiv

International audienceWe study in this paper the consequences of using the Mean Absolute Percentage Error (MAPE) as a measure of quality for regression models. We show that finding the best model under the MAPE is equivalent to doing weighted Mean Absolute Error (MAE) regression. We show that universal consistency of Empirical Risk Minimization remains possible using the MAPE instead of the MAE

Le Grand, Bénédicte

Archive Ouverte en Sciences de l'Information et de la Communication

HAL Id: hal-01162980
https://hal.archives-ouvertes.fr/hal-01162980
Submitted on 11 Jun 2015
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Distributed under a Creative Commons Attribution| 4.0 International License
Using the Mean Absolute Percentage Error for
Regression Models
Arnaud de Myttenaere, Boris Golden, Bénédicte Le Grand, Fabrice Rossi
To cite this version:
Arnaud de Myttenaere, Boris Golden, Bénédicte Le Grand, Fabrice Rossi. Using the Mean Absolute
Percentage Error for Regression Models. European Symposium on Artificial Neural Networks, Com-
putational Intelligence and Machine Learning (ESANN), Apr 2015, Bruges, Belgium. ￿hal-01162980￿
Using the Mean Absolute Percentage Error for
Regression Models
Arnaud de Myttenaere1,2, Boris Golden1, Bénédicte Le Grand3 & Fabrice Rossi2
1 - Viadeo
30 rue de la Victoire, 75009 Paris - France
2 - Université Paris 1 Panthéon - Sorbonne - SAMM EA 4534
90 rue de Tolbiac, 75013 Paris - France
3 - Université Paris 1 Panthéon - Sorbonne - Centre de Recherche en Informatique
90 rue de Tolbiac, 75013 Paris - France
Abstract. We study in this paper the consequences of using the Mean
Absolute Percentage Error (MAPE) as a measure of quality for regres-
sion models. We show that finding the best model under the MAPE is
equivalent to doing weighted Mean Absolute Error (MAE) regression. We
show that universal consistency of Empirical Risk Minimization remains
possible using the MAPE instead of the MAE.
1 Introduction
We study in this paper the classical regression setting in which we assume given
a random pair Z = (X,Y ) with values in X × R, where X is a metric space.
The goal is to learn a mapping g from X to R such that g(X) ≃ Y . To judge
the quality of the regression model g, we need a quality measure. While the
traditional measure is the quadratic error, in some applications, a more useful
measure of the quality of the predictions made by a regression model is given by
the mean absolute percentage error (MAPE). For a target y and a prediction p,
the MAPE is
lMAPE(p, y) =
|p− y|
|y|
,
with the conventions that for all a 6= 0, a0 = ∞ and that
0
0 = 1. The MAPE-risk
of g is then LMAPE(g) = E(lMAPE(g(X), Y )).
We are interested in the consequences of choosing the best regression model
according to the MAPE as opposed to the Mean Absolute Error (MAE) or the
Mean Square Error (MSE), both on a practical point of view and on a theoretical
one. On a practical point of view, it seems obvious that if g is chosen so as
to minimize LMAPE(g) it will perform better according to the MAPE than a
model selected in order to minimize LMSE (and worse according to the MSE).
The practical issue is rather to determine how to perform this optimization: this
is studied in Section 3. On a theoretical point of view, it is well known (see e.g.
[4]) that consistent learning schemes can be obtained by adapting the complexity
of the model class to the data size. As the complexity of a class of models is
partially dependent on the loss function, using the MAPE instead of e.g. the
MSE has some implications that are investigated in this paper, in Section 4. The
following Section introduces the material common to both parts of the analysis.
2 General setting
We use a classical statistical learning setting as in e.g. [4]. We assume given
N independently distributed copies of Z, the training set, D = (Zi)1≤i≤N =
(Xi, Yi)1≤i≤N . Given a loss function l from R
2 to R+ ∪ {∞}, we define the risk
of a predictor g, a (measurable) function from X to R as the expected loss, that
is Ll(g) = E(l(g(X), Y )). The empirical risk is the empirical mean of the loss
computed on the training set, that is:
L̂l(g)N =
1
N
N∑
i=1
l(g(Xi), Yi). (1)
In addition to lMAPE defined in the Introduction, we use lMAE(p, y) = |p − y|
and lMSE(p, y) = (p− y)2.
3 Practical issues
3.1 Optimization
On a practical point of view, the problem is to minimize L̂MAPE(g)N over a
class of models GN , that is to solve
1
ĝMAPE,N = arg min
g∈GN
1
N
N∑
i=1
|g(Xi)− Yi|
|Yi|
.
Optimization wise, this is simply a particular case of median regression (which is
in turn a particular case of quantile regression). Indeed, the quotient by 1|Yi| can
be seen as a fixed weight and therefore, any quantile regression implementation
that supports instance weights can be use to find the optimal model2. Notice
that when GN corresponds to linear models, the optimization problem is a simple
linear programming problem that can be solved by e.g. interior point methods [2].
3.2 An example of typical results
We verified on a toy example (the car data set from [3]) the effects of optimizing
the MAPE, the MAE and the MSE for a simple linear model: the goal is to
predict the distance taken to stop from the speed of the car just before breaking.
There are only 50 observations, the goal being here to illustrate the effects of
changing the loss function. The results on the training set3 are summarized in
Table 1. As expected, optimizing for a particular loss function leads to the best
1We are considering here the empirical risk minimization, but we could of course include a
regularization term. That would not modify the key point which is the use of the MAPE.
2This is the case of quantreg R package [5], among others.
3Notice that the goal here is to verify the effects of optimizing with respect to different
types of loss function, not to claim that one loss function is better than another, something
that would be meaningless. We report therefore the empirical risk, knowing that it is an
underestimation of the real risk for all loss functions.
empirical model as measured via the same risk (or a related one). In practice this
allowed one of us to win a recent datascience.net challenge about electricity
consumption prediction4 which was using the MAPE as the evaluation metric.
Loss function RMSE NMAE MAPE
MSE 0.585 0.322 0.384
MAE 0.601 0.313 0.330
MAPE 0.700 0.343 0.303
Table 1: Empirical risks of the best linear models obtained with the three loss
functions. In order to ease the comparisons between the values, we report the
Normalized Root MSE, that is the square root of the MSE divided by the stan-
dard deviation of the target variable, as well as the Normalized MAE, that is
the MAE devided by the median of the target variable.
4 Theoretical issues
On a theoretical point of view, we are interested in the consistency of standard
learning strategies when the loss function is the MAPE. More precisely, for a
loss function l, we define L∗l = infg Ll(g), where the infimum is taken over all
measurable functions from X to R. We also denote L∗l,G = infg∈G Ll(g) where G
is a class of models. Then a learning algorithm, that is a function which maps
the training set D = (Xi, Yi)1≤i≤N to a model ĝN , is strongly consistent if Ll(ĝN )
converges almost surely to L∗l . We are interested specifically by the Empirical
Risk Minimization (ERM) algorithm, that is by ĝl,N = argming∈GN L̂l(g)N .
The class of models to depend on the data size as this is mandatory to reach
consistency.
It is well known (see e.g. [4] chapter 9) that ERM consistency is related
to uniform laws of large numbers (ULLN). In particular, we need to control
quantities of the following form
P
{
sup
g∈GN
∣∣∣L̂mape(g)N − Lmape(g)
∣∣∣ > ǫ
}
. (2)
This can be done via covering numbers or via the Vapnik-Chervonenkis dimen-
sion (VC-dim) of certain classes of functions derived from GN . One might think
that general results about arbitrary loss functions can be used to handle the case
of the MAPE. This is not the case as those results generally assume a uniform
Lipschitz property of l (see Lemma 17.6 in [1], for instance) that is not fulfilled
by the MAPE.
4.1 Classes of functions
Given a class of models, GN , and a loss function l, we introduce derived classes
H(GN , l) given by
H(GN , l) = {h : X × R → R
+, h(x, y) = l(g(x), y) | g ∈ GN},
4https://datascience.net/fr/challenge/16/details
and H+(GN , l) given by
H+(GN , l) = {h : X × R× R → R
+, h(x, y, t) = It≤l(g(x),y) | g ∈ GN}.
When this is obvious from the context, we abbreviate the notations into e.g.
HN,MAPE for l = lMAPE and for the GN under study.
4.2 Covering numbers
4.2.1 Supremum covering numbers
Let ǫ > 0, a size p supremum ǫ-cover of a class of positive functions F from an
arbitrary set Z to R+ is a finite collection f1, . . . , fp of F such that for all f ∈ F
min
1≤i≤p
sup
z∈Z
|f(z)− fi(z)| < ǫ.
Then the supremum ǫ-covering number of F , N∞(ǫ, F ), is the size of the smallest
supremum ǫ-cover of F . If such a cover does not exists, the covering number is
∞. While controlling supremum covering numbers of H(GN , l) leads easily to
consistency via a uniform law of large numbers (see e.g. Lemma 9.1 in [4]), they
cannot be used with the MAPE without additional assumptions. Indeed, let h1
and h2 be two functions from HN,MAPE, generated by g1 and g2 in GN . Then
‖h1 − h2‖∞ = sup
(x,y)∈X×R
||g1(x) − y| − |g2(x) − y||
|y|
.
In general, this quantity will be unbounded as we cannot control the behavior
of g(x) around y = 0 (indeed, in the supremum, x and y are independent and
thus unless GN is very restricted there is always x and g1 and g2 such that
g1(x) 6= g2(x) 6= 0). Thus we have to assume that there is λ > 0 such that
|Y | ≥ λ. This is not needed when using more traditional loss functions such as
the MSE or the MAE. Then we have
N∞(ǫ,H(GN , lMAPE)) ≤ N∞(λǫ,H(GN , lMAE)).
4.2.2 Lp covering numbers
Lp covering numbers are similar to supremum covering numbers but are based
on a different metric on the class of functions F and are data dependent. Given
a data set D, we define
‖f1 − f2‖p,D =
(
1
N
N∑
i=1
|f1(Zi)− f2(Zi)|
p
) 1
p
,
and derive from this the associated notion of ǫ-cover and of covering number.
It’s then easy to show that
Np(ǫ,H(GN , lMAPE), D) ≤ Np(ǫ min
1≤i≤N
|Yi|, H(GN , lMAE), D).
4.3 Uniform law of large numbers
In order to get a ULLN from a covering number of a class F , one needs a uniform
bound on F . For instance, Theorem 9.1 from [4] assumes that there is a value BF
such that for all f ∈ F and all z ∈ Z, f(z) ∈ [0, BF ]. With classical loss functions
such as MAE and MSE, this is achieved via upper bounding assumptions on
both GN and on |Y |. In the MAPE case, the bound on GN is needed but
the upper bound on |Y | is replaced by the lower bound already needed. Let
us assume indeed that for all g ∈ GN , ‖g‖∞ ≤ BGN . Then if |Y | ≤ BY ,
we have BH(GN ,lMAE) = BGN + BY := BN,MAE , while if |Y | ≥ λ, we have
BH(GN ,lMAPE) = 1 +
BGN
λ
:= BN,MAPE.
Theorem 9.1 from [4] gives then (with BN,l = BH(GN ,l))
P
{
sup
g∈GN
∣∣∣L̂l(g)N − Ll(g)
∣∣∣ > ǫ
}
≤ 8E
(
Np
( ǫ
8
, H(GN , l), D
))
e
− Nǫ
2
128B2
N,l . (3)
The expressions of the two bounds above show that BY and λ play similar
roles on the exponential decrease of the right hand side bound. Loosening the
condition on Y (i.e., taking a large BY or a small λ) slows down the exponential
decrease.
It might seem from the results on the covering numbers that the MAPE
suffers more from the bound needed on Y than e.g. the MAE. This is not the
case as bounds hypothesis on F are also needed to get finite covering numbers
(see the following section for an example). Then we can consider that the lower
bound on |Y | plays an equivalent role for the MAPE to the one played by the
upper bound on |Y | for the MAE/MSE.
4.4 VC-dimension
A convenient way to bound covering numbers is to use VC-dimension. Inter-
estingly replacing the MAE by the MAPE cannot increase the VC-dim of the
relevant class of functions.
Let us indeed consider a set of k points shattered byH+(GN , lMAPE), (v1, . . . , vk),
vj = (xj , yj, tj). Then for each θ ∈ {0, 1}k, there is hθ ∈ HN,MAPE such
that ∀j, It≤hθ(x,y)(xj , yj , tj) = θj . Each hθ corresponds to a gθ ∈ GN and
t ≤ hθ(x, y) ⇔ t ≤
|gθ(x)−y|
|y| . Then the set of k points defined by wj = (zj , |yj |tj)
is shattered by H+(GN , lMAE) because the h
′
θ associated in HN,MAE to the gθ
are such that ∀j, It≤h′
θ
(x,y)(xj , yj, |yj |tj) = θj . Therefore
VMAPE := V Cdim(H
+(GN , lMAPE)) ≤ V Cdim(H
+(GN , lMAE)) := VMAE .
Using theorem 9.4 from [4], we can bound the Lp covering number with a VC-
dim based value. If Vl = V Cdim(H
+(GN , l)) ≥ 2, p ≥ 1, and 0 < ǫ <
BN,l
4 ,
then
Np(ǫ,H(GN , l), D) ≤ 3
(
2eBpN,l
ǫp
log
3eBpN,l
ǫp
)Vl
. (4)
When this bound is plugged into equation (3), it shows the symmetry between
BY and λ as both appears in the relevant BN,l.
4.5 Consistency
Mimicking Theorem 10.1 from [4], we can prove a generic consistency result for
MAPE ERM learning. Assume given a series of classes of models, (Gn)n≥1 such
that
⋃
n≥1 Gn is dense in the set of measurable functions from R
p to R according
to the L1(µ) metric for any probability measure µ. Assume in addition that each
Gn leads to a finite VC-dim Vn = V Cdim(H
+(Gn, lMAPE) and that each Gn is
uniformly bounded by BGn . Notice that those two conditions are compatible
with the density condition only if limn→∞ vn = ∞ and limn→∞ BGn = ∞.
Assume finally that (X,Y ) is such as |Y | ≥ λ (almost surely) and that
limn→∞
vnB
2
Gn
logBGn
n
= 0, then LMAPE(ĝlMAPE ,n) converges almost surely to
L∗MAPE , which shows the consistency of the ERM estimator for the MAPE.
The proof is based on the classical technique of exponential bounding. Plu-
gin equation (4) into equation (3) gives a bound on the deviation between the
empirical mean and the expectation of
K(n, ǫ) = 24
(
16eBn
ǫ
log
24eBn
ǫ
)vn
e
− nǫ
2
128B2n ,
with Bn = 1+
BGn
λ
. Then it is easy to check that the conditions above guarantee
that
∑
n≥1 K(n, ǫ) < ∞ for all ǫ > 0. This is sufficient to show almost sure
convergence of LMAPE(ĝlMAPE ,n) − L
∗
MAPE,Gn
to 0. The conclusion follows
from the density hypothesis.
5 Conclusion
We have shown that learning under the Mean Absolute Percentage Error is
feasible both on a practical point of view and on a theoretical one. In application
contexts where this error measure is adapted (in general when the target variable
is positive by design and remains quite far away from zero, e.g. in price prediction
for expensive goods), there is therefore no reason to use the Mean Square Error
(or another measure) as a proxy for the MAPE. An open theoretical question
is whether the symmetry between the upper bound on |Y | for MSE/MAE and
the lower bound on |Y | for the MAPE is strong enough to allow results such as
Theorem 10.3 in [4] in which a truncated estimator is used to lift the bounded
hypothesis on |Y |.
References
[1] M. Anthony and P. L. Bartlett. Neural Network Learning: Theoretical Foundations. Cam-
bridge University Press, 1999.
[2] S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
[3] M. Ezekiel. Methods of Correlation Analysis. Wiley, 1930.
[4] L. Györfi, M. Kohler, A. Krzyżak, and H. Walk. A Distribution-Free Theory of Nonpara-
metric Regression. Springer, New York, 2002.
[5] R. Koenker. quantreg: Quantile Regression, 2013. R package version 5.05.


Using the Mean Absolute Percentage Error for Regression Models

de Myttenaere, Arnaud

HAL-Paris1

Using the Mean Absolute Percentage Error forRegression ModelsArnaud De Myttenaere, Boris Golden, Be´ne´dicte Le Grand, Fabrice RossiTo cite this version:Arnaud De Myttenaere, Boris Golden, Be´ne´dicte Le Grand, Fabrice Rossi. Using the MeanAbsolute Percentage Error for Regression Models. European Symposium on Artificial NeuralNetworks, Computational Intelligence and Machine Learning (ESANN), Apr 2015, Bruges,Belgium. 2015, Proceedings of the 23-th European Symposium on Artificial Neural Networks,Computational Intelligence and Machine Learning (ESANN 2015). <hal-01162980>HAL Id: hal-01162980https://hal.archives-ouvertes.fr/hal-01162980Submitted on 11 Jun 2015HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.Distributed under a Creative Commons Attribution 4.0 International LicenseUsing the Mean Absolute Percentage Error forRegression ModelsArnaud de Myttenaere1,2, Boris Golden1, Be´ne´dicte Le Grand3 & Fabrice Rossi21 - Viadeo30 rue de la Victoire, 75009 Paris - France2 - Universite´ Paris 1 Panthe´on - Sorbonne - SAMM EA 453490 rue de Tolbiac, 75013 Paris - France3 - Universite´ Paris 1 Panthe´on - Sorbonne - Centre de Recherche en Informatique90 rue de Tolbiac, 75013 Paris - FranceAbstract. We study in this paper the consequences of using the MeanAbsolute Percentage Error (MAPE) as a measure of quality for regres-sion models. We show that finding the best model under the MAPE isequivalent to doing weighted Mean Absolute Error (MAE) regression. Weshow that universal consistency of Empirical Risk Minimization remainspossible using the MAPE instead of the MAE.1 IntroductionWe study in this paper the classical regression setting in which we assume givena random pair Z = (X,Y ) with values in X × R, where X is a metric space.The goal is to learn a mapping g from X to R such that g(X) ' Y . To judgethe quality of the regression model g, we need a quality measure. While thetraditional measure is the quadratic error, in some applications, a more usefulmeasure of the quality of the predictions made by a regression model is given bythe mean absolute percentage error (MAPE). For a target y and a prediction p,the MAPE islMAPE(p, y) =|p− y||y|,with the conventions that for all a 6= 0, a0 =∞ and that00 = 1. The MAPE-riskof g is then LMAPE(g) = E(lMAPE(g(X), Y )).We are interested in the consequences of choosing the best regression modelaccording to the MAPE as opposed to the Mean Absolute Error (MAE) or theMean Square Error (MSE), both on a practical point of view and on a theoreticalone. On a practical point of view, it seems obvious that if g is chosen so asto minimize LMAPE(g) it will perform better according to the MAPE than amodel selected in order to minimize LMSE (and worse according to the MSE).The practical issue is rather to determine how to perform this optimization: thisis studied in Section 3. On a theoretical point of view, it is well known (see e.g.[4]) that consistent learning schemes can be obtained by adapting the complexityof the model class to the data size. As the complexity of a class of models ispartially dependent on the loss function, using the MAPE instead of e.g. theMSE has some implications that are investigated in this paper, in Section 4. Thefollowing Section introduces the material common to both parts of the analysis.2 General settingWe use a classical statistical learning setting as in e.g. [4]. We assume givenN independently distributed copies of Z, the training set, D = (Zi)1≤i≤N =(Xi, Yi)1≤i≤N . Given a loss function l from R2 to R+ ∪ {∞}, we define the riskof a predictor g, a (measurable) function from X to R as the expected loss, thatis Ll(g) = E(l(g(X), Y )). The empirical risk is the empirical mean of the losscomputed on the training set, that is:L̂l(g)N =1NN∑i=1l(g(Xi), Yi). (1)In addition to lMAPE defined in the Introduction, we use lMAE(p, y) = |p − y|and lMSE(p, y) = (p− y)2.3 Practical issues3.1 OptimizationOn a practical point of view, the problem is to minimize L̂MAPE(g)N over aclass of models GN , that is to solve1ĝMAPE,N = arg ming∈GN1NN∑i=1|g(Xi)− Yi||Yi|.Optimization wise, this is simply a particular case of median regression (which isin turn a particular case of quantile regression). Indeed, the quotient by 1|Yi| canbe seen as a fixed weight and therefore, any quantile regression implementationthat supports instance weights can be use to find the optimal model2. Noticethat when GN corresponds to linear models, the optimization problem is a simplelinear programming problem that can be solved by e.g. interior point methods [2].3.2 An example of typical resultsWe verified on a toy example (the car data set from [3]) the effects of optimizingthe MAPE, the MAE and the MSE for a simple linear model: the goal is topredict the distance taken to stop from the speed of the car just before breaking.There are only 50 observations, the goal being here to illustrate the effects ofchanging the loss function. The results on the training set3 are summarized inTable 1. As expected, optimizing for a particular loss function leads to the best1We are considering here the empirical risk minimization, but we could of course include aregularization term. That would not modify the key point which is the use of the MAPE.2This is the case of quantreg R package [5], among others.3Notice that the goal here is to verify the effects of optimizing with respect to differenttypes of loss function, not to claim that one loss function is better than another, somethingthat would be meaningless. We report therefore the empirical risk, knowing that it is anunderestimation of the real risk for all loss functions.empirical model as measured via the same risk (or a related one). In practice thisallowed one of us to win a recent datascience.net challenge about electricityconsumption prediction4 which was using the MAPE as the evaluation metric.Loss function RMSE NMAE MAPEMSE 0.585 0.322 0.384MAE 0.601 0.313 0.330MAPE 0.700 0.343 0.303Table 1: Empirical risks of the best linear models obtained with the three lossfunctions. In order to ease the comparisons between the values, we report theNormalized Root MSE, that is the square root of the MSE divided by the stan-dard deviation of the target variable, as well as the Normalized MAE, that isthe MAE devided by the median of the target variable.4 Theoretical issuesOn a theoretical point of view, we are interested in the consistency of standardlearning strategies when the loss function is the MAPE. More precisely, for aloss function l, we define L∗l = infg Ll(g), where the infimum is taken over allmeasurable functions from X to R. We also denote L∗l,G = infg∈G Ll(g) where Gis a class of models. Then a learning algorithm, that is a function which mapsthe training set D = (Xi, Yi)1≤i≤N to a model ĝN , is strongly consistent if Ll(ĝN )converges almost surely to L∗l . We are interested specifically by the EmpiricalRisk Minimization (ERM) algorithm, that is by ĝl,N = argming∈GN L̂l(g)N .The class of models to depend on the data size as this is mandatory to reachconsistency.It is well known (see e.g. [4] chapter 9) that ERM consistency is relatedto uniform laws of large numbers (ULLN). In particular, we need to controlquantities of the following formP{supg∈GN∣∣∣L̂mape(g)N − Lmape(g)∣∣∣ > } . (2)This can be done via covering numbers or via the Vapnik-Chervonenkis dimen-sion (VC-dim) of certain classes of functions derived from GN . One might thinkthat general results about arbitrary loss functions can be used to handle the caseof the MAPE. This is not the case as those results generally assume a uniformLipschitz property of l (see Lemma 17.6 in [1], for instance) that is not fulfilledby the MAPE.4.1 Classes of functionsGiven a class of models, GN , and a loss function l, we introduce derived classesH(GN , l) given byH(GN , l) = {h : X × R→ R+, h(x, y) = l(g(x), y) | g ∈ GN},4https://datascience.net/fr/challenge/16/detailsand H+(GN , l) given byH+(GN , l) = {h : X × R× R→ R+, h(x, y, t) = It≤l(g(x),y) | g ∈ GN}.When this is obvious from the context, we abbreviate the notations into e.g.HN,MAPE for l = lMAPE and for the GN under study.4.2 Covering numbers4.2.1 Supremum covering numbersLet  > 0, a size p supremum -cover of a class of positive functions F from anarbitrary set Z to R+ is a finite collection f1, . . . , fp of F such that for all f ∈ Fmin1≤i≤psupz∈Z|f(z)− fi(z)| < .Then the supremum -covering number of F , N∞(, F ), is the size of the smallestsupremum -cover of F . If such a cover does not exists, the covering number is∞. While controlling supremum covering numbers of H(GN , l) leads easily toconsistency via a uniform law of large numbers (see e.g. Lemma 9.1 in [4]), theycannot be used with the MAPE without additional assumptions. Indeed, let h1and h2 be two functions from HN,MAPE, generated by g1 and g2 in GN . Then‖h1 − h2‖∞ = sup(x,y)∈X×R||g1(x) − y| − |g2(x) − y|||y|.In general, this quantity will be unbounded as we cannot control the behaviorof g(x) around y = 0 (indeed, in the supremum, x and y are independent andthus unless GN is very restricted there is always x and g1 and g2 such thatg1(x) 6= g2(x) 6= 0). Thus we have to assume that there is λ > 0 such that|Y | ≥ λ. This is not needed when using more traditional loss functions such asthe MSE or the MAE. Then we haveN∞(,H(GN , lMAPE)) ≤ N∞(λ,H(GN , lMAE)).4.2.2 Lp covering numbersLp covering numbers are similar to supremum covering numbers but are basedon a different metric on the class of functions F and are data dependent. Givena data set D, we define‖f1 − f2‖p,D =(1NN∑i=1|f1(Zi)− f2(Zi)|p) 1p,and derive from this the associated notion of -cover and of covering number.It’s then easy to show thatNp(,H(GN , lMAPE), D) ≤ Np( min1≤i≤N|Yi|, H(GN , lMAE), D).4.3 Uniform law of large numbersIn order to get a ULLN from a covering number of a class F , one needs a uniformbound on F . For instance, Theorem 9.1 from [4] assumes that there is a value BFsuch that for all f ∈ F and all z ∈ Z, f(z) ∈ [0, BF ]. With classical loss functionssuch as MAE and MSE, this is achieved via upper bounding assumptions onboth GN and on |Y |. In the MAPE case, the bound on GN is needed butthe upper bound on |Y | is replaced by the lower bound already needed. Letus assume indeed that for all g ∈ GN , ‖g‖∞ ≤ BGN . Then if |Y | ≤ BY ,we have BH(GN ,lMAE) = BGN + BY := BN,MAE , while if |Y | ≥ λ, we haveBH(GN ,lMAPE) = 1 +BGNλ:= BN,MAPE.Theorem 9.1 from [4] gives then (with BN,l = BH(GN ,l))P{supg∈GN∣∣∣L̂l(g)N − Ll(g)∣∣∣ > } ≤ 8E(Np ( 8, H(GN , l), D))e− N2128B2N,l . (3)The expressions of the two bounds above show that BY and λ play similarroles on the exponential decrease of the right hand side bound. Loosening thecondition on Y (i.e., taking a large BY or a small λ) slows down the exponentialdecrease.It might seem from the results on the covering numbers that the MAPEsuffers more from the bound needed on Y than e.g. the MAE. This is not thecase as bounds hypothesis on F are also needed to get finite covering numbers(see the following section for an example). Then we can consider that the lowerbound on |Y | plays an equivalent role for the MAPE to the one played by theupper bound on |Y | for the MAE/MSE.4.4 VC-dimensionA convenient way to bound covering numbers is to use VC-dimension. Inter-estingly replacing the MAE by the MAPE cannot increase the VC-dim of therelevant class of functions.Let us indeed consider a set of k points shattered byH+(GN , lMAPE), (v1, . . . , vk),vj = (xj , yj, tj). Then for each θ ∈ {0, 1}k, there is hθ ∈ HN,MAPE suchthat ∀j, It≤hθ(x,y)(xj , yj , tj) = θj . Each hθ corresponds to a gθ ∈ GN andt ≤ hθ(x, y)⇔ t ≤|gθ(x)−y||y| . Then the set of k points defined by wj = (zj , |yj |tj)is shattered by H+(GN , lMAE) because the h′θ associated in HN,MAE to the gθare such that ∀j, It≤h′θ(x,y)(xj , yj, |yj |tj) = θj . ThereforeVMAPE := V Cdim(H+(GN , lMAPE)) ≤ V Cdim(H+(GN , lMAE)) := VMAE .Using theorem 9.4 from [4], we can bound the Lp covering number with a VC-dim based value. If Vl = V Cdim(H+(GN , l)) ≥ 2, p ≥ 1, and 0 <  <BN,l4 ,thenNp(,H(GN , l), D) ≤ 3(2eBpN,lplog3eBpN,lp)Vl. (4)When this bound is plugged into equation (3), it shows the symmetry betweenBY and λ as both appears in the relevant BN,l.4.5 ConsistencyMimicking Theorem 10.1 from [4], we can prove a generic consistency result forMAPE ERM learning. Assume given a series of classes of models, (Gn)n≥1 suchthat⋃n≥1Gn is dense in the set of measurable functions from Rp to R accordingto the L1(µ) metric for any probability measure µ. Assume in addition that eachGn leads to a finite VC-dim Vn = V Cdim(H+(Gn, lMAPE) and that each Gn isuniformly bounded by BGn . Notice that those two conditions are compatiblewith the density condition only if limn→∞ vn =∞ and limn→∞BGn =∞.Assume finally that (X,Y ) is such as |Y | ≥ λ (almost surely) and thatlimn→∞vnB2GnlogBGnn= 0, then LMAPE(ĝlMAPE ,n) converges almost surely toL∗MAPE , which shows the consistency of the ERM estimator for the MAPE.The proof is based on the classical technique of exponential bounding. Plu-gin equation (4) into equation (3) gives a bound on the deviation between theempirical mean and the expectation ofK(n, ) = 24(16eBnlog24eBn)vne− n2128B2n ,with Bn = 1+BGnλ. Then it is easy to check that the conditions above guaranteethat∑n≥1K(n, ) < ∞ for all  > 0. This is sufficient to show almost sureconvergence of LMAPE(ĝlMAPE ,n) − L∗MAPE,Gnto 0. The conclusion followsfrom the density hypothesis.5 ConclusionWe have shown that learning under the Mean Absolute Percentage Error isfeasible both on a practical point of view and on a theoretical one. In applicationcontexts where this error measure is adapted (in general when the target variableis positive by design and remains quite far away from zero, e.g. in price predictionfor expensive goods), there is therefore no reason to use the Mean Square Error(or another measure) as a proxy for the MAPE. An open theoretical questionis whether the symmetry between the upper bound on |Y | for MSE/MAE andthe lower bound on |Y | for the MAPE is strong enough to allow results such asTheorem 10.3 in [4] in which a truncated estimator is used to lift the boundedhypothesis on |Y |.References[1] M. Anthony and P. L. Bartlett. Neural Network Learning: Theoretical Foundations. Cam-bridge University Press, 1999.[2] S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.[3] M. Ezekiel. Methods of Correlation Analysis. Wiley, 1930.[4] L. Gyo¨rfi, M. Kohler, A. Krzyz˙ak, and H. Walk. A Distribution-Free Theory of Nonpara-metric Regression. Springer, New York, 2002.[5] R. Koenker. quantreg: Quantile Regression, 2013. R package version 5.05.

International audienceWe study in this paper the consequences of using the Mean Absolute Percentage Error (MAPE) as a measure of quality for regression models. We prove the existence of an optimal MAPE model and we show the universal consistency of Empirical Risk Minimization based on the MAPE. We also show that finding the best model under the MAPE is equivalent to doing weighted Mean Absolute Error (MAE) regression, and we apply this weighting strategy to kernel regression. The behavior of the MAPE kernel regression is illustrated on simulated data

Mean Absolute Percentage Error for regression models

https://hal.archives-ouvertes.fr/hal-01162980/document

Using the Mean Absolute Percentage Error for Regression Models

Abstract

Similar works

Full text

Available Versions

Archive Ouverte en Sciences de l'Information et de la Communication

HAL-Paris1

HAL-Paris1