We compare the risk of ridge regression to a simple variant of ordinary least
squares, in which one simply projects the data onto a finite dimensional
subspace (as specified by a Principal Component Analysis) and then performs an
ordinary (un-regularized) least squares regression in this subspace. This note
shows that the risk of this ordinary least squares method is within a constant
factor (namely 4) of the risk of ridge regression.Comment: Appearing in JMLR 14, June 201

Dean P. Foster

Gabor Lugosi

Lyle H. Ungar

Paramveer S. Dhillon

Sham M. Kakade

English

arXiv

We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a principal component analysis) and then performs an ordinary (un- regularized) least squares regression in this subspace. This note shows that the risk of this ordinary least squares method (PCA-OLS) is within a constant factor (namely 4) of the risk of ridge regression (RR)

Dhillon, Paramveer S.

Foster, Dean P

Kakade, Sham M

Ungar, Lyle

ScholarlyCommons@Penn

University of PennsylvaniaScholarlyCommonsStatistics Papers Wharton Faculty Research6-2013A Risk Comparison of Ordinary Least Squares vsRidge RegressionParamveer S. DhillonUniversity of PennsylvaniaDean P. FosterUniversity of PennsylvaniaSham M. KakadeUniversity of PennsylvaniaLyle UngarUniversity of PennsylvaniaFollow this and additional works at: http://repository.upenn.edu/statistics_papersPart of the Computer Sciences CommonsThis paper is posted at ScholarlyCommons. http://repository.upenn.edu/statistics_papers/135For more information, please contact repository@pobox.upenn.edu.Recommended CitationDhillon, P. S., Foster, D. P., Kakade, S. M., & Ungar, L. (2013). A Risk Comparison of Ordinary Least Squares vs Ridge Regression.Journal of Machine Learning Research, 14 1505-1511. Retrieved from http://repository.upenn.edu/statistics_papers/135A Risk Comparison of Ordinary Least Squares vs Ridge RegressionAbstractWe compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simplyprojects the data onto a finite dimensional subspace (as specified by a principal component analysis) and thenperforms an ordinary (un- regularized) least squares regression in this subspace. This note shows that the riskof this ordinary least squares method (PCA-OLS) is within a constant factor (namely 4) of the risk of ridgeregression (RR).Keywordsrisk inflation, ridge regression, pcaDisciplinesComputer SciencesThis journal article is available at ScholarlyCommons: http://repository.upenn.edu/statistics_papers/135Journal of Machine Learning Research 14 (2013) 1505-1511 Submitted 5/12; Revised 3/13; Published 6/13A Risk Comparison of Ordinary Least Squares vs Ridge RegressionParamveer S. Dhillon DHILLON@CIS.UPENN.EDUDepartment of Computer and Information ScienceUniversity of PennsylvaniaPhiladelphia, PA 19104, USADean P. Foster FOSTER@WHARTON.UPENN.EDUDepartment of StatisticsWharton School, University of PennsylvaniaPhiladelphia, PA 19104, USASham M. Kakade SKAKADE@MICROSOFT.COMMicrosoft ResearchOne Memorial DriveCambridge, MA 02142, USALyle H. Ungar UNGAR@CIS.UPENN.EDUDepartment of Computer and Information ScienceUniversity of PennsylvaniaPhiladelphia, PA 19104, USAEditor: Gabor LugosiAbstractWe compare the risk of ridge regression to a simple variant of ordinary least squares, in which onesimply projects the data onto a finite dimensional subspace (as specified by a principal componentanalysis) and then performs an ordinary (un-regularized) least squares regression in this subspace.This note shows that the risk of this ordinary least squares method (PCA-OLS) is within a constantfactor (namely 4) of the risk of ridge regression (RR).Keywords: risk inflation, ridge regression, pca1. IntroductionConsider the fixed design setting where we have a set of n vectors X = {Xi}, and let X denote thematrix where the ith row of X is Xi. The observed label vector is Y ∈ Rn. Suppose that:Y = Xβ+ ε,where ε is independent noise in each coordinate, with the variance of εi being σ2.The objective is to learn E[Y ] = Xβ. The expected loss of a vector β estimator is:L(β) =1nEY[‖Y −Xβ‖2],Let βˆ be an estimator of β (constructed with a sample Y ). DenotingΣ :=1nXTX,c©2013 Paramveer S. Dhillon, Dean P. Foster, Sham M. Kakade and Lyle H. Ungar.DHILLON, FOSTER, KAKADE AND UNGARwe have that the risk (i.e., expected excess loss) is:Risk(βˆ) := Eβˆ[L(βˆ)−L(β)] = Eβˆ‖βˆ−β‖2Σ,where ‖x‖Σ = x⊤Σx and where the expectation is with respect to the randomness in Y .We show that a simple variant of ordinary (un-regularized) least squares always compares fa-vorably to ridge regression (as measured by the risk). This observation is based on the followingbias variance decomposition:Risk(βˆ) = E‖βˆ− β¯‖2Σ︸ ︷︷ ︸Variance+ ‖β¯−β‖2Σ︸ ︷︷ ︸Prediction Bias, (1)where β¯ = E[βˆ].1.1 The Risk of Ridge Regression (RR)Ridge regression or Tikhonov Regularization (Tikhonov, 1963) penalizes the ℓ2 norm of a parametervector β and “shrinks” it towards zero, penalizing large values more. The estimator is:βˆλ = argminβ{‖Y −Xβ‖2+λ‖β‖2}.The closed form estimate is then:βˆλ = (Σ+λI)−1(1nXTY).Note thatβˆ0 = βˆλ=0 = argminβ{‖Y −Xβ‖2},is the ordinary least squares estimator.Without loss of generality, rotate X such that:Σ= diag(λ1,λ2, . . . ,λp),where the λi’s are ordered in decreasing order.To see the nature of this shrinkage observe that:[βˆλ] j :=λ jλ j+λ[βˆ0] j,where βˆ0 is the ordinary least squares estimator.Using the bias-variance decomposition, (Equation 1), we have that:Lemma 1Risk(βˆλ) =σ2n∑j(λ jλ j+λ)2+∑jβ2jλ j(1+λ jλ )2.The proof is straightforward and is provided in the appendix.1506A RISK COMPARISON OF ORDINARY LEAST SQUARES VS RIDGE REGRESSION2. Ordinary Least Squares with PCA (PCA-OLS)Now let us construct a simple estimator based on λ. Note that our rotated coordinate system whereΣ is equal to diag(λ1,λ2, . . . ,λp) corresponds the PCA coordinate system.Consider the following ordinary least squares estimator on the “top” PCA subspace — it usesthe least squares estimate on coordinate j if λ j ≥ λ and 0 otherwise[βˆPCA,λ] j ={[βˆ0] j if λ j ≥ λ0 otherwise.The following claim shows this estimator compares favorably to the ridge estimator (for every λ)–no matter how the λ is chosen, for example, using cross validation or any other strategy.Our main theorem (Theorem 2) bounds the Risk Ratio/Risk Inflation1 of the PCA-OLS and theRR estimators.Theorem 2 (Bounded Risk Inflation) For all λ≥ 0, we have that:0≤ Risk(βˆPCA,λ)Risk(βˆλ)≤ 4,and the left hand inequality is tight.Proof Using the bias variance decomposition of the risk we can write the risk as:Risk(βˆPCA,λ) =σ2n∑j1λ j≥λ + ∑j:λ j<λλ jβ2j .The first term represents the variance and the second the bias.The ridge regression risk is given by Lemma 1. We now show that the jth term in the expressionfor the PCA risk is within a factor 4 of the jth term of the ridge regression risk. First, let’s considerthe case when λ j ≥ λ, then the ratio of jth terms is:σ2nσ2n(λ jλ j+λ)2+β2jλ j(1+λ jλ)2≤σ2nσ2n(λ jλ j+λ)2 =(1+λλ j)2≤ 4.Similarly, if λ j < λ, the ratio of the jth terms is:λ jβ2jσ2n(λ jλ j+λ)2+β2jλ j(1+λ jλ)2≤ λ jβ2jλ jβ2j(1+λ jλ)2=(1+λ jλ)2≤ 4.Since, each term is within a factor of 4 the proof is complete.It is worth noting that the converse is not true and the ridge regression estimator (RR) canbe arbitrarily worse than the PCA-OLS estimator. An example which shows that the left handinequality is tight is given in the Appendix.1. Risk Inflation has also been used as a criterion for evaluating feature selection procedures (Foster and George, 1994).1507DHILLON, FOSTER, KAKADE AND UNGAR3. ExperimentsFirst, we generated synthetic data with p = 100 and varying values of n= {20, 50, 80, 110}. Thedata was generated in a fixed design setting as Y = Xβ+ ε where εi ∼ N (0,1) ∀i = 1, . . . ,n.Furthermore, Xn×p ∼ MVN(0,I) where MVN(µ,Σ) is the Multivariate Normal Distribution withmean vector µ, variance-covariance matrix Σ and β j ∼N (0,1) ∀ j = 1, . . . , p.The results are shown in Figure 1. As can be seen, the risk ratio of PCA (PCA-OLS) and ridgeregression (RR) is never worse than 4 and often its better than 1 as dictated by Theorem 2.Next , we chose two real world data sets, namely USPS (n=1500, p=241) and BCI (n=400,p=117).2Since we do not know the true model for these data sets, we used all the n observations tofit an OLS regression and used it as an estimate of the true parameter β. This is a reasonableapproximation to the true parameter as we estimate the ridge regression (RR) and PCA-OLS modelson a small subset of these observations. Next we choose a random subset of the observations, namely0.2× p, 0.5× p and 0.8× p to fit the ridge regression (RR) and PCA-OLS models.The results are shown in Figure 2. As can be seen, the risk ratio of PCA-OLS to ridge regression(RR) is again within a factor of 4 and often PCA-OLS is better, that is, the ratio < 1.4. ConclusionWe showed that the risk inflation of a particular ordinary least squares estimator (on the “top” PCAsubspace) is within a factor 4 of the ridge estimator. It turns out the converse is not true — this PCAestimator may be arbitrarily better than the ridge one.Appendix A.Proof of Lemma 1. We analyze the bias-variance decomposition in Equation 1. For the variance,EY‖βˆλ− β¯λ‖2Σ = ∑jλ jEY ([βˆλ] j− [β¯λ] j)2= ∑jλ j(λ j+λ)21n2E[n∑i=1(Yi−E[Yi])[Xi] jn∑i′=1(Y ′i −E[Y ′i ])[X ′i ] j]= ∑jλ j(λ j+λ)2σ2nn∑i=1Var(Yi)[Xi]2j= ∑jλ j(λ j+λ)2σ2nn∑i=1[Xi]2j=σ2n∑jλ2j(λ j+λ)2.2. The details about the data sets can be found here: http://olivier.chapelle.cc/ssl-book/benchmarks.html.1508A RISK COMPARISON OF ORDINARY LEAST SQUARES VS RIDGE REGRESSION0 10 20 30 40 500.00.20.40.60.8LambdaRisk Ratio (PCA−OLS/RR)n=0.2p0 10 20 30 40 500.20.61.0LambdaRisk Ratio (PCA−OLS/RR)n=0.5p0 10 20 30 40 500.50.70.91.1LambdaRisk Ratio (PCA−OLS/RR)n=0.8p0 10 20 30 40 500.50.70.9LambdaRisk Ratio (PCA−OLS/RR)n=1.1pFigure 1: Plots showing the risk ratio as a function of λ, the regularization parameter and n, forthe synthetic data set. p=100 in all the cases. The error bars correspond to one standarddeviation for 100 such random trials.0 10 20 30 40 501.051.101.15LambdaRisk Ratio (PCA−OLS/RR)n=0.2p0 10 20 30 40 501.101.201.30LambdaRisk Ratio (PCA−OLS/RR)n=0.5p0 10 20 30 40 501.101.151.201.25LambdaRisk Ratio (PCA−OLS/RR)n=0.8p0 10 20 30 40 501.101.251.40LambdaRisk Ratio (PCA−OLS/RR)n=1.1p0 10 20 30 40 500.9700.985LambdaRisk Ratio (PCA−OLS/RR)0 10 20 30 40 500.9850.995LambdaRisk Ratio (PCA−OLS/RR)0 10 20 30 40 500.9901.000LambdaRisk Ratio (PCA−OLS/RR)0 10 20 30 40 500.9800.9901.0001.010LambdaRisk Ratio (PCA−OLS/RR)Figure 2: Plots showing the risk ratio as a function of λ, the regularization parameter and n, for tworeal world data sets (BCI and USPS–top to bottom).1509DHILLON, FOSTER, KAKADE AND UNGARSimilarly, for the bias,‖β¯λ−β‖2Σ = ∑jλ j([β¯λ] j− [β] j)2= ∑jβ2jλ j(λ jλ j+λ−1)2= ∑jβ2jλ j(1+λ jλ )2,which completes the proof.The risk for RR can be arbitrarily worse than the PCA-OLS estimator.Consider the standard OLS setting described in Section 1 in which X is n× p matrix and Y is an×1 vector.Let X = diag(√1+α,1, . . . ,1), then Σ = X⊤X = diag(1+α,1, . . . ,1) for some (α > 0) andalso choose β = [2+α,0, . . . ,0]. For convenience let’s also choose σ2 = n.Then, using Lemma 1, we get the risk of RR estimator asRisk(βˆλ) =(1+α1+α+λ)2︸ ︷︷ ︸I+(p−1)(1+λ)2︸ ︷︷ ︸II+(2+α)2× (1+α)(1+ 1+αλ )2︸ ︷︷ ︸III.Let’s consider two cases• Case 1: λ < (p−1)1/3−1, then II > (p−1)1/3.• Case 2: λ > 1, then 1+ 1+αλ < 2+α, hence III > (1+α).Combining these two cases we get ∀λ, Risk(βˆλ) > min((p− 1)1/3,(1+α)). If we choose psuch that p−1= (1+α)3, then Risk(βˆλ)> (1+α).The PCA-OLS risk (From Theorem 2) is:Risk(βˆPCA,λ) = ∑j1λ j≥λ + ∑j:λ j<λλ jβ2j .Considering λ ∈ (1,1+α), the first term will contribute 1 to the risk and rest everything will be0. So the risk of PCA-OLS is 1 and the risk ratio isRisk(βˆPCA,λ)Risk(βˆλ)≤ 1(1+α).Now, for large α, the risk ratio ≈ 0.1510A RISK COMPARISON OF ORDINARY LEAST SQUARES VS RIDGE REGRESSIONReferencesD. P. Foster and E. I. George. The risk inflation criterion for multiple regression. The Annals ofStatistics, pages 1947–1975, 1994.A. N. Tikhonov. Solution of incorrectly formulated problems and the regularization method. SovietMath Dokl 4, pages 501–504, 1963.1511

A Risk Comparison of Ordinary Least Squares vs Ridge Regression

Kosmopolis

University of Pennsylvania
ScholarlyCommons
Statistics Papers Wharton Faculty Research
6-2013
A Risk Comparison of Ordinary Least Squares vs
Ridge Regression
Paramveer S. Dhillon
University of Pennsylvania
Dean P. Foster
University of Pennsylvania
Sham M. Kakade
University of Pennsylvania
Lyle Ungar
University of Pennsylvania
Follow this and additional works at: http://repository.upenn.edu/statistics_papers
Part of the Computer Sciences Commons
This paper is posted at ScholarlyCommons. http://repository.upenn.edu/statistics_papers/135
For more information, please contact repository@pobox.upenn.edu.
Recommended Citation
Dhillon, P. S., Foster, D. P., Kakade, S. M., & Ungar, L. (2013). A Risk Comparison of Ordinary Least Squares vs Ridge Regression.
Journal of Machine Learning Research, 14 1505-1511. Retrieved from http://repository.upenn.edu/statistics_papers/135
A Risk Comparison of Ordinary Least Squares vs Ridge Regression
Abstract
We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply
projects the data onto a finite dimensional subspace (as specified by a principal component analysis) and then
performs an ordinary (un- regularized) least squares regression in this subspace. This note shows that the risk
of this ordinary least squares method (PCA-OLS) is within a constant factor (namely 4) of the risk of ridge
regression (RR).
Keywords
risk inflation, ridge regression, pca
Disciplines
Computer Sciences
This journal article is available at ScholarlyCommons: http://repository.upenn.edu/statistics_papers/135
Journal of Machine Learning Research 14 (2013) 1505-1511 Submitted 5/12; Revised 3/13; Published 6/13
A Risk Comparison of Ordinary Least Squares vs Ridge Regression
Paramveer S. Dhillon DHILLON@CIS.UPENN.EDU
Department of Computer and Information Science
University of Pennsylvania
Philadelphia, PA 19104, USA
Dean P. Foster FOSTER@WHARTON.UPENN.EDU
Department of Statistics
Wharton School, University of Pennsylvania
Philadelphia, PA 19104, USA
Sham M. Kakade SKAKADE@MICROSOFT.COM
Microsoft Research
One Memorial Drive
Cambridge, MA 02142, USA
Lyle H. Ungar UNGAR@CIS.UPENN.EDU
Department of Computer and Information Science
University of Pennsylvania
Philadelphia, PA 19104, USA
Editor: Gabor Lugosi
Abstract
We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one
simply projects the data onto a finite dimensional subspace (as specified by a principal component
analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace.
This note shows that the risk of this ordinary least squares method (PCA-OLS) is within a constant
factor (namely 4) of the risk of ridge regression (RR).
Keywords: risk inflation, ridge regression, pca
1. Introduction
Consider the fixed design setting where we have a set of n vectors X = {Xi}, and let X denote the
matrix where the ith row of X is Xi. The observed label vector is Y ∈ Rn. Suppose that:
Y = Xβ+ ε,
where ε is independent noise in each coordinate, with the variance of εi being σ
2.
The objective is to learn E[Y ] = Xβ. The expected loss of a vector β estimator is:
L(β) =
1
n
EY[‖Y −Xβ‖2],
Let β̂ be an estimator of β (constructed with a sample Y ). Denoting
Σ :=
1
n
XT X,
c©2013 Paramveer S. Dhillon, Dean P. Foster, Sham M. Kakade and Lyle H. Ungar.
DHILLON, FOSTER, KAKADE AND UNGAR
we have that the risk (i.e., expected excess loss) is:
Risk(β̂) := Eβ̂[L(β̂)−L(β)] = Eβ̂‖β̂−β‖
2
Σ,
where ‖x‖Σ = x⊤Σx and where the expectation is with respect to the randomness in Y .
We show that a simple variant of ordinary (un-regularized) least squares always compares fa-
vorably to ridge regression (as measured by the risk). This observation is based on the following
bias variance decomposition:
Risk(β̂) = E‖β̂− β̄‖2Σ
︸ ︷︷ ︸
Variance
+ ‖β̄−β‖2Σ
︸ ︷︷ ︸
Prediction Bias
, (1)
where β̄ = E[β̂].
1.1 The Risk of Ridge Regression (RR)
Ridge regression or Tikhonov Regularization (Tikhonov, 1963) penalizes the ℓ2 norm of a parameter
vector β and “shrinks” it towards zero, penalizing large values more. The estimator is:
β̂λ = argmin
β
{‖Y −Xβ‖2 +λ‖β‖2}.
The closed form estimate is then:
β̂λ = (Σ+λI)
−1
(
1
n
XTY
)
.
Note that
β̂0 = β̂λ=0 = argmin
β
{‖Y −Xβ‖2},
is the ordinary least squares estimator.
Without loss of generality, rotate X such that:
Σ= diag(λ1,λ2, . . . ,λp),
where the λi’s are ordered in decreasing order.
To see the nature of this shrinkage observe that:
[β̂λ] j :=
λ j
λ j +λ
[β̂0] j,
where β̂0 is the ordinary least squares estimator.
Using the bias-variance decomposition, (Equation 1), we have that:
Lemma 1
Risk(β̂λ) =
σ2
n
∑
j
(
λ j
λ j +λ
)2
+∑
j
β2j
λ j
(1+
λ j
λ )
2
.
The proof is straightforward and is provided in the appendix.
1506
A RISK COMPARISON OF ORDINARY LEAST SQUARES VS RIDGE REGRESSION
2. Ordinary Least Squares with PCA (PCA-OLS)
Now let us construct a simple estimator based on λ. Note that our rotated coordinate system where
Σ is equal to diag(λ1,λ2, . . . ,λp) corresponds the PCA coordinate system.
Consider the following ordinary least squares estimator on the “top” PCA subspace — it uses
the least squares estimate on coordinate j if λ j ≥ λ and 0 otherwise
[β̂PCA,λ] j =
{
[β̂0] j if λ j ≥ λ
0 otherwise
.
The following claim shows this estimator compares favorably to the ridge estimator (for every λ)–
no matter how the λ is chosen, for example, using cross validation or any other strategy.
Our main theorem (Theorem 2) bounds the Risk Ratio/Risk Inflation1 of the PCA-OLS and the
RR estimators.
Theorem 2 (Bounded Risk Inflation) For all λ ≥ 0, we have that:
0 ≤ Risk(β̂PCA,λ)
Risk(β̂λ)
≤ 4,
and the left hand inequality is tight.
Proof Using the bias variance decomposition of the risk we can write the risk as:
Risk(β̂PCA,λ) =
σ2
n
∑
j
1λ j≥λ + ∑
j:λ j<λ
λ jβ
2
j .
The first term represents the variance and the second the bias.
The ridge regression risk is given by Lemma 1. We now show that the jth term in the expression
for the PCA risk is within a factor 4 of the jth term of the ridge regression risk. First, let’s consider
the case when λ j ≥ λ, then the ratio of jth terms is:
σ2
n
σ2
n
(
λ j
λ j+λ
)2
+β2j
λ j
(1+
λ j
λ
)2
≤
σ2
n
σ2
n
(
λ j
λ j+λ
)2
=
(
1+
λ
λ j
)2
≤ 4.
Similarly, if λ j < λ, the ratio of the j
th terms is:
λ jβ
2
j
σ2
n
(
λ j
λ j+λ
)2
+β2j
λ j
(1+
λ j
λ
)2
≤
λ jβ
2
j
λ jβ
2
j
(1+
λ j
λ
)2
=
(
1+
λ j
λ
)2
≤ 4.
Since, each term is within a factor of 4 the proof is complete.
It is worth noting that the converse is not true and the ridge regression estimator (RR) can
be arbitrarily worse than the PCA-OLS estimator. An example which shows that the left hand
inequality is tight is given in the Appendix.
1. Risk Inflation has also been used as a criterion for evaluating feature selection procedures (Foster and George, 1994).
1507
DHILLON, FOSTER, KAKADE AND UNGAR
3. Experiments
First, we generated synthetic data with p = 100 and varying values of n= {20, 50, 80, 110}. The
data was generated in a fixed design setting as Y = Xβ+ ε where εi ∼ N (0,1) ∀i = 1, . . . ,n.
Furthermore, Xn×p ∼ MV N(0,I) where MVN(µ,Σ) is the Multivariate Normal Distribution with
mean vector µ, variance-covariance matrix Σ and β j ∼ N (0,1) ∀ j = 1, . . . , p.
The results are shown in Figure 1. As can be seen, the risk ratio of PCA (PCA-OLS) and ridge
regression (RR) is never worse than 4 and often its better than 1 as dictated by Theorem 2.
Next , we chose two real world data sets, namely USPS (n=1500, p=241) and BCI (n=400,
p=117).2
Since we do not know the true model for these data sets, we used all the n observations to
fit an OLS regression and used it as an estimate of the true parameter β. This is a reasonable
approximation to the true parameter as we estimate the ridge regression (RR) and PCA-OLS models
on a small subset of these observations. Next we choose a random subset of the observations, namely
0.2× p, 0.5× p and 0.8× p to fit the ridge regression (RR) and PCA-OLS models.
The results are shown in Figure 2. As can be seen, the risk ratio of PCA-OLS to ridge regression
(RR) is again within a factor of 4 and often PCA-OLS is better, that is, the ratio < 1.
4. Conclusion
We showed that the risk inflation of a particular ordinary least squares estimator (on the “top” PCA
subspace) is within a factor 4 of the ridge estimator. It turns out the converse is not true — this PCA
estimator may be arbitrarily better than the ridge one.
Appendix A.
Proof of Lemma 1. We analyze the bias-variance decomposition in Equation 1. For the variance,
EY‖β̂λ − β̄λ‖2Σ = ∑
j
λ jEY ([β̂λ] j − [β̄λ] j)2
= ∑
j
λ j
(λ j +λ)2
1
n2
E
[
n
∑
i=1
(Yi −E[Yi])[Xi] j
n
∑
i′=1
(Y ′i −E[Y ′i ])[X ′i ] j
]
= ∑
j
λ j
(λ j +λ)2
σ2
n
n
∑
i=1
Var(Yi)[Xi]
2
j
= ∑
j
λ j
(λ j +λ)2
σ2
n
n
∑
i=1
[Xi]
2
j
=
σ2
n
∑
j
λ2j
(λ j +λ)2
.
2. The details about the data sets can be found here: http://olivier.chapelle.cc/ssl-book/benchmarks.html.
1508
A RISK COMPARISON OF ORDINARY LEAST SQUARES VS RIDGE REGRESSION
0 10 20 30 40 50
0
.0
0
.2
0
.4
0
.6
0
.8
Lambda
R
is
k
 R
a
ti
o
 (
P
C
A
−
O
L
S
/R
R
)
n=0.2p
0 10 20 30 40 50
0
.2
0
.6
1
.0
Lambda
R
is
k
 R
a
ti
o
 (
P
C
A
−
O
L
S
/R
R
)
n=0.5p
0 10 20 30 40 50
0
.5
0
.7
0
.9
1
.1
Lambda
R
is
k
 R
a
ti
o
 (
P
C
A
−
O
L
S
/R
R
)
n=0.8p
0 10 20 30 40 50
0
.5
0
.7
0
.9
Lambda
R
is
k
 R
a
ti
o
 (
P
C
A
−
O
L
S
/R
R
)
n=1.1p
Figure 1: Plots showing the risk ratio as a function of λ, the regularization parameter and n, for
the synthetic data set. p=100 in all the cases. The error bars correspond to one standard
deviation for 100 such random trials.
0 10 20 30 40 50
1
.0
5
1
.1
0
1
.1
5
Lambda
R
is
k
 R
a
ti
o
 (
P
C
A
−
O
L
S
/R
R
)
n=0.2p
0 10 20 30 40 50
1
.1
0
1
.2
0
1
.3
0
Lambda
R
is
k
 R
a
ti
o
 (
P
C
A
−
O
L
S
/R
R
)
n=0.5p
0 10 20 30 40 50
1
.1
0
1
.1
5
1
.2
0
1
.2
5
Lambda
R
is
k
 R
a
ti
o
 (
P
C
A
−
O
L
S
/R
R
)
n=0.8p
0 10 20 30 40 50
1
.1
0
1
.2
5
1
.4
0
Lambda
R
is
k
 R
a
ti
o
 (
P
C
A
−
O
L
S
/R
R
)
n=1.1p
0 10 20 30 40 50
0
.9
7
0
0
.9
8
5
Lambda
R
is
k
 R
a
ti
o
 (
P
C
A
−
O
L
S
/R
R
)
0 10 20 30 40 50
0
.9
8
5
0
.9
9
5
Lambda
R
is
k
 R
a
ti
o
 (
P
C
A
−
O
L
S
/R
R
)
0 10 20 30 40 50
0
.9
9
0
1
.0
0
0
Lambda
R
is
k
 R
a
ti
o
 (
P
C
A
−
O
L
S
/R
R
)
0 10 20 30 40 500
.9
8
0
0
.9
9
0
1
.0
0
0
1
.0
1
0
Lambda
R
is
k
 R
a
ti
o
 (
P
C
A
−
O
L
S
/R
R
)
Figure 2: Plots showing the risk ratio as a function of λ, the regularization parameter and n, for two
real world data sets (BCI and USPS–top to bottom).
1509
DHILLON, FOSTER, KAKADE AND UNGAR
Similarly, for the bias,
‖β̄λ −β‖2Σ = ∑
j
λ j([β̄λ] j − [β] j)2
= ∑
j
β2jλ j
(
λ j
λ j +λ
−1
)2
= ∑
j
β2j
λ j
(1+
λ j
λ )
2
,
which completes the proof.
The risk for RR can be arbitrarily worse than the PCA-OLS estimator.
Consider the standard OLS setting described in Section 1 in which X is n× p matrix and Y is a
n×1 vector.
Let X = diag(
√
1+α,1, . . . ,1), then Σ = X⊤X = diag(1+α,1, . . . ,1) for some (α > 0) and
also choose β = [2+α,0, . . . ,0]. For convenience let’s also choose σ2 = n.
Then, using Lemma 1, we get the risk of RR estimator as
Risk(β̂λ) =




(
1+α
1+α+λ
)2
︸ ︷︷ ︸
I
+
(p−1)
(1+λ)2
︸ ︷︷ ︸
II




+(2+α)2 × (1+α)
(1+ 1+αλ )
2
︸ ︷︷ ︸
III
.
Let’s consider two cases
• Case 1: λ < (p−1)1/3 −1, then II > (p−1)1/3.
• Case 2: λ > 1, then 1+ 1+αλ < 2+α, hence III > (1+α).
Combining these two cases we get ∀λ, Risk(β̂λ) > min((p− 1)1/3,(1+α)). If we choose p
such that p−1 = (1+α)3, then Risk(β̂λ)> (1+α).
The PCA-OLS risk (From Theorem 2) is:
Risk(β̂PCA,λ) = ∑
j
1λ j≥λ + ∑
j:λ j<λ
λ jβ
2
j .
Considering λ ∈ (1,1+α), the first term will contribute 1 to the risk and rest everything will be
0. So the risk of PCA-OLS is 1 and the risk ratio is
Risk(β̂PCA,λ)
Risk(β̂λ)
≤ 1
(1+α)
.
Now, for large α, the risk ratio ≈ 0.
1510
A RISK COMPARISON OF ORDINARY LEAST SQUARES VS RIDGE REGRESSION
References
D. P. Foster and E. I. George. The risk inflation criterion for multiple regression. The Annals of
Statistics, pages 1947–1975, 1994.
A. N. Tikhonov. Solution of incorrectly formulated problems and the regularization method. Soviet
Math Dokl 4, pages 501–504, 1963.
1511


https://repository.upenn.edu/cgi/viewcontent.cgi?article=1467&amp;context=statistics_papers

A Risk Comparison of Ordinary Least Squares vs Ridge Regression

Abstract

Similar works

Full text

Available Versions

ScholarlyCommons@Penn

Kosmopolis