In this note we give an elementary proof of the Fritz-John and Karush-Kuhn-Tucker conditions for nonlinear finite dimensional programming problems with equality and/or inequality constraints.The proof avoids the implicit function theorem usually applied when dealing with equality constraints and uses a generalization ofFarkas lemma and the Bolzano-Weierstrass property for compact sets.Fritz-John conditions;Karush-Kuhn-Tucker conditions;nonlinear programming

Birbil, S.I.

Frenk, J.B.G.

Still, G.J.

English

Research Papers in Economics

An Elementary Proof of the Fritz-John and
Karush-Kuhn-Tucker Conditions in Nonlinear Programming
S ¸. ˙ I. Birbil†, J. B. G. Frenk‡∗ and G. J. Still\
†Faculty of Engineering and Natural Sciences, Sabancı University
Orhanli-Tuzla, 34956, Istanbul, Turkey.
sibirbil@sabanciuniv.edu
‡Econometric Institute, Erasmus University Rotterdam
P.O. Box 1738, 3000 DR Rotterdam, The Netherlands.
frenk@few.eur.nl
\Department of Mathematical Sciences, University of Twente
P.O. Box 217, 7500 AE Enschede, The Netherlands.
g.still@math.twente.nl
Econometric Institute Report E.I. 2005-43
ABSTRACT. In this note we give an elementary proof of the Fritz-John and Karush-Kuhn-Tucker
conditions for nonlinear ﬁnite dimensional programming problems with equality and/or inequality con-
straints. The proof avoids the implicit function theorem usually applied when dealing with equality
constraints and uses a generalization of Farkas lemma and the Bolzano-Weierstrass property for com-
pact sets.
Keywords. Nonlinear programming. Fritz-John conditions, Karush-Kuhn-Tucker conditions
1 Introduction
LetAbeanm×nmatrixwithrowsa>
k , 1 ≤ k ≤ m, b ∈ Rm anm-dimensionalvector, andfi : Rn → R,
0 ≤ i ≤ q some non-afﬁne, continuously differentiable functions. We consider the optimization problem
min{f0(x) : x ∈ FP}, FP := {x ∈ Rn : a>
k x ≤ bk,1 ≤ k ≤ m, fi(x) ≤ 0,1 ≤ i ≤ q}, (P)
and the program including equalities
min{f0(x) : x ∈ FQ}, FQ := FP ∩ {x ∈ Rn : hj(x) = 0,1 ≤ j ≤ r}, (Q)
where the functions hj : Rn → R, 1 ≤ j ≤ r, are non-afﬁne and continuously differentiable.
∗Corresponding author.
1 October 12, 20052. The FJ and KKT conditions for problems (P) and (Q)
Two basic results covered in every course on nonlinear programming are the Fritz-John (FJ) and
Karush-Kuhn-Tucker (KKT) necessary conditions for the local minimizers of optimization problems (P)
and (Q) [7–9]. Denoting the nonnegative orthant of Rl by Rl
+, the FJ necessary conditions for problem
(P) are given by the following: If xP is a local minimizer of problem (P), then there exist (see for example
[2, 5]) vectors 0 6= λ ∈ R
q+1
+ and ν ∈ Rm
+ satisfying
Pq
i=0 λi∇fi(xP) +
Pm
k=1 νkak = 0,
λifi(xP) = 0, 1 ≤ i ≤ q and νk(a>
k xP − bk) = 0, 1 ≤ k ≤ m.
(FJP)
For optimization problem (Q) the resulting FJ conditions are as follows: If xQ is a local minimizer of
problem (Q), then there exist (see for example [2, 5]) vectors (λ,ν) ∈ R
q+1+m
+ , µ ∈ Rr with (λ,µ) 6= 0
satisfying
Pq
i=0 λi∇fi(xQ) +
Pr
j=1 µj∇hj(xQ) +
Pm
k=1 νkak = 0,
λifi(xQ) = 0, 1 ≤ i ≤ q and νk(a>
k xQ − bk) = 0, 1 ≤ k ≤ m.
(FJQ)
If λ0 given in conditions (FJP) and (FJQ) can be chosen positive, then the resulting necessary conditions
are called the KKT conditions for problems (P) and (Q), respectively. A sufﬁcient condition for λ0 to
be positive is given by a so-called ﬁrst-order constraint qualiﬁcation. In the next section we ﬁrstly give
an elementary proof of the FJ and KKT conditions for problem (P). Then the same proof is given for
optimization problem (Q) by using a perturbation argument but avoiding the implicit function theorem.
2 The FJ and KKT conditions for problems (P) and (Q)
For δ > 0 and ¯ x ∈ Rn, let N(¯ x,δ) denote a δ-neighborhood of ¯ x given by
N(¯ x,δ) := {x ∈ Rn : kx − ¯ xk ≤ δ}.
A vector xP is called a local minimizer of optimization problem (P) (respectively, for optimization prob-
lem (Q) if xP ∈ FP (respectively, xP ∈ FQ) and there exists some δ > 0 such that f0(xP) ≤ f0(x) for
every x ∈ FP ∩ N(xP,δ) (respectively, x ∈ FQ ∩ N(xP,δ)).
We introduce the active index sets I(x) := {1 ≤ i ≤ q : fi(x) = 0} and K(x) = {1 ≤ k ≤ m :
a>
k x = bk}, and denote by B(x), the matrix consisting of the corresponding active rows a>
k ,k ∈ K(x).
Lemma 2.1 If xP is a local minimizer of problem (P), then max{∇fi(xP)>d : i ∈ I(xP) ∪ {0}} ≥ 0
for every d such that B(xP)d ≤ 0.
Proof. Suppose by contradiction there exists some d0 satisfying B(xP)d0 ≤ 0 and
0 > ∇fi(xP)>d0 = limt↓0
fi(xP + td0) − fi(xP)
t
for every i ∈ I(xP) ∪ {0}. By the ﬁniteness of the sets {0,...,q} and {1,...,m} and the continuity of fi
this implies the existence of some t0 > 0 satisfying
fi(xP + td0) < 0,i / ∈ I(xP),fi(xP + td0) < fi(xP),i ∈ I(xP) ∪ {0},A(xP + td0) ≤ b
for every 0 < t ≤ t0. Hence the vector xP + td0 belongs to FP and satisﬁes f0(xP + td0) < f0(xP)
for every 0 < t ≤ t0. This contradicts that xP is a local minimum. 
2 October 12, 20052. The FJ and KKT conditions for problems (P) and (Q)
Remark 2.1 If the function f0 is pseudo-convex and the functions fi,1 ≤ i ≤ q are strictly pseudo-
convex, then for a feasible xP the reverse implication in Lemma 2.1 also holds and in this result local
minimizer is replaced by global minimizer. A proof of this will be given at the end of this section. More-
over, if maxBd≤0,kdk=1{∇fi(xP)>d : i ∈ I(xP) ∪ {0}} > 0 and xP feasible, then one can show
that xP is a local minimum of order one ([5]), i.e., there exists some δ > 0 and c > 0 such that
f0(x) − f0(xP) ≥ ckx − xPk for every x ∈ FP ∩ N(xP,δ) .
The proof of the FJ conditions for problem (P) will be based on the following generalization of Farkas
lemma ([6]). For completeness, a short proof, using the strong duality result for linear programming, will
be given in the appendix.
Lemma 2.2 Let ∆s ⊆ Rs
+ be the unit simplex. If B is a p × n matrix and ci ∈ Rn,1 ≤ i ≤ s, some
given vectors, then the following conditions are equivalent:
1. For every d ∈ Rn satisfying Bd ≤ 0 it holds that max1≤i≤s c>
i d ≥ 0.
2. There exists some λ ∈ ∆s and µ ∈ R
p
+ satisfying
Ps
i=1 λici + B>µ = 0.
Proof. (FJ conditions for problem (P)) By combining Lemma 2.1 and Lemma 2.2, the FJ conditions
follow. 
It is well-known that the KKT conditions follow from the FJ conditions under some constraint qualiﬁ-
cation. We say that the Mangasarian-Fromovitz (MF) constraint qualiﬁcation for problem (P) holds at a
feasible point x if there exists some d0 satisfying
B(x)d0 ≤ 0 and maxi∈I(x){∇fi(x)>d0} < 0
We now show that at a local minimizer xP of problem (P) satisfying the MF constraint qualiﬁcation, the
KKT conditions must hold.
Proof. (KKT conditions for problem (P)). Assume that λ0 = 0 in the FJ conditions. Applying Lemma
2.2 to the FJ conditions with λ0 = 0 we obtain that maxi∈I(xP) ∇fi(xP)>d ≥ 0 for every B(xP)d ≤ 0.
This contradicts the MF constraint qualiﬁcation. 
To prove the FJ and KKT conditions for problem (Q) without using the implicit function theorem
we consider for a local minimizer xQ of problem (Q) and δ > 0 appropriately chosen and  > 0, the
perturbed feasible region
Fδ() := FP ∩ N(xQ,δ) ∩ {x ∈ Rn : hj(x) ≤ ,−hj(x) ≤ ,1 ≤ j ≤ r},
and the associated optimization problem
min{f0(x) + kx − xQk2 : x ∈ Fδ()}. (Qδ())
Since the feasible region is compact a global minimizer xQ() exists for problem (Qδ()). For these
global minimizers one can show the following result.
Lemma 2.3 For any sequence l ↓ 0 it follows that liml↑∞ xQ(l) = xQ.
3 October 12, 20052. The FJ and KKT conditions for problems (P) and (Q)
Proof. Let us assume to the contrary that there exists a sequence xQ(l),l ∈ N which does not converge
to xQ. By kxQ(l) − xQk ≤ δ and the Bolzano-Weierstrass property for compact sets there exists some
subsequence xQ(l),l ∈ L ⊆ N satisfying
liml↑∞, l∈L xQ(l) = x 6= xQ. (2.1)
By continuity x must be feasible for problem (Q). Since xQ is feasible for (Qδ(l)),l ∈ L it follows that
f0(xQ(l)) + kxQ(l) − xQk2 ≤ f0(xQ) (2.2)
for every l ∈ L. Taking now the limit in relation (2.2) we ﬁnd by relation (2.1) that
f0(x) + kx − xQk2 ≤ f0(xQ)
and this contradicts the local optimality of xQ for problem (Q). 
If xQ is a strict local minimizer, i.e f0(xQ) < f0(x) for every x ∈ FQ ∩N(xQ,δ), we do not need in
the above proof the penalty term kx−xQk2. Using Lemma 2.3 one can now give an elementary proof of
the FJ and KKT conditions for a local minimizer xQ of problem (Q).
Proof. (FJ conditions for problem (Q)) Let l be a strictly decreasing sequence and consider the associ-
ated optimal solutions xQ(l) of (Qδ(l)). For notational convenience we denote xQ(l) by x(l) and by
Lemma 2.3 there exists some l ≥ l0 such that kx(l) − xQk < δ for every l ≥ l0. Introduce now the set
Jl := {1 ≤ j ≤ r : hj(x(l)) = l or hj(x(l)) = −l}.
The set of all subsets of the ﬁnite set {1,...,r} is ﬁnite and so the sequence Jl,l ∈ N contains some subset
J ⊆ {1,...,r} such that L := {l ∈ N : Jl = J} is inﬁnite. Applying now for every l ≥ l0 and l ∈ L
the FJ conditions to problem (Qδ(l)) we obtain that there exist vectors λl ∈ R
q+1
+ , µl ∈ R|J|, νl ∈ Rm
+,
0 6= (λl,µl), satisfying
−λ0lg(x(l)) −
Pq
i=1 λil∇fi(x(l)) −
P
j∈J µjl∇hj(x(l)) =
Pm
k=1 νklak
νkl(a>
k x(l) − bk) = 0, 1 ≤ k ≤ m, and λilfi(x(l)) = 0, 1 ≤ i ≤ q.
(2.3)
with g(x) := ∇f0(x)+2(x−xQ). By relation (2.3) and Caratheodory’s lemma (see Appendix) one can
ﬁnd for every l ∈ L some subset Kl ⊆ {1,...,m} and a vector ν∗
l ∈ R
|Kl|
+ satisfying
−λ0lg(x(l)) −
Xq
i=1
λil∇fi(x(l)) −
X
j∈J
µjl∇hj(x(l)) =
X
k∈Kl
ν∗
klak (2.4)
and the vectors ak,k ∈ Kl are linearly independent. Since 0 6= (λl,µl) we may assume in relation (2.4)
that the vector (λl,µl,ν∗
l ) has Euclidean norm 1. Again by selecting an inﬁnite subsequence L0 ⊆ L
if necessary we can assume Kl = K (the same) for all l ∈ L0. By the Bolzano-Weierstrass theorem
the sequence of vectors (λl,µl,ν∗
l ),l ∈ L0 has a converging subsequence, i.e, there exists an inﬁnite set
L1 ⊆ L0 with liml∈L1,l↑∞(λl,µl,ν∗
l ) = (λ,µ,ν) and (λ,µ,ν) having Euclidean norm 1. Moreover, it
follows by Lemma 2.3 and the continuity of hj that J ⊆ {1 ≤ j ≤ r : hj(xQ) = 0}. Applying again
Lemma 2.3 and the continuity of the gradients the desired result follows from relation (2.4) by letting
l ∈ L1 converge to inﬁnity leading to the FJ condition:
Xq
i=0
λi∇fi(xQ) +
X
j∈J µj∇hj(xQ) +
X
k∈K
νkak = 0.
By construction the vectors ak, k ∈ K, are linearly independent. Since (λ,µ,ν) has Euclidean norm 1
and ak, k ∈ K, are linearly independent this implies (λ,µ) 6= 0. 
For problem (Q) we introduce the following constraint qualiﬁcation: The MF constraint qualiﬁcation
for problem (Q) is said to hold at a feasible point x if
4 October 12, 20053. Conclusion.
MF1. ∇hj(x), 1 ≤ j ≤ r are linearly independent.
MF2. lin{∇hj(x),1 ≤ j ≤ r} ∩ lin{ak,k ∈ K(x)} = {0}.
MF3. There exists some d0 satisfying
B(x)d0 ≤ 0,∇hj(x)>d0 = 0,0 ≤ j ≤ r, and maxi∈I(x){∇fi(x)>d0} < 0.
This is a natural condition. Without condition (MF2) a FJ point need not be a KKT point as shown by
the 2-dimensional optimization problem (with minimizer and FJ point xQ = 0)
min{x1 : x2 ≤ 0,−x2 ≤ 0,x2 − x2
1 = 0}.
Proof. (KKT conditions for problem (Q)) To show that at a minimizer xQ of problem (Q) satisfying
the MF constraint qualiﬁcation the KKT condition must hold we assume to the contrary that in the FJ
condition for problem (Q) we have λ0 = 0. By (MF3) it must follow that λ = 0 and using (λ,µ) 6= 0
it follows that µ 6= 0. Applying now (MF2) and (MF3) to the FJ conditions with λ = 0 and µ 6= 0 we
obtain a contradiction. 
As observed in Remark 2.1 we will now show for f0 pseudo-convex and fi,1 ≤ i ≤ q strictly pseudo-
convex on Rn, that for xP ∈ FP the condition max{∇fi(xP)>d : i ∈ I(xP) ∪ {0}} ≥ 0 for every
d such that B(xP)d ≤ 0 implies that xP is an global minimizer of problem (P). Recall that a function
φ : Rn 7→ R is called pseudo-convex on Rnif φ is differentiable on Rn and ∇φ(x)>d ≥ 0 implies
φ(x+d) ≥ φ(x) for every x,d ∈ Rn. It is called strictly pseudo-convex on Rn if φ is differentiable and
∇φ(x)>d ≥ 0 implies φ(x + d) > φ(x) for every x ∈ Rn and 0 6= d ∈ Rn [1].
Proof. (Converse of Lemma 2.1 for f0 pseudo-convex and fi,1 ≤ i ≤ q strictly pseudo-convex)
To prove the converse of Lemma 2.1 let us assume by contradiction that the feasible xP is not an global
minimizerofproblem(P). Hencethereexistssomex0 ∈ FP satisfyingf0(x0) < f0(xP). Bythepseudo-
convexity of f0 this implies that ∇f0(xP)>(x0 − xP) < 0. Also by strict pseudo-convexity of fi,1 ≤
i ≤ q using fi(x0) ≤ 0 = fi(xP),i ∈ I(xP) and x0 6= xP we obtain that ∇fi(xP)>(x0 − xP) < 0
for every i ∈ I(xP). Finally it holds that B(xP)(x0 − xP) ≤ 0 and we arrive at a contradiction to our
initial assumption. 
Combining Lemma 2.1 and 2.2 we immediately obtain the following result ([2]).
Lemma 2.4 Let f0 be pseudo-convex and fi,1 ≤ i ≤ q strictly pseudo-convex. Then it follows that
xP ∈ FP is a global minimizer of (P) if and only if xP satisﬁes the FJ conditions.
3 Conclusion.
In this note we have shown that the basic results in nonlinear programming are a natural and direct
consequence of basic results in linear programming and analysis. In our proof we could avoid the implicit
function theorem usually applied in the proof of the FJ conditions for problem (Q) (see for example
[2, 5]). The proof of the implicit function theorem ([11]) and its understanding is in general difﬁcult
for undergraduate/graduate students in the applied computational sciences. This concern was also the
main objective for constructing an alternative elementary proof by McShane ([10]) for the FJ and KKT
conditions for problem (Q). By not regarding separately linear and nonlinear inequalities the result in
5 October 12, 20054. Appendix.
[10] is weaker than ours (also the linear independence constraint qualiﬁcation for (Q) is used) and his
proof uses the penalty approach of nonlinear programming (see also [3] for a similar proof). As such this
technique and the technique used in this paper have their pros and cons. An advantage of the presented
approach for problem (P) is the fact that it can easily identify the class of functions for which the FJ
conditions for problem (P) are not only necessary but also sufﬁcient. This seems to be difﬁcult to show
by means of the penalty approach of McShane. However, to our belief the main advantage of our proof
technique is its display of a natural connection between linear and nonlinear programming.
4 Appendix.
In this appendix we give a short proof of Lemma 2.2 by means of the strong duality theorem for linear
programming.
Proof. To verify 1 ⇒ 2 we observe that
0 = minBd≤0 max1≤i≤s c>
i d =minBd≤0, c>
i d−z≤ 0 , 1≤i≤s z. (4.1)
This is a linear programming problem and by the strong duality theorem of linear programming (cf.[4])
we obtain
minBd≤0, c>
i d−z≤0, 1≤i≤s z = max{0>

λ
µ

:
Xs
i=1 λici + B>µ = 0,λ ∈ ∆s,µ ∈ R
p
+}. (4.2)
Applying now relations (4.1) and (4.2) we know that the feasible region of the dual problem is not empty
and so there exist some λ ∈ ∆p and µ ∈ R
p
+ satisfying
Ps
i=1 λic>
i d + B>µ = 0. To show the reverse
implication it follows that there exists some λ ∈ ∆s and µ ∈ R
p
+ satisfying
Ps
i=1 λic>
i d = −µ>Bd for
every d ∈ Rn. Hence for Bd ≤ 0 and using µ ∈ R
p
+ we obtain max1≤i≤s c>
i d ≥
Ps
i=1 λic>
i d ≥ 0. 
In our analysis we also use the following result known as Caratheodory’s lemma.
Lemma 4.1 Let v ∈ Rm be represented as cone combination v =
Pm
k=1 νkak, νk ≥ 0. Then there is a
representation v =
P
k∈K νkak, νk > 0, k ∈ K such that ak, k ∈ K are linearly independent.
Proof. We can assume
v =
Xm
k=1 νkak, with νk > 0, (4.3)
and suppose that the vectors ak,k = 1,...,m are linearly dependent. So there is a non-trivial combina-
tion 0 =
Pm
k=1 τkak. By multiplying this relation by a factor ρ and adding to (4.3) we ﬁnd
v =
Xm
k=1
(νk + ρτk)ak
and see that we can choose ρ ∈ R in such a way that (at least) one of the coefﬁcients (νk + ρτk) is zero
and the others ≥ 0. This can be done untill the desired representation is attained. 
References
[1] Avriel, M,Diewert, W.E., Schaible,S. and I.,Zang. Generalized Concavity, volume 36 of Mathemat-
ical concepts and Methods in Science and Engineering. Pleneum Press, New York, 1988.
6 October 12, 2005REFERENCES
[2] Bazaraa, M.S., Sherali, H.D. and C.M. Shetty. Nonlinear Programming: Theory and Algorithms
(second edition). Wiley-Interscience series in Discrete Mathematics and Optimization. Wiley, New
York, 1993.
[3] Bertsekas, D.P. Nonlinear programming. Athena Scientiﬁc, Massachusetts, 1995.
[4] Chv´ atal, V. Linear Programming. W.H Freeman and Company, New York, 1999.
[5] Faigle, U., Kern, K. and G. Still. Algorithmic Principles to Mathematical Programming). Kluwer
Academic publishers, Dordrecht, 2002.
[6] Farkas, J. Theorie der einfachen Ungleichungen. Journal f¨ ur die reine und angewandte Mathematic,
124:1–27, 1902.
[7] John, F. Extremum problems with inequalities as side conditions. In Friedrichs, K.O, Neuge-
bauer, O.E and J.J. Stoker, editor, Studies and Essays, Courant Anniversary Volume. Wiley-
Interscience, 1948.
[8] Karush, W. Minima of functions of several variables with inequalities as side conditions. Master’s
thesis, Department of Mathematics, University of Chicago, 1939.
[9] Kuhn, H.W and A.W. Tucker. Nonlinear programming. In Neyman, J, editor, Proc. 2nd Berkeley
Symposium on Mathematical Statistics and Probability. University of California Press, 1951.
[10] McShane, E.J. The Lagrange multiplier rule. The American Mathematical Monthly, 80(8):922–925,
1973.
[11] Rudin, W. Principles of Mathematical Analysis (third edition). Mc-Graw Hill, New York, 1976.
7 October 12, 2005

Algorithmic Principles to Mathematical Programming).

and I.,Zang. Generalized Concavity,

Extremum problems with inequalities as side conditions.

Minima of functions of several variables with inequalities as side conditions.

Nonlinear Programming: Theory and Algorithms (second edition).

Nonlinear programming.

Nonlinear programming. Athena Scientiﬁc,

Principles of Mathematical Analysis (third edition).

The Lagrange multiplier rule.

Theorie der einfachen Ungleichungen. Journal f¨ ur die reine und angewandte Mathematic,

An elementary proof of the Fritz-John and Karush-Kuhn-Tucker conditions in nonlinear programming

An elementary proof of the Fritz-John and Karush-Kuhn-Tucker conditions in nonlinear programming

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics