Applied Mathematics and Computation 190 (2007) 1030–1039

www.elsevier.com/locate/amc

Two differential equation systemsfor equality-constrained optimization q

Li Jin a,b,*, Li-Wei Zhang a, Xian-Tao Xiao a

a Department of Applied Mathematics, Dalian University of Technology, Dalian 116024, Liaoning Province, Chinab School of Mathematics, Physics and Information Science, Zhejiang Ocean University, Zhoushan, Zhejiang 316004, China

Abstract

This paper presents two differential systems, involving first and second order derivatives of problem functions, respec-tively, for solving equality-constrained optimization problems. Local minimizers to the optimization problems are provedto be asymptotically stable equilibrium points of the two differential systems. First, the Euler discrete schemes with con-stant stepsizes for the two differential systems are presented and their convergence theorems are demonstrated. Second, weconstruct algorithms in which directions are computed by these two systems and the stepsizes are generated by Armijo linesearch to solve the original equality-constrained optimization problem. The constructed algorithms and the Runge–Kuttamethod are employed to solve the Euler discrete schemes and the differential equation systems, respectively. We prove thatthe discrete scheme based on the differential equation system with the second order information has the locally quadraticconvergence rate under the local Lipschitz condition. The numerical results given here show that Runge–Kutta method hasbetter stability and higher precision and the numerical method based on the differential equation system with the secondinformation is faster than the other one.� 2006 Elsevier Inc. All rights reserved.

Keywords: Nonlinear equality-constrained optimization; Constraint qualification; Differential equation; Asymptotical stability; Equilib-rium point

1. Introduction

Differential equation methods, for solving equality-constrained optimization problems, have been proposedby Arrow and Hurwicz [1], Fiacco and mccormick [2] and Evtushenko [3], and developed by Yamadhita [4],Pan [5] and Evtushenko and Zhandan [6–10]. The main idea of this type of methods is to construct asystem of differential equations so that its equilibrium points of this system coincide with the solutions to

0096-3003/$ - see front matter � 2006 Elsevier Inc. All rights reserved.

doi:10.1016/j.amc.2006.11.041

q The research is supported by the National Natural Science Foundation of China under project Grant No. 10471015 and by theScientific Research Foundation for the Returned Overseas Chinese Scholars, State Education Ministry.

* Corresponding author. Address: School of Mathematics, Physics and Information Science, Zhejiang Ocean University, Zhoushan,Zhejiang 316004, China.

E-mail addresses: jinli_70@yahoo.com.cn (L. Jin), lwzhang@dlut.edu.cn (L.-W. Zhang).

L. Jin et al. / Applied Mathematics and Computation 190 (2007) 1030–1039 1031

the equality-constrained problem. Evtushenko and Zhandan have studied, by using the stability theory of dif-ferential equations, differential equation approaches for optimization problems over closed sets with equalityconstraints. This class of problems include inequality-constrained problems as special cases. They convert theconcerned problem to an equality-constrained one by using the so-called space transformations, and then con-struct a system of differential equations to solve the resulted equality-constrained problem. For equality-con-strained problems, the systems defined by Yamadhita [4] and Evtushenko [6] are quite similar, and both are ofthe form

BðxÞdx=dt ¼ �ðrf ðxÞ þ rhðxÞuðxÞÞ;dhðxÞ=dt ¼ �shðxÞ; ð1:1Þ

where BðxÞ 2 Rn;n is symmetrically positive definite, h : Rn ! Rl;rhðxÞ is the Jacobian transpose of h at x ands > 0 is a constant. Actually, if we choose s � 1, then the system in Yamashita [4] is obtained and if B(x) �In·n, then the above system becomes the one of Evtushenko and Zhandan [7]. Along this line, Zhang [11]studied a modified version to (1.1) for solving equality-constrained optimization problems.

The paper [12] presented two modified versions to the differential system proposed by Evtushenko [3].These methods are described in Section 2. In Section 3, On the base of [12], a differential system which involvethe second order derivatives of problem functions is constructed and the Euler discrete scheme for the differ-ential system with constant stepsizes is proved to be convergent. Furthermore, an Euler discrete scheme withArmijo line search for the second system is proposed and its global convergence and the locally quadratic con-vergence rate are demonstrated. In Section 4, the Runge–Kutta method and the Euler discrete scheme withArmijo line search are implemented to solve the two differential equation systems. The numerical resultsobtained show that Runge–Kutta method has good stability and high precision and the Euler discrete schemewith Armijo line search has the fast convergence.

2. The first differential system

We consider the equality-constrained problem of the form

min f ðxÞs:t: hiðxÞ ¼ 0; i ¼ 1; . . . ; l;

ð2:1Þ

where f : Rn ! R, and hi : Rn ! R; 1 6 i 6 l are twice continuous differentiable functions, whose gradientsand Hessian matrices will be denoted by $f (x), $hi(x), 1 6 i 6 l andr2

xxf ðxÞ; r2xxhiðxÞ; 1 6 i 6 l, respectively.

Furthermore, we use the notation I(x) = {ijhi(x) = 0, i = 1, . . . , l}, which denote the active set of indices at x,X = {x 2 Rnjh(x) = 0} and $h(x) = ($h1(x), . . . ,$hm(x)). Then the solution x� 2 Rn of the Problem (2.1) satis-fies the Karush–Kuhn–Tucker (KKT) conditions, i.e., there exists a vector u� 2 Rm, such that

rxLðx�; u�Þ ¼ rf ðx�Þ þ rhðx�Þu� ¼ 0; hðx�Þ ¼ 0; ð2:2Þ

where $h(x*) is full of column rank and

Lðx; uÞ ¼ f ðxÞ þ uThðxÞ;

is the Lagrangian function, whose gradient and Hessian matrix will be denoted by

rxLðx; uÞ ¼ rf ðxÞ þXl

i¼1

uirhiðxÞ;

r2xxLðx; uÞ ¼ r2

xxf ðxÞ þXl

i¼1

uir2xxhiðxÞ

and (x*,u*) is the KKT pair. Let Z 2 Rn,n�l be a matrix whose columns form a set of basis vectors of Null

($h(x*)T) so that $h(x*)TZ = 0, then

ZTr2xxLðx�; u�ÞZ > 0:

1032 L. Jin et al. / Applied Mathematics and Computation 190 (2007) 1030–1039

Definition 2.1. The constraint qualification CQ for Problem (2.1), if the set of vectors {$hi(x)ji 2 I(x)} arelinearly independent for each x 2 X.

For equality-constrained problems (2.1), the system defined in Zhang et al. [12] is described by the followingdifferential equations:

dx=dt ¼ �rxLðx; uðxÞÞ; ð2:3ÞdhðxÞ=dt ¼ GðhðxÞÞuðxÞ þ UðhðxÞÞ; ð2:4Þ

where Gð�Þ ¼ ðG1ð�Þ; . . . ;Glð�ÞÞT : Rl ! Rl;l and U : Rl ! Rl. We pose four conditions on G(Æ) and U(Æ) neededin this section.

(a1) G1(Æ), . . . ,Gl(Æ) are differentiable on U, a neighborhood of the origin of Rl, G(0) = 0 and($h(x)T$h(x) + G(h(x))) is nonsingular for x 2 D, where D is a neighborhood of x*.

(a2) U(Æ) is differentiable on U, 0 is the unique solution of U(h) = 0 on U.(a3) All eigenvalues of the matrix

rhUð0ÞT þXl

i¼1

u�irhGið0ÞT

have negative real parts.

(a4) G(h(x)) 62 Range (G(h(x))) for h(x) 5 0.

We introduce some notations

MðxÞ ¼ ðrhðxÞTrhðxÞ þ GðhðxÞÞÞ�1; P ðxÞ ¼ In �rhðxÞMðxÞrhðxÞT: ð2:5Þ

Since M(x) is nonsingular on D, u(x) is uniquely determined from (2.4)

uðxÞ ¼ �MðxÞðrhðxÞTrf ðxÞ þ UðhðxÞÞÞ: ð2:6Þ

Now system (2.3) can be expressed as

dx=dt ¼ �rf ðxÞ þ rhðxÞMðxÞðrhðxÞTrf ðxÞ þ UðhðxÞÞÞ: ð2:7Þ

By combining (2.3)–(2.5), we obtain also that

dx=dt ¼ �PðxÞrxLðx; uðxÞÞ þ rhðxÞMðxÞðGðhðxÞÞuðxÞ þ UðhðxÞÞÞ: ð2:8Þ

Integrating the equation defined by (2.3) using Euler method, one obtains the iterate process

xkþ1 ¼ xk � akrxLðxk; uðxkÞÞ;uðxkÞ ¼ �MðxkÞðrhðxkÞTrf ðxkÞ þ UðhðxkÞÞÞ;

ð2:9Þ

where ak is a stepsize.

Lemma 2.1. If (x*, u*) is a Kuhn–Tucker pair of (2.1), $h(x*) has a full column rank, then x* is an equilibrium

point of (2.3) (or (2.8)). If x* is an equilibrium point of (2.3) satisfying that $h(x*)T is of full row rank and

assumptions (a1)–(a4) are valid, then (x*, u*) is a Kuhn–Tucker pair of (2.1).

Proof. If (x*,u*) is a Kuhn–Tucker pair, namely

rxLðx�; u�Þ ¼ 0; hðx�Þ ¼ 0;

and this implies

u� ¼ �ðrhðx�ÞTrhðx�ÞÞ�1rhðx�ÞTrf ðx�Þ ¼ uðx�Þ;

where u(x) is defined by (2.6), thus x* is an equilibrium point of (2.3).

If x* is an equilibrium point of (2.3), we have $xL(x*,u*) = 0 and (2.8) implies that

rhðx�ÞMðx�ÞðGðhðx�ÞÞuðx�Þ þ Uðhðx�ÞÞÞ ¼ 0:

L. Jin et al. / Applied Mathematics and Computation 190 (2007) 1030–1039 1033

Noting that $h(x*)T is of full row rank and M(x*) is nonsingular, we have from the above equality thatG(h(x*))u(x*) + U(h(x*)) = 0. If u(x*) = 0, than $f(x*) = 0 and the definition of u(x*) shows thatM(x*)U(h(x*)) = 0 which implies h(x*) = 0. Thus (x*,u(x*)) is a Kuhn–Tucker pair. If u(x*) 5 0, then we havefrom assumption (a4) that h(x*) = 0.

Now we prove the system (2.8) is asymptotically stable and the iterative process (2.9) is locallyconvergent. h

Theorem 2.1. Let (x*, u*) is a Kuhn–Tucker pair of (2.1), $h(x*) has a full column rank; let constraint qualifica-

tion CQ hold at x 2 D. Then the system (2.8) with G(Æ) and U(Æ) satisfying conditions (a1)–(a4) is locally asymp-

totically stable at the local solution x* of the Problem (2.1).

Proof. Denote dx(t) = x(t,x0) � x*, where x0 2 D is close enough to x* and linearize system (2.8) in the neigh-borhood of x* on D. Then we obtain the equation of the first-order approximation of (2.8) about the equilib-rium point x*

d _x ¼ �Qðx�Þdx; ð2:10Þ

where

Qðx�Þ ¼ �P ðx�Þr2xxLðx�; u�Þ þ rhðx�ÞMðx�Þ rhUð0ÞT þ

Xm

i¼1

u�irhGið0ÞT !

rhðx�ÞT: ð2:11Þ

We choose Z 2 Rn;n�l satisfying

rhðx�ÞTZ ¼ 0; ZZT ¼ P ðx�Þ; Rankðrhðx�Þ; ZÞ ¼ n:

Consider the characteristic equation

detðQðx�Þ � kInÞ ¼ 0; ð2:12Þ

which is equivalent to the equation

detðNðx�; kÞÞ ¼ 0;

where

Nðx�; kÞ ¼ rhðx�ÞT

ZT

!ðQðx�Þ � kInÞðrhðx�Þ; ZÞ:

Since M(x*) = ($h(x*)T$h(x*))�1, we have

Nðx�; kÞ ¼N 1ðx�; kÞ 0

N 2ðx�; kÞ N 3ðx�; kÞ

� �;

where

N 1ðx�; kÞ ¼ rhUð0ÞT þXm

i¼1

u�irhGið0ÞT !

� kIl

!rhðx�ÞTrhðx�Þ;

N 2ðx�; kÞ ¼ �ZTZZTr2xxLðx�; u�Þrhðx�Þ;

N 3ðx�; kÞ ¼ �ZTZðZTr2xxLðx�; u�ÞZ � kIn�lÞ;

then we have the equality

detðNðx�; kÞÞ ¼ detðN 1ðx�; kÞÞ � detðN 3ðx�; kÞÞ ¼ 0:

Noting that $h(x*)T $h(x*), ZTr2xxLðx�; u�ÞZ and ZTZ are positive define, we have that n � l eigenvalues of

Q(x*) coincide with those of �ZTr2xxLðx�; u�ÞZ and the rest l eigenvalues of Q(x*) are those of

rhUð0ÞT þPl

i¼1u�irhGið0ÞT.

1034 L. Jin et al. / Applied Mathematics and Computation 190 (2007) 1030–1039

Thus all eigenvalues of Q(x*) have negative real parts. From Lyapunov’s stability theorem of the first-orderapproximation, it holds that x* is a local asymptotically stable equilibrium of (2.8). h

Theorem 2.2. Let the condition of Theorem 2.1 be satisfied. Then there exists an �a > 0 such that for any

0 < ak < �a the iterations defined by (2.9) converge locally to x*.

Proof. Since x* is a fixed point of the mapping

F ðxÞ ¼ x� aðP ðxÞrxLðx; uðxÞÞÞ � rhðxÞMðxÞðGðhðxÞÞuðxÞ þ UðhðxÞÞÞ;

where u(x) is defined by (2.6). The convergence of the iterations (2.9) will be proved if we demonstrate that a �tcan be chosen such that the iterations defined by

xkþ1 ¼ F ðxkÞ

converge to x* whenever x0 is in a neighborhood of x* and 0 < ak < �a. Let k1, . . . , kn�l be the n � l eigenvalues

of ZTr2z Lðz�;w�ÞZ and bj = aj + ibj, j = 1, . . . , l be the l eigenvalues of matrix rhUð0ÞT þ

Pli¼1u�irhGið0ÞT. Let

$F(z) be Jacobian transpose of F(z). From the proof of Theorem 2.1, one has that the eigenvalues of $F(x*)T,denoted by l1, . . . ,ln have the expression

lj ¼ 1� akj; j ¼ 1; . . . ; n� l;

lj ¼ ð1þ aaj�nþlÞ þ aibj�nþl; j ¼ n� lþ 1; . . . ; n:

The condition jljj < 1 can be written as

a < �a1 ¼ minf2=kjj1 6 j 6 n� lg

and

a < �a2 ¼ minf�2aj=ða2j þ b2

j Þj1 6 j 6 mþ lg:

Let

�a ¼ minf�a1; �a2g

the spectral radius of $F(x*)T is strictly less than 1 for a < �a, and the iterations generated by the scheme (2.9) islocally convergence to x* [6]. The proof is completed. h

We adopt the Armijo line search rule to determine steplengths because the number �a > 0 in Theorem 2.2 isnot available in prior when the Euler method is used. Let /(x) = k$xL(x,u(x))k2 be the merit function.

Algorithm 2.1

Step 1: Given b 2 (0,1), s > 0, r 2 (0, 1/2), e P 0 and the initial point x0, k :¼ 1.Step 2: If /(x) 6 e, then stop; otherwise, computer the search direction

dðxkÞ ¼ �rxLðxk; uðxkÞÞ:

Step 3: Computer the approximate iterate point

xkþ1 ¼ xk þ hkdðxkÞ;

where hk ¼ aik , ik is the smallest nonnegative integer i satisfying

/ðxk þ bik dðxkÞÞ 6 ð1� 2rbik Þ/ðxkÞ;

Step 4: k :¼ k + 1, go to Step 2.

Theorem 2.3. Let the problem functions of Problem (2.1) be twice continuously differentiable. Let {xk} be gen-

erated by Algorithm 2.1 and x be a cluster point of this sequence. Then the iterations {xk} converge to a solution

to /(x) = 0.

L. Jin et al. / Applied Mathematics and Computation 190 (2007) 1030–1039 1035

Proof. Obviously, we have that the sequence {/(xk)} is monotone decreasing and convergent. Let �x be a clus-ter point of {xk}, then there is fxkjg, such that

xkj ! �x; j!1;

where {kj} � {1,2,3, . . .}. From Armijo line search rule, we have that

/ðxkjÞ � /ðxkj þ bkj dkjÞP 2rbkj/ðxkjÞ;/ðxkjÞ � /ðxkj þ bkj�1dkjÞ < 2rbkj�1/ðxkjÞ:

Assume that /ð�xÞ 6¼ 0, then from the first inequality above, we have for j!1, bkj ! 0. So from the secondinequality, we obtain

/ðxkj þ bkj�1dkjÞ � /ðxkjÞbkj�1

> �2r/ðxkjÞ;

hr/ð�xÞ; �diP �2r/ð�xÞ;ð2:13Þ

where �d ¼ �rxLð�x; �uÞ; �u ¼ uð�xÞ. Since

hr/ð�xÞ; �di ¼ �h2rxLð�x; �uÞ;rxLð�x; �uÞi ¼ �2krxLð�x; �uÞk2 ¼ �2/ð�xÞ ð2:14Þ

by combining (2.13) and (2.14), we obtain that

0 6 ð2� 2rÞ/ð�xÞ 6 0;

therefore /ð�xÞ ¼ 0, which contradicts with the assumption /ð�xÞ 6¼ 0. Therefore we must have /ð�xÞ ¼ 0. Weconclude that any cluster point of {xk} is the solution of /(x) = 0. h

3. The second differential system

The method in Section 2 can be viewed as a gradient approach, through the system (2.3), for solving Prob-lem (2.1). Now we discuss the Newton version. The continuous version of Newton method leads to the initialvalue problem for the following system of ordinary differential equations:

KðxÞdx=dt ¼ diag16i6n

cirxLðx; uðxÞÞ; ð3:1Þ

� rhðxÞTrxLðx; uðxÞÞ ¼ GðhðxÞÞuðxÞ þ UðhðxÞÞ; ð3:2Þ

where (c1, . . . ,cn) is a scaling vector, K(x) is the Jacobian matrix of the mapping $xL(x,u(x))

KðxÞ ¼ r2xxLðx; uðxÞÞ þ rhðxÞruðxÞT: ð3:3Þ

Let us assume that ci = 1, 1 6 i 6 n, by differentiating equality (3.2) with respect x, we obtain

� EðxÞ � rhðxÞTr2xxLðx; uðxÞÞ � rhðxÞTrhðxÞruðxÞT

¼Xl

i¼1

uirhGiðhðxÞÞTrhðxÞT þ GðhðxÞÞruðxÞT þrhUðhðxÞÞTrhðxÞT;

eij ¼Xn

k¼1

o2hiðxÞoxkoxj

oLðx; uðxÞÞoxk

;

ð3:4Þ

where E ¼ ½eij� 2 Rl�n. From (3.2) and (3.4) we find that

uðxÞ ¼ �ðrhðxÞTrhðxÞ þ GðhðxÞÞÞ�1ðrhðxÞTrf ðxÞ þ UðhðxÞÞÞ;ruðxÞT ¼ �ðGðhðxÞÞ þ rhðxÞTrhðxÞÞ�1ðEðxÞ þ rhðxÞTr2

xxLðx; uðxÞÞ

þXl

i¼1

uirhGiðhðxÞÞTrhðxÞT þrhUðhðxÞÞTrhðxÞTÞ: ð3:5Þ

1036 L. Jin et al. / Applied Mathematics and Computation 190 (2007) 1030–1039

From (3.3) and (3.5), we can easily get the formula

KðxÞ ¼ r2xxLðx; uðxÞÞ � rhðxÞðGðhðxÞÞ

þ rhðxÞTrhðxÞÞ�1 EðxÞ þ rhðxÞTr2xxLðx; uðxÞÞ þ

Xl

i¼1

uirhGiðhðxÞÞrhðxÞT þrhUðhðxÞÞTrhðxÞT !

:

ð3:6Þ

Lemma 3.1. Let the conditions of Theorem 2.1 be satisfied at the point (x*,l*). Then K(x*) is a nonsingular

matrix.

Proof. Let (x*,l*) be a Kuhn–Tucker point of (2.1), then E(x*) = 0, we obtain

Kðx�Þ ¼ r2xxLðx�; uðx�ÞÞ � rhðx�ÞðGð0Þ

þ rhðx�ÞTrhðx�ÞÞ�1 rhðx�ÞTr2xxLðx�; uðx�ÞÞ þ

Xl

i¼1

u�irhGið0Þrhðx�ÞT þrhUð0ÞTrhðx�ÞT !

:

From (2.5) we have

Kðx�Þ ¼ P ðx�Þr2xxLðx�; uðx�ÞÞ � rhðx�ÞMðx�Þ rhUð0ÞT þ

Xl

i¼1

u�irhGið0ÞT !

rhðx�ÞT:

Hence the matrix K(x*) coincides with the matrix – Q(x*) which was introduced in (2.11). It was shown in theproof of Theorem 2.1 that all eigenvalues of this matrix have negative parts. Therefore, K(x*) isnonsingular. h

Theorem 3.1. Let the conditions of Theorem 2.1 be satisfied at the point (x*,l*). Then the system (3.1) is asymp-

totically stable at x*.

Proof. From the second order smoothness of $xL(x,u(x)) around x*, we have

rxLðx; uðxÞÞ ¼ rxLðx�; uðx�ÞÞ þ Kðx�ÞdðxÞ þHðdðxÞÞ;

where d(z) = x � x*, H(d(x)) = O(kd(x)k2).

Linearizing system (3.1) at the point x*, we obtain

ddðxÞ=dt ¼ �bQðx�ÞdðxÞ; bQðx�Þ ¼ Kðx�Þ�1 diag16i6n

ciKðx�Þ:

The stability of this system is determined by the properties of the roots of the characteristic equation

detðbQðx�Þ � kInÞ ¼ 0:

Matrix bQðx�Þ is similar to matrix diag16i6nci; therefore, they have the same eigenvalues ki = ci > 0, 1 6 i 6 n.According to Lyapunov linearization principle, we have that the equilibrium point x* is asymptotically stable.

Integrating the system (3.1) by the Euler method, one obtains the iterate process

xkþ1 ¼ xk � hk diag16i6n

ciKðxkÞ�1rxLðxk; uðxkÞÞ; ð3:7Þ

uðxkÞ ¼ �MðxkÞðrhðxkÞTrf ðxkÞ þ UðhðxkÞÞÞ: �

Theorem 3.2. Let the conditions of Theorem 2.1 be satisfied at the point (x*,l*). If the stepsize hk is fixed and

0 < hk < 2/max16i6nci, then the iterations defined by (3.7) converges locally to x*.

The proof of the statements of this theorem is nearly identical to the proof of Theorem 2.2.We adopt the Armijo line search rule to determine steplengths because the number �t > 0 in Theorem 3.2 is

not available in prior when the Euler method is used. Let E(x) = k$xL(x,u(x))k2 be the merit function.

L. Jin et al. / Applied Mathematics and Computation 190 (2007) 1030–1039 1037

Algorithm 3.1

Step 1: Given a 2 (0,1), s > 0, ci = 1 (1 6 i 6 n), r 2 (0,1/2), e P 0 and initial point x0, k :¼ 1.Step 2: If E(x) 6 e, then stop; otherwise, compute the search direction

dk ¼ �KðxkÞ�1rxLðxk; uðxkÞÞ:

Step 3: Compute the approximate iterate point

xkþ1 ¼ xk þ hkdk;

where hk ¼ aik , ik is the smallest nonnegative integer i satisfying

Eðxk þ aik dkÞ 6 ð1� 2raik ÞEðxkÞ:

Step 4: k :¼ k + 1, go to Step 2.

Theorem 3.3. Let the problem functions of Problem (1.1) be twice continuously differentiable. Let {xk} begenerated by Algorithm 3.1 satisfying that for any cluster point x of this sequence, K(x) is nonsingular. Then

the iterations {xk} converge to a solution to E(x) = 0.

The proof of the statements of this theorem is nearly identical to the proof of Theorem 2.3.

Theorem 3.4. Let the conditions of Theorem 2.1 be satisfied at the the point (x*,l*), K(x) satisfies a Lipschitz

condition in a neighborhood of x*. Assume that the sequence {xk} generated by Algorithm 3.1 converges to x*,

then there is an integer K large enough such that hk = 1 for k > K, the sequence {xk} generated by Algorithm 3.1converges quadratically to x*.

Proof. Let H(x) = $xL(x,u(x)), then H(x*) = 0n. From Lemma 3.1 we have that K(x*) is a nonsingular matrix,then there are constants c0 > 0, c1 > 0, c2 > 0, c3 > 0, and a neighborhood Nðx�Þ, such that

kHðxÞk ¼ kHðxÞ � Hðx�ÞkP c0kx� x�k;kHðxÞk ¼ kHðxÞ � Hðx�Þk 6 c1kx� x�k;kKðxÞ � Kðx�Þk 6 c2kx� x�k;

kKðxÞ�1k 6 c3; x 2Nðx�Þ

ð3:8Þ

for sufficiently large K, when k P K, we have

kxk � x�k 6 r0; xk 2Nðx�Þ; ð3:9Þ

where r0 ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffi1� 2rp

c0=ðc1c3c2ð2þ c3c1ÞÞ and therefore

kdkk ¼ kKðxkÞ�1HðxkÞk 6 c3c1kxk � x�k: ð3:10Þ

We can deduce from (3.8)–(3.10) that

kHðxk þ dkÞk ¼ kHðxk þ dkÞ � HðxkÞ þ KðxkÞdkk 6 sup06t61

kðKðxk þ tdkÞ � KðxkÞÞdkk

6 sup06t61

kðKðxk þ tdkÞ � Kðx�ÞÞdkk þ kðKðxkÞ � Kðx�ÞÞdkk

6 2c2kxk � x�kkdkk þ c2kdkk26 c1c3c2ð2þ c3c1Þkxk � x�k2

; ð3:11Þ

therefore, when k P K,

EðxkÞ � Eðxk þ dkÞ � 2rEðxkÞ ¼ ð1� 2rÞkHðxkÞk2 � kHðxk þ dkÞk2

P ð1� 2rÞc20kxk � x�k2 � ½c1c3c2ð2þ c3c1Þ�2kxk � x�k4

P ½ð1� 2rÞc20 � ½c1c3c2ð2þ c3c1Þ�2r2

0�kxk � x�k2 P 0; ð3:12Þ

1038 L. Jin et al. / Applied Mathematics and Computation 190 (2007) 1030–1039

which implies that the steplength hk = 1 will satisfy Armijo line search rule when k P K in Algorithm 3.1 andhence Algorithm 3.1 becomes the classical Newton method and the rate of convergence is quadratic, see forinstance [13]. h

At the end of this section, we discuss some special choices of G(Æ) and U(Æ), consider the following simpleform of U(Æ):

TableNume

No.

1 [15,

2 [16,

3 [17,

4 [17,

n StanKutta

UðhðxÞÞ ¼ �shðxÞ; s > 0

and two special forms of G:

(1) G(h(x)) = kh(x)k2Il. Conditions (a1) and (a2) are obviously valid. Noting that $hGi(0) = 0 and$hU(0) = �sh, we have that condition (a3) holds.

(2) G(h(x)) = �diag16i6l (hi). In view of [14], one has that $h(x)T$h(x) + G(h(x)) is positive definite if theconstraint qualification holds, and condition (a1) is valid. Noting that

rhUð0ÞT þXl

i¼1

u�irhGið0ÞT ¼ � diag16i6l

u�i � sIl

if s > ju*j1, then condition (a4) holds and such (G,U) can be chosen as a candidate pair.

4. Computation results

The results of some computational tests are reported in this section. we implement a code complied basedon Algorithms 2.1 and 3.1 and the Runge–Kutta method for the differential equation system (2.3) and system(3.1), where U(H(z)) = �sH(z), G(h (x)) = �diag16i6l(hi). The test problems chosen from [15–17] are used tocompare the numerical performances of these two approaches.

To compare the numerical results obtained by the four different implementations, we adopted the same theinitial point and s in computing each of the problems. The numerical results in Table 1 show that Runge–Kutta method has good stability and high precision and Algorithm 3.1 is fast convergent.

1rical results

Algorithm IT k$xL(x,u(x))k2 Accuracy Time (s)

Problem 269] n = 5 R(3.1) 104 8.327355 · 10�10 0.002 1.1376 · 103

m = 3 R(2.2) 44 1.037953 · 10�9 0.002 36.12s = 2 3.1 19 4.107369 · 10�10 0.002 304.689

2.1 23 8.59763 · 10�7 0.002 39.1760

P3 (Powell)] n = 5 R(3.1) 38 1.224301 · 10�8 1.5051 · 10�5 635.4940m = 3 R(2.2) 98 7.097067 · 10�8 0.0013 1335.6s = 2 3.1 10 7.541356 · 10�8 1.3732 · 10�4 182.0620

2.1 30 9.835789 · 10�8 0.0019 99.663

Tp56] n = 7 R(3.1) 68 8.310316 · 10�8 1.4228 · 10�4 1775.7m = 4 R(2.1) 158 7.507266 · 10�7 8.9264 · 10�4 233.3850s = 1 3.1 11 3.818743 · 10�8 1.8613 · 10�4 236.029

2.1 74 9.771242 · 10�7 9.5348 · 10�5 7690.1

Tp51] n = 5 R(3.1) 46 2.929458 · 10�8 7.5655 · 10�5 1197.1m = 3 R(2.2) 68 3.465951 · 10�8 7.3994 · 10�5 185.8180s = 1 3.1 15 3.755046 · 10�8 8.5375 · 10�5 116.8780

2.1 33 5.878657 · 10�8 6.9512 · 10�5 63.8350

ds for the dimension of variables, m for the number of constraints, IT for the number of iterations, R(3.1) and R(2.3) for the Runge–method for the differential equation system (3.1) and (2.3), respectively.

L. Jin et al. / Applied Mathematics and Computation 190 (2007) 1030–1039 1039

References

[1] K.J. Arrow, L. Hurwicz, Reduction of constrained maxima to saddle point problems, in: J. Neyman (Ed.), Proceedings of the 3rdBerkeley Symposium on Mathematical Statistics and Probability, University of California Press, Berkeley, 1956, pp. 1–26.

[2] A.V. Fiacco, G.P. Mccormick, Sequential Unconstrained Minimization Techniques, Nonlinear Programming, John Wiely, NewYork, 1968.

[3] Yu.G. Evtushenko, Two numerical methods of solving nonlinear programming problems, Sov. Math. Dokl. 15 (2) (1974) 420–423.[4] H. Yamadhita, A differential equation approach to nonlinear programming, Math. Prog. 18 (1980) 115–168.[5] P.Q. Pan, New ODE methods for equality constrained optimization (1)-equations, J. Comput. Math. 10 (1) (1992) 77–92.[6] Yu.G. Evtushenko, Numerical Optimization Techniques, Optimization Software Inc., Publication Division, New York, 1985.[7] Yu.G. Evtushenko, V.G. Zhandan, Barrier-projective methods for nonlinear programming, Comp. Math. Math. Phys. 34 (5) (1994)

579–590.[8] Yu.G. Evtushenko, V.G. Zhandan, Stable barrier-projection and barrier-Newton methods in nonlinear programming, Opt. Meth.

Software 3 (1994) 237–256.[9] Yu.G. Evtushenko, V.G. Zhandan, Stable barrier-projection and barrier-Newton methods for linear and nonlinear programming, in:

E. Spedicato (Ed.), Algorithms for Continuous Optimization, Kulwer Academic Publishers, 1994, pp. 255–285.[10] Yu.G. Evtushenko, V.G. Zhandan, Stable barrier-projection and barrier-Newton methods in linear programming, Comput. Opt.

Appl. 3 (1994) 289–303.[11] L.W. Zhang, A modified version to the differential system of Evtushenko and Zhanda for solving nonlinear programming, in:

Ya-xiang Yuan (Ed.), Numerical Linear Algebra and Optimization, Science Press, Beijing, New York, 1999, pp. 161–168.[12] Liwei Zhang, Qin Li, Xin Zhang, Two differential systems for solving nonlinear programming problems, OR Trans. 4 (4) (2000) 33–

46.[13] J.M. Ortega, W.C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, New York, 1970.[14] Z.Y. Gao, G.P. He, A generalized projection gradient method for constrained optimization problems, Chin. Sci. Bull. 36 (19) (1991)

1444–1447.[15] Schittkowski, More test examples for nonlinear programming codes, Lecture Note in Economics and Mathematical System, vol. 282,

Springer, Berlin, 1987.[16] Wagner Michael, J. Todd Michael, Least-change quasi-Newton updates for equality-constrained optimization, Math. Program. 87

(2000) 317–350.[17] W. Hock, K. Schittkowski, Test examples for nonlinear programming codes’, Lecture Notes in Economics and Mathematical

Systems, vol. 187, Springer, 1981.