Cojugate Gradient Method

Zhengru Zhang ( 张争茹 )

zrzhang@bnu.edu.cn

Office: Math. Building 413(West)

2010 年教学实践周 7.12-7.16

Outline• Aim

• Method of Gauss Elimination

• Basic Iterative Methods

• Conjugate Gradient Method

– Derivation

– Theory

– Algorithm

• References

• Homework & Project

AimSolve linear algebraic system like

a11 x1 + a12 x2 + ... + a1n xn = b1

a21 x1 + a22 x2 + ... + a2n xn = b2

...

an1 x1 + an2 x2 + ... + ann xn = bn

Using matrix, the above system can be written as

Ax=bA is a N x N matrix, b is a N x 1 vector

Consider the case: A is large and sparse

Method of Gauss Elimination

U = A, L = I

for k = 1 to N-1

for j = k +1 to N

ljk = ujk/ukk

uj,k:m = uj,k:m – ljkuk,k:m

Algorithm of Gaussian Elimination without Pivoting

• LU Factorization, let A=LU

• Solve Ly=b

• Solve Ux=y

Operation Count of Gauss Elimination

• Gauss Elimination and Back Substitution

• There are 3 loops

• There are 2 flops per entry

• For each k, the inner loop is repeated for rows k +1, …, N

• Cost: about About N 3 flops23

23

Instability of Gaussian Elimination without Pivoting

11

10

11

110 20

A1= A2=

Examples

Remedy• Pivoting• Partial Pivoting• Complete Pivoting

Algorithm of Gaussian Elimination with Partial Pivoting

U = A, L = I

For k = 1 to N-1

for j = k +1 to N

ljk = ujk/ukk

uj,k:m = uj,k:m – ljkuk,k:m

Basic Iterative Methods

• How to construct iterative sequence?

• Convergence? Conditions?

• Convergence rate?

Jacobi iteration

X[k+1] = D-1(L+U) X[k] + D-1 bB = D-1(L+U)

Gauss Seidel iteration

X[k+1] = (D-L)-1 U X[k] + (D-L)-1 bB = (D-L)-1 U

• Iterative method X[k+1] = BX[k] +g converges if and only if

(B) < 1

Convergence rate

||X[k]-X*|| ||X[1]-X[0]||, where q =||B||<11

kq

q

Steepest Decent Method• Consider the case: A is symmetric positive definite

• Quadratic functional

(x) = xTAx - 2bTx

• The solution of Ax=b is equivalent to find the minimizer of the functional (x)

• Method of optimization: find a direction pk and a step k

Steepest Decent MethodDetermine pk and k

• Suppose that pk is determined. Let’s start from xk

• Let f() = (xk + pk)

= (xk + pk)TA(xk + pk)-2bT(xk + pk)

= 2pkTApk - 2 rk

Tpk + (xk)

where rk = b - Axk (Residual)

• By calculas f’() = 2pkTApk- 2rk

Tpk =0

• Then let xk+1 = xk + k pk

Tk k

k Tk k

r p

p Ap

Algorithm for Steepest Decent Method

• Verify (xk+1) - (xk) = (xk + k pk) - (xk)

= k2pk

TApk - 2k rkTpk

• How to determine the direction pk ?

take as the negative gradient pk = rk

2( )0 (if 0)

TTk kk kT

k k

r pp Ap

p Ap

Algorithm Convergence Theorem

Suppose the eigenvalues of A

then there holds

where

Conjugate Gradient MethodDerivation

• Negative gradient direction rk is the locally

steepest

decent direction, but it may not be the global one

• Consider a new direction: combination of rk and

pk-1

• Initially, take p0 = r0 , x1 = x0 + 0p0

• For step k +1, choose and to minimize

• By calculas

• The corresponding minimizer is

0 and 0 satisfy

take

Let

• In summary,

01

0k

0

Algorithm for CG method

Operations involved:

• Transpose,

• Scalar Multiply,

• Matrix Add,

• Matrix Multiply

Where and are obtainedin a simple form

Properties for CG method

Orthogonalproperties

• Theoretically, CG method is an exact method. Actually,

works as an iterative method.

• Convergence rate:

where

References

• 徐树方，高立，张平文， 数值线性代数，北京大学出版社，北京， 2007

• 袁亚湘，孙文瑜， 最优化理论与方法，科学出版社，北京， 2000

• Yousef Saad, Iterative Methods for Sparse Linear Systems, 2000

Home & ProjectDue at the end of this week

• Solve the following linear systems using CG method2 2

1

0

1(1 ) ( ) 1, 2, , 1

2 2 20

j j j j

n

h hu u u f j n

u u

where 2 2 1(1 4 )sin , j

jf h

n n

• Set n = 100, 200, 300, 400, 500

• Use Matlab to graph the solution (j, uj)

Problem: Minimize the functional E(u)=∫(|u|2+u2-2fu )dx

The corresponding Euler-Lagrange equation is

E/u=-2u+2u-2f=0 or -u+u=f

-u xx + u =f 0<x<1

f=(1+42)sin2x

u(0)=u(1)=0

Home & ProjectDue at the end of this week

• Solve the following linear systems using CG method2 2

1, 1, , 1 , 1

,0 ,0

0, ,

1(1 ) ( )

4 4 4 , 1, 2, , 1

0, 0,1, 2 ,

0, 0,1,2 ,

ij i j i j i j i j ij

i n

j n j

h hu u u u u f

i j n

u u i n

u u j n

2 2 2 2 1

200( ( ) ) 200( ( ) ) 100(( ) )(( ) ), ij

j j i i i i i if h

n n n n n n n n n

• The unknowns can be ordered as below

,1 ,2 , 1 2,1 2,2 2, 1 1,1 1,2 1, 1, , , , , , , , ,i i i n n n n n nu u u u u u u u u

• The coefficient matrix is of the block tridiagonal form

2 2( 1) ( 1)

n n

S B

B S B

B SA

B

B S

( 1) ( 1)

1

4 n nB I Where

S Tridiagonal matrix with

diagonal entry:

other entry:

2

14

h

1

4

• Set n=20,40,80,100. Find the solution

• Use Matlab to graph the solution (i, j, uij)

-u+u=f (x,y)(0,1)(0,1)

u(x,y)=100(x2-x)(y2-y)

f=200(y-y2) + 200(x-x2) + 100(x2-x)(y2-y)

The End